Обзор технологий выявления модифицированного контента класса DeepFake
Аннотация
В работе приводится обзор публикаций об основных технологиях выявления модифицированного контента класса DeepFake, в том числе, список публично доступных датасетов и результаты тестирования решений, имеющихся в публичном доступе. В работе также приводятся результаты независимого тестирования систем обнаружения DeepFake, полученные в ходе открытого конкурса DFDC – 2019 (DeepFake Detection Contest), а также краткие обзоры проводимых подобных конкурсов в 2020 -2021 годах в КНР.
Статья также описывает новый подход к обнаружению фото/видеоматериалов, созданных с применением технологий DeepFake. Проведенный авторами анализ решений дал возможность создать новый способ выявления поддельного контента, который был запатентован авторами в Роспатенте. Предложенный авторами механизм выявления DeepFake имеет ряд существенных преимуществ по сравнению с решениями, описанными в обзоре, что позволяет рассчитывать на более эффективное обнаружение попыток обхода биометрических систем.
Литература
1. Tolosana R., Vera-Rodriguez R., Fierrez J., Morales A., Ortega-Garcia J. DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection. Information Fusion. 2020;64:131-148. doi: https://doi.org/10.1016/j.inffus.2020.06.014
2. Li Y., Chang M.-C., Lyu S. In Ictu Oculi: Exposing AI Created Fake Videos by Detecting Eye Blinking. In: 2018 IEEE International Workshop on Information Forensics and Security (WIFS). Hong Kong, China: IEEE Computer Society; 2018. p. 1-7. doi: https://doi.org/10.1109/WIFS.2018.8630787
3. Korshunov P., Marcel S. Deepfakes: a New Threat to Face Recognition? Assessment and Detection. arXiv:1812.08685. 2018. doi: https://doi.org/10.48550/arXiv.1812.08685
4. Rössler A., Cozzolino D., Verdoliva L., Riess C., Thies J., Niessner M. FaceForensics++: Learning to Detect Manipulated Facial Images. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE Computer Society; 2019. p. 1-11. doi: https://doi.org/10.1109/ICCV.2019.00009
5. Li Y., Yang X., Sun P., Qi H., Lyu S. Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA: IEEE Computer Society; 2020. p. 3204-3213. doi: https://doi.org/10.1109/CVPR42600.2020.00327
6. Dolhansky B., Howes R., Pflaum B., Baram N., Ferrer C.C. The Deepfake Detection Challenge (DFDC) Preview Dataset. arXiv:1910.08854. 2019. doi: https://doi.org/10.48550/arXiv.1910.08854
7. Matern F., Riess C., Stamminger M. Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations. In: 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW). Waikoloa, HI, USA: IEEE Computer Society; 2019. p. 83-92. doi: https://doi.org/10.1109/WACVW.2019.00020
8. Yang X., Li Y., Lyu S. Exposing Deep Fakes Using Inconsistent Head Poses. In: ICASSP 2019 ‒ 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Brighton, UK: IEEE Computer Society; 2019. p. 8261-8265. doi: https://doi.org/10.1109/ICASSP.2019.8683164
9. Agarwal S., Farid H. Protecting World Leaders Against Deep Fakes. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Long Beach, CA, USA; 2019. p. 1-8. Available at: https://farid.berkeley.edu/downloads/publications/cvpr19/cvpr19a.pdf (accessed 03.09.2022).
10. Jung T., Kim S., Kim K. DeepVision: Deepfakes Detection Using Human Eye Blinking Pattern. IEEE Access. 2020;8:83144-83154. doi: https://doi.org/10.1109/ACCESS.2020.2988660
11. Li Y., Lyu S. Exposing DeepFake Videos By Detecting Face Warping Artifacts. arXiv:1811.00656. 2019. doi: https://doi.org/10.48550/arXiv.1811.00656
12. Afchar D., Nozick V., Yamagishi J., Echizen I. MesoNet: a Compact Facial Video Forgery Detection Network. In: 2018 IEEE International Workshop on Information Forensics and Security (WIFS). Hong Kong, China: IEEE Computer Society; 2018. p. 1-7. doi: https://doi.org/10.1109/WIFS.2018.8630761
13. Zhou P., Han X., Morariu V.I., Davis L.S. Two-Stream Neural Networks for Tampered Face Detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Honolulu, HI, USA: IEEE Computer Society; 2017. p. 1831-1839. doi: https://doi.org/10.1109/CVPRW.2017.229
14. Nguyen H.Y., Fang F., Yamagishi J., Echizen I. Multi-task Learning for Detecting and Segmenting Manipulated Facial Images and Videos. In: 2019 IEEE 10th International Conference on Biometrics Theory, Applications and Systems (BTAS). Tampa, FL, USA: IEEE Computer Society; 2019. p. 1-8. doi: https://doi.org/10.1109/BTAS46853.2019.9185974
15. Nguyen H.H., Yamagishi J., Echizen I. Use of a Capsule Network to Detect Fake Images and Videos. arXiv:1910.12467. 2019. doi: https://doi.org/10.48550/arXiv.1910.12467
16. Dang H., Liu F., Stehouwer J., Liu X., Jain A. On the Detection of Digital Face Manipulation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA: IEEE Computer Society; 2020. p. 5780-5789. doi: https://doi.org/10.1109/CVPR42600.2020.00582
17. Wang Y., Dantcheva A. A Video is Worth More than 1000 Lies. Comparing 3DCNN Approaches for Detecting Deepfakes. In: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020). Buenos Aires, Argentina: IEEE Computer Society; 2020. p. 515-519. doi: https://doi.org/10.1109/FG47880.2020.00089
18. Güera D., Delp E. Deepfake Video Detection Using Recurrent Neural Networks. In: 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). Auckland, New Zealand: IEEE Computer Society; 2018. p. 1-6. doi: https://doi.org/10.1109/AVSS.2018.8639163
19. Sabir E., Cheng J., Jaiswal A., AbdAlmageed W., Masi I., Natarajan P. Recurrent Convolutional Strategies for Face Manipulation Detection in Videos. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops'2019). Long Beach, CA, USA: Computer Vision Foundation; 2019. p. 80-87. doi: https://doi.org/10.48550/arXiv.1905.00582
20. Tolosana R., Romero-Tapiador S., Fierrez J., Vera-Rodriguez R. DeepFakes Evolution: Analysis of Facial Regions and Fake Detection Performance. In: Del Bimbo A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science. Vol. 12665. Cham: Springer; 2021. p. 442-456. doi: https://doi.org/10.1007/978-3-030-68821-9_38
21. Wang R., Juefei-Xu F., Ma L., Xie X., Huang Y., Wang J., Liu Y. FakeSpotter: a simple yet robust baseline for spotting AI-synthesized fake faces. In: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI'20). Yokohama, Yokohama, Japan; 2021. Article number: 476. p. 3444-3451. Available at: https://dl.acm.org/doi/pdf/10.5555/3491440.3491916 (accessed 03.09.2022).
22. Nataraj L., Mohammed T.M., Manjunath B.S., Chandrasekaran S., Flenner A., Bappy J.H., Roy-Chowdhury A.K. Detecting GAN generated Fake Images using Co-occurrence Matrices. In: Proc. IS&T Int’l. Symp. on Electronic Imaging: Media Watermarking, Security, and Forensics. IS&T; 2019. p. 532. doi: https://doi.org/10.2352/ISSN.2470-1173.2019.5.MWSF-532
23. Bharati A., Singh R., Vatsa M., Bowyer K.W. Detecting Facial Retouching Using Supervised Deep Learning. IEEE Transactions on Information Forensics and Security. 2016;11(9):1903-1913. doi: https://doi.org/10.1109/TIFS.2016.2561898
24. Jain A., Singh R., Vatsa M. On Detecting GANs and Retouching based Synthetic Alterations. In: 2018 IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS). Redondo Beach, CA, USA: IEEE Computer Society; 2018. p. 1-7. doi: https://doi.org/10.1109/BTAS.2018.8698545
25. Tariq S., Lee S., Kim H., Shin Y., Woo S.S. Detecting Both Machine and Human Created Fake Face Images In the Wild. In: Proceedings of the 2nd International Workshop on Multimedia Privacy and Security (MPS'18). New York, NY, USA: Association for Computing Machinery; 2018. p. 81-87. doi: https://doi.org/10.1145/3267357.3267367
26. Wang S., Wang O., Owens A., Zhang R., Efros A. Detecting Photoshopped Faces by Scripting Photoshop. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE Computer Society; 2019. p. 10071-10080. doi: https://doi.org/10.1109/ICCV.2019.01017
27. Marra F., Saltori C., Boato G., Verdoliva L. Incremental learning for the detection and classification of GAN-generated images. In: 2019 IEEE International Workshop on Information Forensics and Security (WIFS). Delft, Netherlands: IEEE Computer Society; 2019. p. 1-6. doi: https://doi.org/10.1109/WIFS47025.2019.9035099
28. Zhang X., Karaman S., Chang S.-F. Detecting and Simulating Artifacts in GAN Fake Images. In: 2019 IEEE International Workshop on Information Forensics and Security (WIFS). Delft, Netherlands: IEEE Computer Society; 2019. p. 1-6. doi: https://doi.org/10.1109/WIFS47025.2019.9035107
29. Rathgeb C., Botaljov A., Stockhardt F., Isadskiy S., Debiasi L., Uhl A., Busch C. PRNU-based Detection of Facial Retouching. IET Biometrics. 2020;9(4):154-164. doi: https://doi.org/10.1049/iet-bmt.2019.0196
30. Amerini I., Galteri L., Caldelli R., Del Bimbo A. Deepfake Video Detection through Optical Flow Based CNN. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). Seoul, Korea (South): IEEE Computer Society; 2019. p. 1205-1207. doi: https://doi.org/10.1109/ICCVW.2019.00152
31. Peng B. et al. DFGC 2021: A DeepFake Game Competition. In: 2021 IEEE International Joint Conference on Biometrics (IJCB). Shenzhen, China: IEEE Computer Society; 2021. p. 1-8. doi: https://doi.org/10.1109/IJCB52358.2021.9484387
32. Peng B. et al. DFGC 2022: The Second DeepFake Game Competition. In: 2022 IEEE International Joint Conference on Biometrics (IJCB). Abu Dhabi, United Arab Emirates: IEEE Computer Society; 2022. p. 1-10. doi: https://doi.org/10.1109/IJCB54206.2022.10007991
2. Li Y., Chang M.-C., Lyu S. In Ictu Oculi: Exposing AI Created Fake Videos by Detecting Eye Blinking. In: 2018 IEEE International Workshop on Information Forensics and Security (WIFS). Hong Kong, China: IEEE Computer Society; 2018. p. 1-7. doi: https://doi.org/10.1109/WIFS.2018.8630787
3. Korshunov P., Marcel S. Deepfakes: a New Threat to Face Recognition? Assessment and Detection. arXiv:1812.08685. 2018. doi: https://doi.org/10.48550/arXiv.1812.08685
4. Rössler A., Cozzolino D., Verdoliva L., Riess C., Thies J., Niessner M. FaceForensics++: Learning to Detect Manipulated Facial Images. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE Computer Society; 2019. p. 1-11. doi: https://doi.org/10.1109/ICCV.2019.00009
5. Li Y., Yang X., Sun P., Qi H., Lyu S. Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA: IEEE Computer Society; 2020. p. 3204-3213. doi: https://doi.org/10.1109/CVPR42600.2020.00327
6. Dolhansky B., Howes R., Pflaum B., Baram N., Ferrer C.C. The Deepfake Detection Challenge (DFDC) Preview Dataset. arXiv:1910.08854. 2019. doi: https://doi.org/10.48550/arXiv.1910.08854
7. Matern F., Riess C., Stamminger M. Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations. In: 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW). Waikoloa, HI, USA: IEEE Computer Society; 2019. p. 83-92. doi: https://doi.org/10.1109/WACVW.2019.00020
8. Yang X., Li Y., Lyu S. Exposing Deep Fakes Using Inconsistent Head Poses. In: ICASSP 2019 ‒ 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Brighton, UK: IEEE Computer Society; 2019. p. 8261-8265. doi: https://doi.org/10.1109/ICASSP.2019.8683164
9. Agarwal S., Farid H. Protecting World Leaders Against Deep Fakes. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Long Beach, CA, USA; 2019. p. 1-8. Available at: https://farid.berkeley.edu/downloads/publications/cvpr19/cvpr19a.pdf (accessed 03.09.2022).
10. Jung T., Kim S., Kim K. DeepVision: Deepfakes Detection Using Human Eye Blinking Pattern. IEEE Access. 2020;8:83144-83154. doi: https://doi.org/10.1109/ACCESS.2020.2988660
11. Li Y., Lyu S. Exposing DeepFake Videos By Detecting Face Warping Artifacts. arXiv:1811.00656. 2019. doi: https://doi.org/10.48550/arXiv.1811.00656
12. Afchar D., Nozick V., Yamagishi J., Echizen I. MesoNet: a Compact Facial Video Forgery Detection Network. In: 2018 IEEE International Workshop on Information Forensics and Security (WIFS). Hong Kong, China: IEEE Computer Society; 2018. p. 1-7. doi: https://doi.org/10.1109/WIFS.2018.8630761
13. Zhou P., Han X., Morariu V.I., Davis L.S. Two-Stream Neural Networks for Tampered Face Detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Honolulu, HI, USA: IEEE Computer Society; 2017. p. 1831-1839. doi: https://doi.org/10.1109/CVPRW.2017.229
14. Nguyen H.Y., Fang F., Yamagishi J., Echizen I. Multi-task Learning for Detecting and Segmenting Manipulated Facial Images and Videos. In: 2019 IEEE 10th International Conference on Biometrics Theory, Applications and Systems (BTAS). Tampa, FL, USA: IEEE Computer Society; 2019. p. 1-8. doi: https://doi.org/10.1109/BTAS46853.2019.9185974
15. Nguyen H.H., Yamagishi J., Echizen I. Use of a Capsule Network to Detect Fake Images and Videos. arXiv:1910.12467. 2019. doi: https://doi.org/10.48550/arXiv.1910.12467
16. Dang H., Liu F., Stehouwer J., Liu X., Jain A. On the Detection of Digital Face Manipulation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA: IEEE Computer Society; 2020. p. 5780-5789. doi: https://doi.org/10.1109/CVPR42600.2020.00582
17. Wang Y., Dantcheva A. A Video is Worth More than 1000 Lies. Comparing 3DCNN Approaches for Detecting Deepfakes. In: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020). Buenos Aires, Argentina: IEEE Computer Society; 2020. p. 515-519. doi: https://doi.org/10.1109/FG47880.2020.00089
18. Güera D., Delp E. Deepfake Video Detection Using Recurrent Neural Networks. In: 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). Auckland, New Zealand: IEEE Computer Society; 2018. p. 1-6. doi: https://doi.org/10.1109/AVSS.2018.8639163
19. Sabir E., Cheng J., Jaiswal A., AbdAlmageed W., Masi I., Natarajan P. Recurrent Convolutional Strategies for Face Manipulation Detection in Videos. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops'2019). Long Beach, CA, USA: Computer Vision Foundation; 2019. p. 80-87. doi: https://doi.org/10.48550/arXiv.1905.00582
20. Tolosana R., Romero-Tapiador S., Fierrez J., Vera-Rodriguez R. DeepFakes Evolution: Analysis of Facial Regions and Fake Detection Performance. In: Del Bimbo A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science. Vol. 12665. Cham: Springer; 2021. p. 442-456. doi: https://doi.org/10.1007/978-3-030-68821-9_38
21. Wang R., Juefei-Xu F., Ma L., Xie X., Huang Y., Wang J., Liu Y. FakeSpotter: a simple yet robust baseline for spotting AI-synthesized fake faces. In: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI'20). Yokohama, Yokohama, Japan; 2021. Article number: 476. p. 3444-3451. Available at: https://dl.acm.org/doi/pdf/10.5555/3491440.3491916 (accessed 03.09.2022).
22. Nataraj L., Mohammed T.M., Manjunath B.S., Chandrasekaran S., Flenner A., Bappy J.H., Roy-Chowdhury A.K. Detecting GAN generated Fake Images using Co-occurrence Matrices. In: Proc. IS&T Int’l. Symp. on Electronic Imaging: Media Watermarking, Security, and Forensics. IS&T; 2019. p. 532. doi: https://doi.org/10.2352/ISSN.2470-1173.2019.5.MWSF-532
23. Bharati A., Singh R., Vatsa M., Bowyer K.W. Detecting Facial Retouching Using Supervised Deep Learning. IEEE Transactions on Information Forensics and Security. 2016;11(9):1903-1913. doi: https://doi.org/10.1109/TIFS.2016.2561898
24. Jain A., Singh R., Vatsa M. On Detecting GANs and Retouching based Synthetic Alterations. In: 2018 IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS). Redondo Beach, CA, USA: IEEE Computer Society; 2018. p. 1-7. doi: https://doi.org/10.1109/BTAS.2018.8698545
25. Tariq S., Lee S., Kim H., Shin Y., Woo S.S. Detecting Both Machine and Human Created Fake Face Images In the Wild. In: Proceedings of the 2nd International Workshop on Multimedia Privacy and Security (MPS'18). New York, NY, USA: Association for Computing Machinery; 2018. p. 81-87. doi: https://doi.org/10.1145/3267357.3267367
26. Wang S., Wang O., Owens A., Zhang R., Efros A. Detecting Photoshopped Faces by Scripting Photoshop. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE Computer Society; 2019. p. 10071-10080. doi: https://doi.org/10.1109/ICCV.2019.01017
27. Marra F., Saltori C., Boato G., Verdoliva L. Incremental learning for the detection and classification of GAN-generated images. In: 2019 IEEE International Workshop on Information Forensics and Security (WIFS). Delft, Netherlands: IEEE Computer Society; 2019. p. 1-6. doi: https://doi.org/10.1109/WIFS47025.2019.9035099
28. Zhang X., Karaman S., Chang S.-F. Detecting and Simulating Artifacts in GAN Fake Images. In: 2019 IEEE International Workshop on Information Forensics and Security (WIFS). Delft, Netherlands: IEEE Computer Society; 2019. p. 1-6. doi: https://doi.org/10.1109/WIFS47025.2019.9035107
29. Rathgeb C., Botaljov A., Stockhardt F., Isadskiy S., Debiasi L., Uhl A., Busch C. PRNU-based Detection of Facial Retouching. IET Biometrics. 2020;9(4):154-164. doi: https://doi.org/10.1049/iet-bmt.2019.0196
30. Amerini I., Galteri L., Caldelli R., Del Bimbo A. Deepfake Video Detection through Optical Flow Based CNN. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). Seoul, Korea (South): IEEE Computer Society; 2019. p. 1205-1207. doi: https://doi.org/10.1109/ICCVW.2019.00152
31. Peng B. et al. DFGC 2021: A DeepFake Game Competition. In: 2021 IEEE International Joint Conference on Biometrics (IJCB). Shenzhen, China: IEEE Computer Society; 2021. p. 1-8. doi: https://doi.org/10.1109/IJCB52358.2021.9484387
32. Peng B. et al. DFGC 2022: The Second DeepFake Game Competition. In: 2022 IEEE International Joint Conference on Biometrics (IJCB). Abu Dhabi, United Arab Emirates: IEEE Computer Society; 2022. p. 1-10. doi: https://doi.org/10.1109/IJCB54206.2022.10007991
Опубликована
2022-10-24
Как цитировать
BALASHOV, Aleksandr Victorovich et al.
Обзор технологий выявления модифицированного контента класса DeepFake.
Современные информационные технологии и ИТ-образование, [S.l.], v. 18, n. 3, p. 680-690, oct. 2022.
ISSN 2411-1473. Доступно на: <http://sitito.cs.msu.ru/index.php/SITITO/article/view/920>. Дата доступа: 04 may 2024
doi: https://doi.org/10.25559/SITITO.18.202203.680-690.
Раздел
Теоретические и прикладные аспекты кибербезопасности
Это произведение доступно по лицензии Creative Commons «Attribution» («Атрибуция») 4.0 Всемирная.
Редакционная политика журнала основывается на традиционных этических принципах российской научной периодики и строится с учетом этических норм работы редакторов и издателей, закрепленных в Кодексе поведения и руководящих принципах наилучшей практики для редактора журнала (Code of Conduct and Best Practice Guidelines for Journal Editors) и Кодексе поведения для издателя журнала (Code of Conduct for Journal Publishers), разработанных Комитетом по публикационной этике - Committee on Publication Ethics (COPE). В процессе издательской деятельности редколлегия журнала руководствуется международными правилами охраны авторского права, нормами действующего законодательства РФ, международными издательскими стандартами и обязательной ссылке на первоисточник.
Журнал позволяет авторам сохранять авторское право без ограничений. Журнал позволяет авторам сохранить права на публикацию без ограничений.
Издательская политика в области авторского права и архивирования определяются «зеленым цветом» в базе данных SHERPA/RoMEO.
Все статьи распространяются на условиях лицензии Creative Commons «Attribution» («Атрибуция») 4.0 Всемирная, которая позволяет другим использовать, распространять, дополнять эту работу с обязательной ссылкой на оригинальную работу и публикацию в этом журналe.
