Оцінка методу прунінгу SNIP на сучасній моделі детекції обличчя

dc.citation.epage22
dc.citation.issue1
dc.citation.journalTitleОбчислювальні проблеми електротехніки
dc.citation.spage18
dc.contributor.affiliationNational Technical University of Ukraine “Igor Sikorsky Kyiv Polytechnic Institute”
dc.contributor.authorМельниченко, Артем
dc.contributor.authorШалденко, Олексій
dc.contributor.authorMelnychenko, Artem
dc.contributor.authorShaldenko, Oleksii
dc.coverage.placenameЛьвів
dc.coverage.placenameLviv
dc.date.accessioned2024-04-11T09:15:11Z
dc.date.available2024-04-11T09:15:11Z
dc.date.created2023-02-28
dc.date.issued2023-02-28
dc.description.abstractЗавдяки швидкому розвитку машинного навчання та як наслідок глибокого навчання глибокі нейронні мережі досягли помітних результатів у різних сферах. Однак зі збільшенням точності навчених моделей нові архітектури нейронних мереж створюють нові виклики, оскільки потребують великої кількості обчислювальних потужностей для навчання та подальшого використання. Ця стаття має на меті переглянути підходи до зменшення обчислювальних потужностей та часу, потрібних для навчання нейронних мереж, оцінити та вдосконалити один із таких методів на моделі для детекції облич. Результати показали, що пропонований метод може усунути 69 % параметрів, втрачаючи лише 1,4 % точності. Його можна додатково покращити, зменшивши втрату точності до 0,7 %, вилучивши контекстні модулі мережі із методу.
dc.description.abstractWith rapid development of machine learning and subsequently deep learning, deep neural networks achieved remarkable results in solving various tasks. However, with increasing the accuracy of trained models, new architectures of neural networks present new challenges as they require significant amount of computing power for training and inference. This paper aims to review existing approaches to reducing computational power and training time of the neural network, evaluate and improve one of existing pruning methods for a face detection model. Obtained results show that the presented method can eliminate 69% of parameters while accuracy being declined only by 1.4%, which can be further improved to 0.7% by excluding context network modules from the pruning method.
dc.format.extent18-22
dc.format.pages5
dc.identifier.citationМельниченко А. Оцінка методу прунінгу SNIP на сучасній моделі детекції обличчя / Артем Мельниченко, Олексій Шалденко // Обчислювальні проблеми електротехніки. — Львів : Видавництво Львівської політехніки, 2023. — Том 13. — № 1. — С. 18–22.
dc.identifier.citationenMelnychenko A. Evaluating Snip Pruning Method on the State-of-the-Art Face Detection Model / Artem Melnychenko, Oleksii Shaldenko // Computational Problems of Electrical Engineering. — Lviv : Lviv Politechnic Publishing House, 2023. — Vol 13. — No 1. — P. 18–22.
dc.identifier.doidoi.org/10.23939/jcpee2023.01.018
dc.identifier.issn2224-0977
dc.identifier.urihttps://ena.lpnu.ua/handle/ntb/61717
dc.language.isouk
dc.publisherВидавництво Львівської політехніки
dc.publisherLviv Politechnic Publishing House
dc.relation.ispartofОбчислювальні проблеми електротехніки, 1 (13), 2023
dc.relation.ispartofComputational Problems of Electrical Engineering, 1 (13), 2023
dc.relation.references[1] G. Hinton et al., “Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups”, in IEEE Signal Processing Magazine, Vol. 29, No. 6, pp. 82–97, Nov. 2012.
dc.relation.references[2] K. He, X. Zhang, S. Ren and J. Sun, “Deep Residual Learning for Image Recognition”, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 770-778.
dc.relation.references[3] K. Zhang, Z. Zhang, Z. Li and Y. Qiao, “Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks”, in IEEE Signal Processing Letters, Vol. 23, No. 10, pp. 1499–1503, Oct. 2016.
dc.relation.references[4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 1 (Long and Short Papers), pp. 4171–4186, Minneapolis, Minnesota.
dc.relation.references[5] Brown, T et al. “Language models are few-shot learners”, Advances in neural information processing systems, 33, pp. 1877–1901.
dc.relation.references[6] Schwartz, Roy, Jesse Dodge, Noah Smith and Oren Etzioni. “Green AI.” Communications of the ACM 63, 2019, pp. 54–63.
dc.relation.references[7] Ben Taylor, Vicent Sanz Marco, Willy Wolff, Yehia Elkhatib, and Zheng Wang. “Adaptive deep learning model selection on embedded systems”, in Proc. 19th ACM SIGPLAN/SIGBED International Conference on Languages, Compilers, and Tools for Embedded Systems, New York, USA, pp. 31–43, 2018.
dc.relation.references[8] Han, Song, Huizi Mao and William J. Dally. “Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding”, arXiv: Computer Vision and Pattern Recognition, 2015.
dc.relation.references[9] S. Teerapittayanon, B. McDanel and H. T. Kung, “Distributed Deep Neural Networks Over the Cloud, the Edge and End Devices”, 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), Atlanta, USA, pp. 328–339, 2017.
dc.relation.references[10] Misha Denil, Babak Shakibi, Laurent Dinh, Marc'Aurelio Ranzato, and Nando de Freitas. “Predicting parameters in deep learning”, in Proc. 26th International Conference on Neural Information Processing Systems, vol. 2 (NIPS'13). Curran Associates Inc., Red Hook, USA, pp. 2148–2156, 2013.
dc.relation.references[11] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. “Speeding up Convolutional Neural Networks with Low Rank Expansions”, In Proceedings of the British Machine Vision Conference. BMVA Press, September 2014.
dc.relation.references[12] Novikov, A., Podoprikhin, D., Osokin, A. and Vetrov, D.P.. “Tensorizing neural networks”, Advances in neural information processing systems, 28, 2015.
dc.relation.references[13] Song Han, Jeff Pool, John Tran, and William J. Dally. “Learning both weights and connections for efficient neural networks”, In Proceedings of the 28th International Conference on Neural Information Processing Systems, Vol. 1 (NIPS'15), MIT Press, Cambridge, USA, pp. 1135–1143, 2015.
dc.relation.references[14] S. Park, J. Lee, S. Mo and J. Shin, , “Lookahead: A far-sighted alternative of magnitude-based pruning”, arXiv preprint arXiv:2002.04809, 2020.
dc.relation.references[15] B. Hassibi and D. G. Stork. “Second order derivatives for network pruning: optimal brain surgeon”, in Proc. 5th International Conference on Neural Information Processing Systems (NIPS'92), Morgan Kaufmann Publishers Inc., San Francisco, USA, pp. 164–171, 1992.
dc.relation.references[16] J. Frankle, and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks”, in Proc. 7th International Conference on Learning Representations, New Orleans, USA, May 6–9, 2019.
dc.relation.references[17] N. Lee, T. Ajanthan and P.H. Torr, “The lottery ticket hypothesis: Finding sparse, trainable neural networks”, in Proc. 7th International Conference on Learning Representations, New Orleans, USA, May 6–9, 2019.
dc.relation.references[18] J. Deng, J. Guo, E. Ververas, I. Kotsia, Stefanos Zafeiriou, “RetinaFace: Single-stage Dense Face Localisation in the Wild“,in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5203–5212, 2020.
dc.relation.referencesen[1] G. Hinton et al., "Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups", in IEEE Signal Processing Magazine, Vol. 29, No. 6, pp. 82–97, Nov. 2012.
dc.relation.referencesen[2] K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition", 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 770-778.
dc.relation.referencesen[3] K. Zhang, Z. Zhang, Z. Li and Y. Qiao, "Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks", in IEEE Signal Processing Letters, Vol. 23, No. 10, pp. 1499–1503, Oct. 2016.
dc.relation.referencesen[4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 1 (Long and Short Papers), pp. 4171–4186, Minneapolis, Minnesota.
dc.relation.referencesen[5] Brown, T et al. "Language models are few-shot learners", Advances in neural information processing systems, 33, pp. 1877–1901.
dc.relation.referencesen[6] Schwartz, Roy, Jesse Dodge, Noah Smith and Oren Etzioni. "Green AI." Communications of the ACM 63, 2019, pp. 54–63.
dc.relation.referencesen[7] Ben Taylor, Vicent Sanz Marco, Willy Wolff, Yehia Elkhatib, and Zheng Wang. "Adaptive deep learning model selection on embedded systems", in Proc. 19th ACM SIGPLAN/SIGBED International Conference on Languages, Compilers, and Tools for Embedded Systems, New York, USA, pp. 31–43, 2018.
dc.relation.referencesen[8] Han, Song, Huizi Mao and William J. Dally. "Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding", arXiv: Computer Vision and Pattern Recognition, 2015.
dc.relation.referencesen[9] S. Teerapittayanon, B. McDanel and H. T. Kung, "Distributed Deep Neural Networks Over the Cloud, the Edge and End Devices", 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), Atlanta, USA, pp. 328–339, 2017.
dc.relation.referencesen[10] Misha Denil, Babak Shakibi, Laurent Dinh, Marc'Aurelio Ranzato, and Nando de Freitas. "Predicting parameters in deep learning", in Proc. 26th International Conference on Neural Information Processing Systems, vol. 2 (NIPS'13). Curran Associates Inc., Red Hook, USA, pp. 2148–2156, 2013.
dc.relation.referencesen[11] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. "Speeding up Convolutional Neural Networks with Low Rank Expansions", In Proceedings of the British Machine Vision Conference. BMVA Press, September 2014.
dc.relation.referencesen[12] Novikov, A., Podoprikhin, D., Osokin, A. and Vetrov, D.P.. "Tensorizing neural networks", Advances in neural information processing systems, 28, 2015.
dc.relation.referencesen[13] Song Han, Jeff Pool, John Tran, and William J. Dally. "Learning both weights and connections for efficient neural networks", In Proceedings of the 28th International Conference on Neural Information Processing Systems, Vol. 1 (NIPS'15), MIT Press, Cambridge, USA, pp. 1135–1143, 2015.
dc.relation.referencesen[14] S. Park, J. Lee, S. Mo and J. Shin, , "Lookahead: A far-sighted alternative of magnitude-based pruning", arXiv preprint arXiv:2002.04809, 2020.
dc.relation.referencesen[15] B. Hassibi and D. G. Stork. "Second order derivatives for network pruning: optimal brain surgeon", in Proc. 5th International Conference on Neural Information Processing Systems (NIPS'92), Morgan Kaufmann Publishers Inc., San Francisco, USA, pp. 164–171, 1992.
dc.relation.referencesen[16] J. Frankle, and M. Carbin, "The lottery ticket hypothesis: Finding sparse, trainable neural networks", in Proc. 7th International Conference on Learning Representations, New Orleans, USA, May 6–9, 2019.
dc.relation.referencesen[17] N. Lee, T. Ajanthan and P.H. Torr, "The lottery ticket hypothesis: Finding sparse, trainable neural networks", in Proc. 7th International Conference on Learning Representations, New Orleans, USA, May 6–9, 2019.
dc.relation.referencesen[18] J. Deng, J. Guo, E. Ververas, I. Kotsia, Stefanos Zafeiriou, "RetinaFace: Single-stage Dense Face Localisation in the Wild",in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5203–5212, 2020.
dc.rights.holder© Національний університет “Львівська політехніка”, 2023
dc.subjectpruning
dc.subjectdeep neural networks
dc.subjectinference
dc.subjectoptimization
dc.subjectface detection
dc.titleОцінка методу прунінгу SNIP на сучасній моделі детекції обличчя
dc.title.alternativeEvaluating Snip Pruning Method on the State-of-the-Art Face Detection Model
dc.typeArticle

Files

Original bundle

Now showing 1 - 2 of 2
Thumbnail Image
Name:
2023v13n1_Melnychenko_A-Evaluating_Snip_Pruning_18-22.pdf
Size:
1.05 MB
Format:
Adobe Portable Document Format
Thumbnail Image
Name:
2023v13n1_Melnychenko_A-Evaluating_Snip_Pruning_18-22__COVER.png
Size:
552.36 KB
Format:
Portable Network Graphics

License bundle

Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.83 KB
Format:
Plain Text
Description: