Hyperparameter Optimization of CNN for Map Building
Abstract
This article describes an approach for solving the task of finding hyperparameters of an artificial neural network, which is used for making a 2D land map. The main goal of research was an analysis of methods for finding hyperparameters and creating a better method for solving this task, which would be based on existing methods.
We considered on various hyperparameters such as velocity of training, coefficient of regularization, size of batch, probability of drop out, shifting, used for batch normalization. Among existing methods for finding hyperparameters we considered on the random search method, searching by grid, the Bayesian optimization, the evolution algorithm, the optimization, based on gradients, and the spectral method. As a result, we created a new method for finding hyperparameters which showed a better result in most of the use cases, which we have (mostly for middle European part of Russia).
The main idea of the method for finding hyperparameters is consisted in an approach for optimization of the quality function with a simple condition for lower and upper limits and a demand that the value of the function needed to be an integer number. This task may be solved with a simple genetic algorithm.
Using the optimization algorithm without evaluating derivatives gives decreasing time complexity of the algorithm without losing quality of the algorithm. In many cases the quality of result was better than results of existing methods.
References
[2] Hutter F., Hoos H.H., Leyton-Brown K. Sequential Model-Based Optimization for General Algorithm Configuration. In: Coello C.A.C. (ed.) Learning and Intelligent Optimization. LION 2011. Lecture Notes in Computer Science. 2011; 6683:507-523. Springer, Berlin, Heidelberg. (In Eng.) DOI: https://doi.org/10.1007/978-3-642-25566-3_40
[3] Thornton C., Hutter F., Hoos H.H., Leyton-Brown K. Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms. In: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD'13). Association for Computing Machinery, New York, NY, USA; 2013. p. 847-855. (In Eng.) DOI: https://doi.org/10.1145/2487575.2487629
[4] Claesen M., De Moor B. Hyperparameter Search in Machine Learning. arXiv:1502.02127. 2015. (In Eng.)
[5] Larsen J., Hansen L.K., Svarer S., Ohlsson M. Design and regularization of neural networks: the optimal use of a validation set. In: Neural Networks for Signal Processing VI. Proceedings of the 1996 IEEE Signal Processing Society Workshop, Kyoto, Japan; 1996. p. 62-71. (In Eng.) DOI: https://doi.org/10.1109/NNSP.1996.548336
[6] Bergstra J., Bardenet R., Bengio Y., Kégl B. Algorithms for hyper-parameter optimization. In: Proceedings of the 24th International Conference on Neural Information Processing Systems (NIPS'11). Curran Associates Inc., Red Hook, NY, USA; 2011. p. 2546-2554. (In Eng.)
[7] Hazan E., Klivans A., Yuan Y. Hyperparameter Optimization: A Spectral Approach. arXiv:1706.00764. 2018. (In Eng.)
[8] Feurer M., Springenberg J.T., Hutter F. Initializing bayesian hyperparameter optimization via meta-learning. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI'15). AAAI Press; 2015. p. 1128-1135. (In Eng.)
[9] Mendoza H., Klein A., Feurer M., Springenberg J.T., Urban M., Burkart M. Towards Automatically-Tuned Deep Neural Networks. In: Hutter F., Kotthoff L., Vanschoren J. (ed.) Automated Machine Learning. The Springer Series on Challenges in Machine Learning. Springer, Cham; 2019. p. 135-149. (In Eng.) DOI: https://doi.org/10.1007/978-3-030-05318-5_7
[10] Olson R.S., Bartley N., Urbanowicz R.J., Moore J.H. Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science. In: Proceedings of the Genetic and Evolutionary Computation Conference 2016 (GECCO'16). Association for Computing Machinery, New York, NY, USA; 2016. p. 485-492. (In Eng.) DOI: https://doi.org/10.1145/2908812.2908918
[11] Feurer M., Klein A., Eggensperger K., Springenberg J.T., Blum M., Hutter F. Efficient and Robust Automated Machine Learning. In: Cortes C., Lawrence N., Lee D., Sugiyama M., Garnett R. Advances in Neural Information Processing Systems. 2015; 28:2962-2970. Curran Associates, Inc. Available at: https://proceedings.neurips.cc/paper/2015/file/11d0e6287202fced83f79975ec59a3a6-Paper.pdf (accessed 14.07.2020). (In Eng.)
[12] Conn A.R., Scheinberg K., Vicente L.N. Introduction to Derivative-Free Optimization. Society for Industrial and Applied Mathematics; 2009. (In Eng.) DOI: https://doi.org/10.1137/1.9780898718768
[13] Gutmann H.-M. A Radial Basis Function Method for Global Optimization. Journal of Global Optimization. 2001; 19(3):201-227. (In Eng.) DOI: https://doi.org/10.1023/A:1011255519438
[14] Burkov E., Lempitsky V. Deep neural networks with box convolutions. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS'18). Curran Associates Inc., Red Hook, NY, USA; 2018. p. 6214-6224. (In Eng.)
[15] Long J., Shelhamer E., Darrell T. Fully convolutional networks for semantic segmentation. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA; 2015. p. 3431-3440. (In Eng.) DOI: https://doi.org/10.1109/CVPR.2015.7298965
[16] Yu F., Koltun V. Multi-Scale Context Aggregation by Dilated Convolutions. arXiv:1511.07122. 2016. (In Eng.)
[17] Chen L., Papandreou G., Kokkinos I., Murphy K., Yuille A.L. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2018; 40(4):834-848. (In Eng.) DOI: https://doi.org/10.1109/TPAMI.2017.2699184
[18] Jégou S., Drozdzal M., Vazquez D., Romero A., Bengio Y. The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Honolulu, HI; 2017. p. 1175-1183. (In Eng.) DOI: https://doi.org/10.1109/CVPRW.2017.156
[19] Ronneberger O., Fischer P., Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Navab N., Hornegger J., Wells W., Frangi A. (ed.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. MICCAI 2015. Lecture Notes in Computer Science. 2015; 9351:234-241. Springer, Cham. (In Eng.) DOI: https://doi.org/10.1007/978-3-319-24574-4_28
[20] Badrinarayanan V., Kendall A., Cipolla R. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2017; 39(12):2481-2495. (In Eng.) DOI: https://doi.org/10.1109/TPAMI.2016.2644615
[21] Paszke A., Chaurasia A., Kim S., Culurciello E. ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation. arXiv:1606.02147. 2016. (In Eng.)
[22] Akinina A.V., Nikiforov M.B., Savin A.V. Multiscale image segmentation using normalized cuts in image recognition on satellite images. In: 2018 7th Mediterranean Conference on Embedded Computing (MECO). Budva, 2018. p. 1-3. (In Eng.) DOI: https://doi.org/10.1109/MECO.2018.8406066
[23] Akinin M.V., Akinina A.V., Sokolov A.V., Tarasov A.S. Application of EM algorithm in problems of pattern recognition on satellite images. In: 2017 6th Mediterranean Conference on Embedded Computing (MECO). Bar, 2017. p. 1-4. (In Eng.) DOI: https://doi.org/10.1109/MECO.2017.7977190
[24] Nurshazlyn Mohd Aszemi, Dominic P.D.D. Hyperparameter Optimization in Convolutional Neural Network using Genetic Algorithms. International Journal of Advanced Computer Science and Applications (IJACSA). 2019; 10(6):269-278. (In Eng.) DOI: http://dx.doi.org/10.14569/IJACSA.2019.0100638
[25] Wistuba M., Schilling N., Schmidt-Thieme L. M. Hyperparameter Optimization Machines. In: 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA). Montreal, QC; 2016. p. 41-50. (In Eng.) DOI: https://doi.org/10.1109/DSAA.2016.12
This work is licensed under a Creative Commons Attribution 4.0 International License.
Publication policy of the journal is based on traditional ethical principles of the Russian scientific periodicals and is built in terms of ethical norms of editors and publishers work stated in Code of Conduct and Best Practice Guidelines for Journal Editors and Code of Conduct for Journal Publishers, developed by the Committee on Publication Ethics (COPE). In the course of publishing editorial board of the journal is led by international rules for copyright protection, statutory regulations of the Russian Federation as well as international standards of publishing.
Authors publishing articles in this journal agree to the following: They retain copyright and grant the journal right of first publication of the work, which is automatically licensed under the Creative Commons Attribution License (CC BY license). Users can use, reuse and build upon the material published in this journal provided that such uses are fully attributed.
