Comparative Analysis of Hyperparameter Optimization Using Optuna and Hyperopt for Convolutional Neural Networks

Abstract

The process of training a neural network model is controlled by selecting optimal hyperparameters that have a significant impact on its quality and performance. This impact has been proven both theoretically and empirically by many studies. The task is labor-intensive if manual search is chosen. The most common enumeration methods include grid search, random search, and sequential model-based optimization, in which the procedure for estimating the objective function is quite fast. But all these methods create problems in applications with convolutional neural networks, where the parameter space is so large that even a shortened enumeration of their possible combinations is expensive in terms of the required computing power. An alternative is automatic hyperparameter tuning tools compatible with machine learning frameworks and using high-speed probabilistic estimates of the objective function with additional mechanisms. The paper compares the performance of training convolutional neural networks using exactly such tools – Python libraries for automatic hyperparameter optimization – Hyperopt and Optuna. Their comparative analysis is performed for image classification applications. It is shown that the use of libraries makes it possible to overcome the most important problems of hyperparameter optimization of such applications, including the large dimensionality of their search space and sensitivity to the choice of adjustable hyperparameters. Comparing the performance of the model in terms of accuracy and training loss, it can be concluded that both hyperparameter optimization methods are effective, providing training accuracy of more than 99% and a loss level of less than 0.03. The implementation of the optimization algorithm in the Optuna package turned out to be slightly better than in Hyperopt, providing high performance indicators on both training and test data.