Title: A Comparative Study of Hyperparameter Optimization Techniques for Deep Learning
Abstract: Algorithms for deep learning (DL) have been widely employed in a variety of applications and fields. The hyperparameters of a deep learning model must be optimized to match different challenges. For deep learning models, choosing the optimum hyperparameter configuration has a direct influence on the model's performance. It typically involves a thorough understanding of deep learning algorithms and their hyperparameter optimization (HPO) techniques. Although there are various automatic optimization approaches available, each has its own set of advantages and disadvantages when applied to different datasets and architectures. In this paper, we analyzed which algorithm takes the longest optimization time to optimize an architecture and whether the performance of HPO algorithms is consistent across different datasets and architectures. We selected VGG16 and ResNet50 architectures, CIFAR10 and Intel Image Classification Dataset, as well as Grid search (GS), Genetic algorithm (GA), Bayesian optimization (BO), Random search (RS), Hyperband (HB) and Particle swarm optimization (PSO) HPO algorithms for comparison. Due to the lack of pattern, it is challenging to determine which approach obtains the best performance on different datasets and architecture. The results show that all of the algorithms have similar results in terms of optimization time. This research is expected to aid DL users, developers, data analysts, and researchers in their attempts to use and adapt DL models utilizing appropriate HPO methodologies and frameworks. It will also help to better understand the challenges that currently exist in the HPO field, allowing future research into HPO and DL applications to move forward.
Publication Year: 2022
Publication Date: 2022-01-01
Language: en
Type: book-chapter
Indexed In: ['crossref']
Access and Citation
Cited By Count: 5
AI Researcher Chatbot
Get quick answers to your questions about the article from our AI researcher chatbot