Please use this identifier to cite or link to this item: https://hdl.handle.net/10419/315188 
Year of Publication: 
2024
Citation: 
[Journal:] AStA Advances in Statistical Analysis [ISSN:] 1863-818X [Volume:] 108 [Issue:] 2 [Publisher:] Springer [Place:] Berlin, Heidelberg [Year:] 2024 [Pages:] 441-460
Publisher: 
Springer, Berlin, Heidelberg
Abstract: 
Hyperparameter tuning is one of the most time-consuming parts in machine learning. Despite the existence of modern optimization algorithms that minimize the number of evaluations needed, evaluations of a single setting may still be expensive. Usually a resampling technique is used, where the machine learning method has to be fitted a fixed number of k times on different training datasets. The respective mean performance of the k fits is then used as performance estimator. Many hyperparameter settings could be discarded after less than k resampling iterations if they are clearly inferior to high-performing settings. However, resampling is often performed until the very end, wasting a lot of computational effort. To this end, we propose the sequential random search (SQRS) which extends the regular random search algorithm by a sequential testing procedure aimed at detecting and eliminating inferior parameter configurations early. We compared our SQRS with regular random search using multiple publicly available regression and classification datasets. Our simulation study showed that the SQRS is able to find similarly well-performing parameter settings while requiring noticeably fewer evaluations. Our results underscore the potential for integrating sequential tests into hyperparameter tuning.
Subjects: 
Machine learning
Hyperparameter tuning
Sequential testing
Persistent Identifier of the first edition: 
Creative Commons License: 
cc-by Logo
Document Type: 
Article
Document Version: 
Published Version

Files in This Item:
File
Size





Items in EconStor are protected by copyright, with all rights reserved, unless otherwise indicated.