Algorithms and Datasets

Benchmarks Overview

To run these algorithms and datasets with hyperparameter optimizers you need to install

  1. the HPOlib software from here
  2. the benchmark data: An algorithm and depending on the benchmark a wrapper and/or data

Then the benchmarks can easily be used, as described here; Our software allows to integrate your own benchmarks as well. Here is the HowTo.

NOTE: For all bechmarks crossvalidation is possible, but not extra listed. Although possible, it obviously makes no sense to do crossvalidation on functions like Branin and pre-computed results like the LDA ongrid. Whether it makes sense to do so is indicated in the column CV.

Available Benchmarks
Algorithm # hyperparams(condition.) contin./discr. Dataset Size(Train/Valid/Test) runtime programming language CV
Branin 2(-) 2/- - - < 1s Python no
RKHS 1(-) 1/- - - < 1s Python no
Camelback function 2(-) 2/- - - < 1s Ruby no
Hartmann 6d 6(-) 6/- - - < 1s Python no
Michalewicz 10(-) 10/- - - < 1s Python no
LDA ongrid 3(-) -/3 wikipedia articles - <1s Python no
SVM ongrid 3(-) -/3 UniPROBE - <1s Python no
Logistic Regression 4(-) 4/- MNIST 50k/10k/10k <1m (Intel Xeon E5-2650 v2; [email protected]) Python yes
hp-nnet 14(4) 7/7 MRBI
convex
10k/2k/50k
6.5k/1.5k/50k
~25m (GPU, NVIDIA Tesla M2070)
~6m (GPU, NVIDIA Tesla M2070)
Python yes
hp-dbnet 38(29) 19/17 MRBI
convex
10k/2k/50k
6.5k/1.5k/50k
~15m (GPU, Gefore GTX780)
~10m (GPU, Gefore GTX780)
Python yes
autoweka 786(784) 296/490 convex 6.5k/1.5k/50k ~15m Python/Java yes

Description

Branin, RKHS, Hartmann 6d, Michalewicz and Camelback Function

This benchmark already comes with the basic HPOlib bundle.

Dependencies: None
Recommended: None

Branin, RKHS, Camelback, Michalewicz and the Hartmann 6d function are five simple test functions, which are easy and cheap to evaluate. More test functions can be found here.

Branin has three global minima at (-pi, 12.275), (pi, 2.275), (9.42478, 2.475) where f(x)=0.397887.
RKHS has single global minima at x=0.89235 where f(x)=5.73839.
Camelback has two global minima at (0.0898, -0.7126) and (-0.0898, 0.7126) where f(x) = -1.0316
Hartmann 6d is more difficult with 6 local minima and one global optimum at (0.20169, 0.150011, 0.476874, 0.275332, 0.311652, 0.6573) where f(x)=3.32237.
Michalewicz is usually evaluated on the hypercube xi∈ [0, pi], for all i = 1, …, d. For d=10 its global minima value is f(x) = -9.66015.

LDA ongrid/SVM ongrid

This benchmark already comes with the basic HPOlib bundle.

Dependencies: None
Recommended: None

Online Latent Dirichlet Allocation (LDA) is a very expensive algorithm to evaluate. To make this less time consuming, a 6x6x8 grid of hyperparameter configurations resulting in 288 data points was preevaluated. This grid forms the search space.

Same holds for the Support Vector Machine task, which has 1400 evaluated configurations.

The Online LDA code is written by Hoffman et. al. and the procedure is explained in Online Learning for Latent Dirichlet Allocation. Latent Structured Support Vector Machine code is written by Kevin Mill et. al. and explained in the paper Max-Margin Min-Entropy Models. The grid search was performed by Jasper Snoek and previously used in Practical Bayesian Optimization of Machine Learning Algorithms.

Logistic Regression

Dependencies: theano, scikit-data
Recommended: CUDA

NOTE: scikit-data downloads the dataset from the internet when using the benchmark for the first time.
NOTE: This benchmarks can use a gpu, but this feature is switched off to run it off-the-shelf. To use a gpu you need to change the THEANO flags in config.cfg. See the HowTo for changing to gpu and for further information about the THEANO configuration here
NOTE: In order to run the benchmark you must adjust the paths in the config files.

You can download this benchmark by clicking here or running this command from a shell:

wget http://www.automl.org/logreg.tar.gz
tar -xf logistic.tar.gz

This benchmark performs a logistic regression to classifiy the popular MNIST dataset. The implementation is Theano based, so that a GPU can be used. The software is written by Jasper Snoek and was first used in the paper Practical Bayesian Optimization of Machine Learning Algorithms.

NOTE: This benchmark comes with the version of hyperopt-nnet which we used for our experiments. There might be a newer version with improvements.

HP-NNet and HP-DBNet

Dependencies: theano, scikit-data
Recommended: CUDA

NOTE: This benchmark comes with the version of hyperopt-nnet which we used for our experiments. There might be a newer version with improvements.
NOTE: scikit-data downloads the dataset from the internet when using the benchmark for the first time.
NOTE: In order to run the benchmark you must adjust the paths in the config files.

You can download this benchmark by clicking here or running this command from a shell:

The HP-Nnet (HP-DBNet) is a Theano based implementation of a (deep) neural network. It can be run on a CPU, but is drastically faster on a GPU (please follow the theano flags instructions of the logistic regression example). Both of them are written by James Bergstra and were used in the papers Random Search for Hyper-Parameter Optimization and Algorithms for Hyper-Parameter Optimization.

AutoWEKA

NOTE: AutoWEKA is not yet available for download!

Fork me on GitHub