Hyperopt list of values per hyperparameter

Question

I'm trying to use Hyperopt on a regression model such that one of its hyperparameters is defined per variable and needs to be passed as a list. For example, if I have a regression with 3 independent variables (excluding constant), I would pass hyperparameter = [x, y, z] (where x, y, z are floats).

The values of this hyperparameter have the same bounds regardless of which variable they are applied to. If this hyperparameter was applied to all variables, I could simply use hp.uniform('hyperparameter', a, b). What I want the search space to be instead is a cartesian product of hp.uniform('hyperparameter', a, b) of length n, where n is the number of variables in a regression (so, basically, itertools.product(hp.uniform('hyperparameter', a, b), repeat = n))

I'd like to know whether this is possible within Hyperopt. If not, any suggestions for an optimizer where this is possible are welcome.

Answer 1

As noted in my comment, I am not 100% sure what you are looking for, but here is an example of using hyperopt to optimize 3 variables combination:

import random

# define an objective function
def objective(args):
    v1 = args['v1']
    v2 = args['v2']
    v3 = args['v3']
    result = random.uniform(v2,v3)/v1
    return result

# define a search space
from hyperopt import hp

space = {
    'v1': hp.uniform('v1', 0.5,1.5),
    'v2': hp.uniform('v2', 0.5,1.5),
    'v3': hp.uniform('v3', 0.5,1.5),
}

# minimize the objective over the space
from hyperopt import fmin, tpe, space_eval
best = fmin(objective, space, algo=tpe.suggest, max_evals=100)

print(best)

they all have the same search space in this case (as I understand this was your problem definition). Hyperopt aims to minimize the objective function, so running this will end up with v2 and v3 near the minimum value, and v1 near the maximum value. Since this most generally minimizes the result of the objective function.

Answer 2

You could use this function to create the space:

def get_spaces(a, b, num_spaces=9):
    return_set = {}
    for set_num in range(9):
        name = str(set_num)
        return_set = {
                **return_set,
                **{name: hp.uniform(name, a, b)}
                }
                
    return return_set

Answer 3

I would first define my pre-combinatorial space as a dict. The keys are names. The values are a tuple.

from hyperopt import hp
space = {'foo': (hp.choice, (False, True)), 'bar': (hp.quniform, 1, 10, 1)}

Next, produce the required combinatorial variants using loops or itertools. Each name is kept unique using a suffix or prefix.

types = (1, 2)
space = {f'{name}_{type_}': args for type_ in types for name, args in space.items()}

>>> space 
{'foo_1': (<function hyperopt.pyll_utils.hp_choice(label, options)>,
  (False, True)),
 'bar_1': (<function hyperopt.pyll_utils.hp_quniform(label, *args, **kwargs)>,
  1, 10, 1),
 'foo_2': (<function hyperopt.pyll_utils.hp_choice(label, options)>,
  (False, True)),
 'bar_2': (<function hyperopt.pyll_utils.hp_quniform(label, *args, **kwargs)>,
  1, 10, 1)}

Finally, initialize and create the actual hyperopt space:

space = {name: fn(name, *args) for name, (fn, *args) in space.items()}
values = tuple(space.values())

>>> space
{'foo_1': <hyperopt.pyll.base.Apply at 0x7f291f45d4b0>,
 'bar_1': <hyperopt.pyll.base.Apply at 0x7f291f45d150>,
 'foo_2': <hyperopt.pyll.base.Apply at 0x7f291f45d420>,
 'bar_2': <hyperopt.pyll.base.Apply at 0x7f291f45d660>}

This was done with hyperopt 0.2.7. As a disclaimer, I strongly advise against using hyperopt because in my experience it has significantly poor performance relative to other optimizers.

Answer 4

Hi so I implemented this solution with optuna. The advantage of optuna is that it will create a hyperspace for all individual values, but optimizes this values in a more intelligent way and just uses one hyperparameter optimization. For example I optimized a neural network with the Batch-SIze, Learning-rate and Dropout-Rate:

The search space is much larger than the actual values being used. This safes a lot of time instead of an grid search.

The Pseudo-Code of the implementation is:

def function(trial): #trials is the parameter of optuna, which selects the next hyperparameter
    distribution = [0 , 1]
    a = trials.uniform("a": distribution) #this is a uniform distribution
    b = trials.uniform("a": distribution)

    return (a*b)-b
    #This above is the function which optuna tries to optimze/minimze

For more detailed source-Code visit Optuna. It saved a lot of time for me and it was a really good result.