Optimizer

Optimizer

The Optimizer class. Used to perform a search for minimum or maximum loss for a given set of parameters. Also used for a grid or random sweep of parameter space.

Note that any keyword argument not in the following will be passed onto the Experiment constructor. For example, you can pass project_name and logging arguments by listing them here.

Args:

  • config: optional, if COMET_OPTIMIZER_ID is configured, otherwise is either a config dictionary, optimizer id, or a config filename.
  • trials: int (optional, default 1) number of trials per parameter set to test.
  • verbose: boolean (optional, default 1) verbosity level where 0 means no output, and 1 (or greater) means to show more detail.
  • experiment_class: string or callable (optional, default None), class to use (for example, OfflineExperiment).

Examples:

# Assume COMET_OPTIMIZER_ID is configured:
>>> opt = Optimizer()

# An optimizer config dictionary:
>>> opt = Optimizer({"algorithm": "bayes", ...})

# An optimizer id:
>>> opt = Optimizer("73745463463")

# A filename to a optimizer config file:
>>> opt = Optimizer("/tmp/mlhacker/optimizer.config")

To pass arguments to the Experiment constructor, pass them into the opt.get_experiments() call, like so:

>>> opt = Optimizer("/tmp/mlhacker/optimizer.config")
>>> for experiment in opt.get_experiments(
...     project_name="my-project",
...     auto_metric_logging=False,
... ):
...     loss = fit(model)
...     experiment.log_metric("loss", loss")
...     experiment.end()

Optimizer.__init__

__init__(self, config=None, trials=None, verbose=1,
    experiment_class="Experiment", api_key=None, **kwargs)

The Optimizer constructor.

Args:

  • config: (optional, if COMET_OPTIMIZER_ID is configured). Can an optimizer config id, an optimizer config dictionary, or an optimizer config filename.
  • trials: int (optional, default 1) number of trials per parameter value set
  • verbose: int (optional, default 1) level of details to show; 0 means quiet
  • experiment_class: string (optional). Supported values are "Experiment" (the default) to use online Experiments or "OfflineExperiment" to use offline Experiments. It can also be a callable (a function or a method) that returns an instance of Experiment, OfflineExperiment, ExistingExperiment or ExistingOfflineExperiment.

See above for examples.

Optimizer.end

end(self, experiment)

Optimizer.end() is called at end of experiment. Usually, you would not call this manually, as it is called directly when the experiment ends.

Optimizer.get_experiments

get_experiments(self, **kwargs)

Optimizer.get_experiments() will iterate over all possible experiments for this sweep or search, n at a time. All experiments will have a unique set of parameter values (unless performing multiple trials per parameter set).

Example:

>>> for experiment in optimizer.get_experiments():
...     loss = fit(x, y)
...     experiment.log_metric("loss", loss)

Optimizer.get_id

get_id(self)

Get the id of this optimizer, with Comet config variable.

Example:

>>> opt.get_id()
COMET_OPTIMIZER_ID=87463746374647364

Optimizer.get_parameters

get_parameters(self)

Optimizer.get_parameters() will iterate over all possible parameters for this sweep or search. All parameters combinations will be emitted once (unless performing multiple trials per parameter set).

Example:

>>> for parameters in optimizer.get_parameters():
...     experiment = comet_ml.Experiment()
...     loss = fit(x, y)
...     experiment.log_metric("loss", loss)

Optimizer.next

next(self, **kwargs)

Optimizer.next() will return the next experiment for this sweep or search. All experiments will have a unique set of parameter values (unless performing multiple trials per parameter set).

Normally, you would not call this directly, but use the generator Optimizer.get_experiments()

Args:

kwargs: (optional). Any keyword argument will be passed to the

  Experiment class for creation. The API key is passed directly.

Example:

>>> experiment = optimizer.next()
>>> if experiment is not None:
...     loss = fit(x, y)
...     experiment.log_metric("loss", loss)

Optimizer.next_data

next_data(self)

Optimizer.next_data() will return the next parameters in the Optimizer sweep.

Normally, you would not call this directly, but use the generator Optimizer.get_parameters()

Example:

>>> experiment = optimizer.next_data()

Optimizer.status

status(self)

Get the status from the optimizer server for this optimizer.

Example:

>>> opt.status()
{'algorithm': 'grid',
'comboCount': 32,
'configSpaceSize': 10,
'endTime': None,
'id': 'c20b90ecad694e71bdb5702778eb8ac7',
'lastUpdateTime': None,
'maxCombo': 0,
'name': 'c20b90ecad694e71bdb5702778eb8ac7',
'parameters': {'x': {'max': 20,
'min': 0,
'scalingType': 'uniform',
'type': 'integer'}},
'retryCount': 0,
'spec': {'gridSize': 10,
'maxCombo': 0,
'metric': 'loss',
'minSampleSize': 100,
'randomize': False,
'retryLimit': 20,
'seed': 2997936454},
'startTime': 1558793292216,
'state': {
'sequence_i': 0,
'sequence_retry': 11,
'sequence_trial': 0,
'total': 32,
},
'status': 'running',
'trials': 1,
'version': '1.0.0'}