Configure the Optimizer¶

Comet Optimizer offers you an easy-to-use interface for model tuning which supports any ml framework and can easily be integrated in any of your workflows.

The main set-up step is to define the tuning configuration for Optimizer inside a configuration dictionary.

Below, we provide you with a complete reference for the Optimizer configuration specs followed by configuration examples for all supported search algorithms.

The structure of the configuration dictionary¶

The Optimizer configuration is defined as a JSON dictionary that consists of five sections:

The configuration dictionary can be defined in code as a map or inside a configuration file to be accessed within your script.

Deep-dive into the dictionary configuration¶

You can find here a detailed description of the five sections, i.e. keys, of the Optimizer configuration dictionary.

The algorithm field¶

algorithm is a mandatory configuration key that specifies the search algorithm to use.

Possible options are: random, grid, and bayes.

You can discover more about each algorithm from our Hyperparameter Optimization with Comet blog post.

The spec field¶

spec is a mandatory configuration key used to define algorithm-specific specifications.

Note that the search algorithms slightly vary on supported specs, as highlighted in the "Use it for" column in the table below.

When setting the values for these specs, keep in mind that:

• objective is only needed for the bayes algorithm to drive its smart hyperparameter selection.

• maxCombo should be left to default for a complete grid search, but consider setting it to a lower value than the maximum number of combinations for random search and bayes search to take advantage of their underlying hyperparameter selection logic.

  Note that the maximum number of combinations is defined as:

  \[ max\_combos = (gridSize * n\_continuous) \times (\prod_{i=1}^{n\_discrete} n\_discrete\_values_i) \]

  where \(n\_continuous\) refers to the number of continuous parameters, \(n\_discrete\) refers to the number of discrete parameters, and \(n\_discrete\_values\) refers to the number of possible values (for the discrete parameter).

  Find example calculations in the Examples of configuration dictionary section below.

• gridSize, and minSampleSize are useful to help control the computational complexity of the optimization process in the compromise between search completeness and cost, which is fundamental when dealing with large search spaces.

  Note that for the bayes search algorithm, each parameter is considered as a distribution to sample according to the performance of the previous hyperparameter selection hence these two spec keys don't apply.

• retryLimit failures could be due to numerical instability, resource constraints, or data issues as the script runs. retryAssignLimit failures could be do to network instability or resource allocation issues before the script runs.

1

my_combined_metric = 0.2 * metric_1 + 0.8 * metric_2

The parameters field¶

parameters is a mandatory configuration key that expects a dictionary of all the hyperparameters to be optimized.

The following example shows a possible parameters definition with three different parameter types:

1
2
3
4
5

"parameters": {
    "hidden-layer-size": {"type": "integer", "scaling_type": "uniform", "min": 5, "max": 100},
    "batch_size": {"type": "discrete", "values": [16, 32, 64]},
    "momentum": {"type": "float", "scaling_type": "normal", "mu": 10, "sigma": 5},
 }

The hyperparameter format is inspired by Google's Vizier as exemplified by the open source version called Advisor. As such, each hyperparameter is defined as a key-value pair where the key is the hyperparameter name and the value is a dictionary with the following attributes:

• type: one of integer, float, double, categorical, discrete.

• For categorical and discrete types only:

  • values: a list of string or numerical values, respectively.

• For integer, float, and double types only:

  • scaling_type: one of linear (default), uniform, normal, loguniform, and lognormal.

  • For linear, uniform, and loguniform scaling types only:

    • min: lower bound of the range.
    • max: upper bound of the range.

  • For normal and lognormal scaling types only:

    • mu: mean of the distribution.
    • sigma: standard deviation of the distribution.

• For grid and random search algorithm only:

  • grid_size: integer resolution size for how to divide up the parameter space into discrete bins. You can also specify this at a higher level for all parameters with the gridSize spec key.

The name field¶

name is an optional configuration key that you can use to specify a name for your tuning runs. The name is logged in the Other tab of the Single Experiment page under the key "optimizer_name".

The default value is the Optimizer.ID.

Use it to easily track and organize multiple optimization experiments.

The trials field¶

trials is an optional configuration key that allows you to specify the integer number of tuning trials to run for a single parameter set.

The default value is 1.

Use it to compensate for the variability in training runs due to the non-deterministic nature of ML by averaging performance across trials.

Examples of configuration dictionary¶

Below we provide an example of configuration dictionary for each algorithm type, with associated considerations on the chosen specs and tuning output.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

{
    "algorithm": "grid",
    "spec": {
        "metric": "loss",
        "gridSize": 10,
    },
    "parameters": {
        "learning_rate": {"type": "integer", "scaling_type": "log_uniform", "min": 0.00001, "max": 0.001},
        "batch_size": {"type": "discrete", "values": [16, 32, 64]},
    },
    "name": "My Grid Search",
    "trial": 1,
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

{
    "algorithm": "random",
    "spec": {
        "metric": "accuracy",
        "maxCombo": 25,
        "gridSize": 5,
        "minSampleSize": 150,
    },
    "parameters": {
        "hidden-layer-size": {"type": "integer", "scaling_type": "linear", "min": 5, "max": 100},
        "hidden2-layer-size": {"type": "discrete", "values": [16, 32, 64]},
        "optimizer": {"type": "categorical", "values": ["SVG", "Adam", "NAdam"]},
    },
    "name": "My Random Search",
    "trials": 2,
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

{
    "algorithm": "bayes",
    "spec": {
        "objective": "maximize",
        "metric": "auc",
        "maxCombo": 20,
        "gridSize": 10,
        "retryLimit": 10,
        "retryAssignLimit": 10,
    },
    "parameters": {
        "gamma": {"type": "float", "scaling_type": "uniform", "min": 0.01, "max": 0.9},
        "eta": {"type": "float", "scaling_type": "uniform", "min": 0.01, "max": 0.9},
        "n_estimators": {"type": "integer", "scaling_type": "lognormal", "mu": 100, "sigma": 20}},
    },
    "name": "My Bayesian Search",
    "trials": 1,
}