Skip to content

Continue an Experiment

This page shows you how you can resume logging to an experiment that already exists in Comet.

There are scenarios where you need to continue logging data to an existing experiment run.

Here are some:

  • The run is interrupted due to an error or a loss of internet connectivity.
  • Your training and evaluation operations happen in different scripts.

To handle these, it is best to use Comet's ExistingExperiment object.

  • You need to be able to get access to the current experiment created in another part of the code but can't pass the experiment object directly.

To handle this scenario, it is best to use Comet's get_global method.

Use the ExistingExperiment method

Open In Colab

Comet provides an ExistingExperiment object that allows you to continue logging from where you left off.

ExistingExperiment is particularly useful when you need to continue logging system metrics from your resumed run, or would like to pass in different command-line arguments to the resumed experiment. By default, these flags will be turned off when trying to resume an experiment run.

The following snippets illustrate how you would structure your code to continue experiments.

Create an Experiment

experiment = comet_ml.Experiment(
    api_key="<Your API Key>",
    project_name="<Your Project Name>"
)
experiment.log_metric('my-metric', 1)
That code creates an Experiment in the Comet UI with an Experiment key. You can use this key to resume your experiment at a different point in time.

Note

There are alternatives to setting the API key programatically. See more here.

Resume an Experiment

resume = True

if resume:
    experiment = comet_ml.ExistingExperiment(
        api_key="<Your API Key>",
        experiment_key="<Key of experiment to resume>",
    )

else:
    experiment = comet_ml.Experiment(
        api_key="<Your API Key>",
        project_name="<Your Project Name>",
    )

# your model training or evaluation code

There are two ways to set the experiment_key parameter. The first is by setting it directly in the source code. The second, by setting the following environment variable:

export COMET_EXPERIMENT_KEY=<The experiment key of the existing experiment>

When resuming logging, note that ExistingExperiment does not alter or destroy previously logged information. To override or add to previous information, you must set the following parameters in the ExistingExperiment object to True:

log_codelog_graphparse_args
log_env_detailslog_git_metadatalog_git_patch
log_env_gpulog_env_cpulog_env_host

Use the get_global method

The get_global method lets you get access to the current experiment created in another part of the code. So:

import comet_ml
experiment = comet_ml.get_global_experiment()

Info

The details of comet_ml.get_global_experiment() may change in the future.

If you are trying to continue an experiment from a different session, worker, process, or thread, it is recommended you use the method described in the previous section.

Try it out!

Try our example for Comet in Jupyter Notebooks:

Open In Colab

Mar. 27, 2024