Continue an Experiment¶
This page shows you how you can resume logging to an experiment that already exists in Comet.
There are scenarios where you need to continue logging data to an existing experiment run.
Here are some:
- The run is interrupted due to an error or a loss of internet connectivity.
- Your training and evaluation operations happen in different scripts.
To handle these, it is best to use Comet's
- You need to be able to get access to the current experiment created in another part of the code but can't pass the experiment object directly.
To handle this scenario, it is best to use Comet's
Comet provides an
ExistingExperiment object that allows you to continue logging from where you left off.
ExistingExperiment is particularly useful when you need to continue logging system metrics from your resumed run, or would like to pass in different command-line arguments to the resumed experiment. By default, these flags will be turned off when trying to resume an experiment run.
The following snippets illustrate how you would structure your code to continue experiments.
Create an Experiment¶
experiment = comet_ml.Experiment( api_key="<Your API Key>", project_name="<Your Project Name>" ) experiment.log_metric('my-metric', 1)
There are alternatives to setting the API key programatically. See more here.
Resume an Experiment¶
resume = True if resume: experiment = comet_ml.ExistingExperiment( api_key="<Your API Key>", experiment_key="<Key of experiment to resume>", ) else: experiment = comet_ml.Experiment( api_key="<Your API Key>", project_name="<Your Project Name>", ) # your model training or evaluation code
There are two ways to set the
experiment_key parameter. The first is by setting it directly in the source code. The second, by setting the following environment variable:
export COMET_EXPERIMENT_KEY=<The experiment key of the existing experiment>
When resuming logging, note that
ExistingExperiment does not alter or destroy previously logged information. To override or add to previous information, you must set the following parameters in the
ExistingExperiment object to
get_global method lets you get access to the current experiment created in another part of the code. So:
import comet_ml experiment = comet_ml.get_global_experiment()
The details of
comet_ml.get_global_experiment() may change in the future.
If you are trying to continue an experiment from a different session, worker, process, or thread, it is recommended you use the method described in the previous section.
Try it out!¶
Try our example for Comet in Jupyter Notebooks: