Skip to content

OfflineExperiment

This module contains the code related to offline feature

OfflineExperiment.init

__init__(project_name=None, workspace=None, log_code=True, log_graph=True,
    auto_param_logging=True, auto_metric_logging=True, parse_args=True,
    auto_output_logging="default", log_env_details=True,
    log_git_metadata=True, log_git_patch=True, disabled=False,
    offline_directory=None, log_env_gpu=True, log_env_host=True,
    api_key=None, display_summary=None, log_env_cpu=True,
    log_env_network=True, log_env_disk=True, display_summary_level=None,
    auto_weight_logging=None, auto_log_co2=False,
    auto_metric_step_rate=10, auto_histogram_tensorboard_logging=False,
    auto_histogram_epoch_rate=1, auto_histogram_weight_logging=False,
    auto_histogram_gradient_logging=False,
    auto_histogram_activation_logging=False, experiment_key=None,
    **kwargs)

Creates a new experiment and serialize it on disk. The experiment file will need to be uploaded manually later to appear on the frontend.

Args:

  • project_name: Optional. Send your experiment to a specific project. Otherwise, will be sent to Uncategorized Experiments. If project name does not already exist Comet.ml will create a new project.
  • workspace: Optional. Attach an experiment to a project that belongs to this workspace
  • log_code: Default(True) - allows you to enable/disable code logging
  • log_graph: Default(True) - allows you to enable/disable automatic computation graph logging.
  • auto_param_logging: Default(True) - allows you to enable/disable hyper-parameters logging
  • auto_metric_logging: Default(True) - allows you to enable/disable metrics logging
  • auto_metric_step_rate: Default(10) - controls how often batch metrics are logged
  • auto_histogram_tensorboard_logging: Default(False) - allows you to enable/disable automatic histogram logging
  • auto_histogram_epoch_rate: Default(1) - controls how often histograms are logged
  • auto_histogram_weight_logging: Default(False) - allows you to enable/disable automatic histogram logging of biases and weights
  • auto_histogram_gradient_logging: Default(False) - allows you to enable/disable automatic histogram logging of gradients
  • auto_histogram_activation_logging: Default(False) - allows you to enable/disable automatic histogram logging of activations
  • auto_output_logging: Default("default") - allows you to select which output logging mode to use. You can pass "native" which will log all output even when it originated from a C native library. You can also pass "simple" which will work only for output made by Python code. If you want to disable automatic output logging, you can pass False. The default is "default" which will detect your environment and deactivate the output logging for IPython and Jupyter environment and sets "native" in the other cases.
  • auto_log_co2: Default(True) - automatically tracks the CO2 emission of this experiment if codecarbon package is installed in the environment
  • parse_args: Default(True) - allows you to enable/disable automatic parsing of CLI arguments
  • log_env_details: Default(True) - log various environment information in order to identify where the script is running
  • log_env_gpu: Default(True) - allow you to enable/disable the automatic collection of gpu details and metrics (utilization, memory usage etc..). log_env_details must also be true.
  • log_env_cpu: Default(True) - allow you to enable/disable the automatic collection of cpu details and metrics (utilization, memory usage etc..). log_env_details must also be true.
  • log_env_network: Default(True) - allow you to enable/disable the automatic collection of network details and metrics (sent, receive rates, etc..). log_env_details must also be true.
  • log_env_disk: Default(True) - allow you to enable/disable the automatic collection of disk utilization metrics (usage, IO rates, etc.). log_env_details must also be true.
  • log_env_host: Default(True) - allow you to enable/disable the automatic collection of host information (ip, hostname, python version, user etc...). log_env_details must also be true.
  • log_git_metadata: Default(True) - allow you to enable/disable the automatic collection of git details
  • display_summary_level: Default(1) - control the summary detail that is displayed on the console at end of experiment. If 0, the summary notification is still sent. Valid values are 0 to 2.
  • disabled: Default(False) - allows you to disable all network communication with the Comet.ml backend. It is useful when you want to test to make sure everything is working, without actually logging anything.
  • offline_directory: the directory used to save the offline archive for the experiment.
  • experiment_key: Optional. If provided, will be used as the experiment key. If an experiment with the same key already exists, it will raise an Exception during upload. Could be set through configuration as well. Must be an alphanumeric string whose length is between 32 and 50 characters.

OfflineExperiment.add_tag

add_tag(tag)

Add a tag to the experiment. Tags will be shown in the dashboard.

Args:

  • tag: String. A tag to add to the experiment.

OfflineExperiment.add_tags

add_tags(tags)

Add several tags to the experiment. Tags will be shown in the dashboard.

Args:

  • tags: List. Tags list to add to the experiment.

OfflineExperiment.clean

clean()

Clean the experiment loggers, useful in case you want to debug your scripts with IPDB.

OfflineExperiment.context_manager

context_manager(context)

A context manager to mark the beginning and the end of the training phase. This allows you to provide a namespace for metrics/params. For example:

experiment = Experiment(api_key="MY_API_KEY")
with experiment.context_manager("validation"):
    model.fit(x_train, y_train)
    accuracy = compute_accuracy(model.predict(x_validate), y_validate)
    # returns the validation accuracy
    experiment.log_metric("accuracy", accuracy)
    # this will be logged as validation_accuracy based on the context.

OfflineExperiment.create_confusion_matrix

create_confusion_matrix(y_true=None, y_predicted=None, labels=None,
    matrix=None, title="Confusion Matrix", row_label="Actual Category",
    column_label="Predicted Category", max_examples_per_cell=25,
    max_categories=25, winner_function=None,
    index_to_example_function=None, cache=True, selected=None, images=None,
    **kwargs)

Create a confusion matrix for use over multiple epochs.

Args:

  • y_true: (optional) list of vectors representing the targets, or a list of integers representing the correct label. If not provided, then matrix may be provided.
  • y_predicted: (optional) list of vectors representing predicted values, or a list of integers representing the output. If not provided, then matrix may be provided.
  • images: (optional) a list of data that can be passed to Experiment.log_image()
  • labels: (optional) a list of strings that name of the columns and rows, in order. By default, it will be "0" through the number of categories (e.g., rows/columns).
  • matrix: (optional) the confusion matrix (list of lists). Must be square, if given. If not given, then it is possible to provide y_true and y_predicted.
  • title: (optional) a custom name to be displayed. By default, it is "Confusion Matrix".
  • row_label: (optional) label for rows. By default, it is "Actual Category".
  • column_label: (optional) label for columns. By default, it is "Predicted Category".
  • max_example_per_cell: (optional) maximum number of examples per cell. By default, it is 25.
  • max_categories: (optional) max number of columns and rows to use. By default, it is 25.
  • winner_function: (optional) a function that takes in an entire list of rows of patterns, and returns the winning category for each row. By default, it is argmax.
  • index_to_example_function: (optional) a function that takes an index and returns either a number, a string, a URL, or a {"sample": str, "assetId": str} dictionary. See below for more info. By default, the function returns a number representing the index of the example.
  • cache: (optional) should the results of index_to_example_function be cached and reused? By default, cache is True.
  • selected: (optional) None, or list of selected category indices. These are the rows/columns that will be shown. By default, select is None. If the number of categories is greater than max_categories, and selected is not provided, then selected will be computed automatically by selecting the most confused categories.
  • kwargs: (optional) any extra keywords and their values will be passed onto the index_to_example_function.
  • file_name: (optional) logging option, by default is "confusion-matrix.json",
  • overwrite: (optional) logging option, by default is False
  • step: (optional) logging option, by default is None
  • epoch: (optional) logging option, by default is None

See the executable Jupyter Notebook tutorial at Comet Confusion Matrix.

Notes:

Uses winner_function to compute winning categories for y_true and y_predicted, if they are vectors.

Examples:

>>> experiment = Experiment()

# If you have a y_true and y_predicted:
>>> y_predicted = model.predict(x_test)
>>> experiment.log_confusion_matrix(y_true, y_predicted)

# Or, if you have already computed the matrix:
>>> experiment.log_confusion_matrix(labels=["one", "two", "three"],
                                    matrix=[[10, 0, 0],
                                            [ 0, 9, 1],
                                            [ 1, 1, 8]])

# Or, if you have the categories for y_true or y_predicted
# you can just pass those in:
>>> experiment.log_confusion_matrix([0, 1, 2, 3],
                                    [2, 2, 2, 2]) # guesses 2 for all

# However, if you want to reuse examples from previous runs,
# you can reuse a Confusion Matrix instance.

>>> cm = experiment.create_confusion_matrix()
>>> y_predicted = model.predict(x_test)
>>> cm.compute_matrix(y_true, y_predicted)
>>> experiment.log_confusion_matrix(matrix=cm)

# Log again, using previously cached values:
>>> y_predicted = model.predict(x_test)
>>> cm.compute_matrix(y_true, y_predicted)
>>> experiment.log_confusion_matrix(matrix=cm)

For more details and example uses, please see: https://www.comet.com/docs/python-sdk/Comet-Confusion-Matrix/

Also, for more low-level information, see comet_ml.utils.ConfusionMatrix

OfflineExperiment.create_embedding_image

create_embedding_image(image_data, image_size,
    image_preprocess_function=None, image_transparent_color=None,
    image_background_color_function=None)

Create an embedding image (a sprite sheet). Returns the image and the url to the image.

Args:

  • image_data: list of arrays or Images
  • image_size: the size of each image
  • image_preprocess_function: (optional) if image_data is an array, apply this function to each element first
  • image_transparent_color: a (red, green, blue) tuple
  • image_background_color_function: a function that takes an index, and returns a (red, green, blue) color tuple

Returns: image and url

>>> def label_to_color(index):
...     label = labels[index]
...     if label == 0:
...         return (255, 0, 0)
...     elif label == 1:
...         return (0, 255, 0)
...     elif label == 2:
...         return (0, 0, 255)
...     elif label == 3:
...         return (255, 255, 0)
...     elif label == 4:
...         return (0, 255, 255)
...     elif label == 5:
...         return (128, 128, 0)
...     elif label == 6:
...         return (0, 128, 128)
...     elif label == 7:
...         return (128, 0, 128)
...     elif label == 8:
...         return (255, 0, 255)
...     elif label == 9:
...         return (255, 255, 255)
...
>>> image, image_url = experiment.create_embedding_image(inputs,
...     image_preprocess_function=lambda matrix: np.round(matrix/255,0) * 2,
...     image_transparent_color=(0, 0, 0),
...     image_size=(28, 28),
...     image_background_color_function=label_to_color)
...

OfflineExperiment.disable_mp

disable_mp()

Disabling the auto-collection of metrics and monkey-patching of the Machine Learning frameworks.

OfflineExperiment.display

display(*args, **kwargs)

Do nothing

OfflineExperiment.display_project

display_project(*args, **kwargs)

Do nothing

OfflineExperiment.end

end()

Use to indicate that the experiment is complete.

OfflineExperiment.flush

flush()

Used to flush all pending data to the Comet server. It works similar to .end() but without ending the run. This is a blocking operation that will wait for all the data logged so far to be delivered to the Comet server.

.flush() method returns a Boolean True or False indicating whether the flush was successful or not

OfflineExperiment.get_artifact

get_artifact(*args, **kwargs)

Getting an artifact is only supported for Online Experiment.

OfflineExperiment.get_callback

get_callback(framework, *args, **kwargs)

Get a callback for a particular framework.

When framework == 'keras' then return an instance of Comet.ml's Keras callback.

When framework == 'tf-keras' then return an instance of Comet.ml's TensorflowKeras callback.

When framework == "tf-estimator-train" then return an instance of Comet.ml's Tensorflow Estimator Train callback.

When framework == "xgboost" then return an instance of Comet.ml's xgboost.callback.TrainingCallback implementation

When framework == "fastai" then return an instance of Comet.ml's fastai.learner.Callback implementation.

Notes:

The keras callbacks are added to your Keras model.fit() callbacks list automatically to report model training metrics to Comet.ml so you do not need to add them manually.

Notes:

The lightgbm callback is added to the lightgbm.train() callbacks list automatically to report model training metrics to Comet.ml so you do not need to add it manually.

OfflineExperiment.get_keras_callback

get_keras_callback()

This method is deprecated. See Experiment.get_callback("keras")

OfflineExperiment.get_key

get_key()

Returns the experiment key, useful for using with the ExistingExperiment class Returns: Experiment Key (String)

OfflineExperiment.get_metric

get_metric(name)

Get a metric from those logged.

Args:

  • name: str, the name of the metric to get

OfflineExperiment.get_name

get_name()

Get the name of the experiment, if one.

Example:

>>> experiment.set_name("My Name")
>>> experiment.get_name()
'My Name'

OfflineExperiment.get_other

get_other(name)

Get an other from those logged.

Args:

  • name: str, the name of the other to get

OfflineExperiment.get_parameter

get_parameter(name)

Get a parameter that was logged previously in this Experiment instance. Doesn't retrieve parameter that were logged before when using the ExistingExperiment class.

If this method is called inside a context, like test, train, validate or context_manager, the current context name will be automatically added at the front of parameter name.

For example:

experiment = Experiment(api_key="MY_API_KEY")

experiment.log_parameter("training_rate", 0.0001)
with experiment.train():
    experiment.log_parameter("batch_size", 64)

assert experiment.get_parameter("training_rate") == 0.0001
assert experiment.get_parameter("train_batch_size") == 64

with experiment.train():
    assert experiment.get_parameter("batch_size") == 64

Args:

  • name: str, the name of the parameter to get

Raises:

  • KeyError: if parameter with given name not found

OfflineExperiment.get_tags

get_tags()

Return the tags of this experiment. Returns: set. The set of tags.

OfflineExperiment.log_artifact

log_artifact(artifact)

Logging an artifact is only supported for Online Experiment.

OfflineExperiment.log_asset

log_asset(file_data, file_name=None, overwrite=False, copy_to_tmp=True,
    step=None, metadata=None)

Logs the Asset determined by file_data.

Args:

  • file_data: String or File-like - either the file path of the file you want to log, or a file-like asset.
  • file_name: String - Optional. A custom file name to be displayed. If not provided the filename from the file_data argument will be used.
  • overwrite: if True will overwrite all existing assets with the same name.
  • copy_to_tmp Deprecated - If file_data is a file-like object, then this flag determines if the file is first copied to a temporary file before upload. If copy_to_tmp is False, then it is sent directly to the cloud.
  • step: Optional. Used to associate the asset to a specific step.
  • metadata: Optional. Some additional data to attach to the the audio asset. Must be a JSON-encodable dict.

Examples:

>>> experiment.log_asset("model1.h5")

>>> fp = open("model2.h5", "rb")
>>> experiment.log_asset(fp,
...                      file_name="model2.h5")
>>> fp.close()

>>> fp = open("model3.h5", "rb")
>>> experiment.log_asset(fp,
...                      file_name="model3.h5",
...                      copy_to_tmp=False)
>>> fp.close()

OfflineExperiment.log_asset_data

log_asset_data(data, name=None, overwrite=False, step=None, metadata=None,
    file_name=None, epoch=None)

Logs the data given (str, binary, or JSON).

Args:

  • data: data to be saved as asset
  • name: String, optional. A custom file name to be displayed If not provided the filename from the temporary saved file will be used.
  • overwrite: Boolean, optional. Default False. If True will overwrite all existing assets with the same name.
  • step: Optional. Used to associate the asset to a specific step.
  • epoch: Optional. Used to associate the asset to a specific epoch.
  • metadata: Optional. Some additional data to attach to the the asset data. Must be a JSON-encodable dict.

See also: APIExperiment.get_asset_list(), APIExperiment.get_asset(), and APIExperiment.get_asset_by_name()

OfflineExperiment.log_asset_folder

log_asset_folder(folder, step=None, log_file_name=None, recursive=False)

Logs all the files located in the given folder as assets.

Args:

  • folder: String - the path to the folder you want to log.
  • step: Optional. Used to associate the asset to a specific step.
  • log_file_name: Optional. if True, log the file path with each file.
  • recursive: Optional. if True, recurse folder and save file names.

If log_file_name is set to True, each file in the given folder will be logged with the following name schema: FOLDER_NAME/RELPATH_INSIDE_FOLDER. Where FOLDER_NAME is the basename of the given folder and RELPATH_INSIDE_FOLDER is the file path relative to the folder itself.

OfflineExperiment.log_audio

log_audio(audio_data, sample_rate=None, file_name=None, metadata=None,
    overwrite=False, copy_to_tmp=True, step=None)

Logs the audio Asset determined by audio data.

Args:

  • audio_data: String or a numpy array - either the file path of the file you want to log, or a numpy array given to scipy.io.wavfile.write for wav conversion.
  • sample_rate: Integer - Optional. The sampling rate given to scipy.io.wavfile.write for creating the wav file.
  • file_name: String - Optional. A custom file name to be displayed. If not provided, the filename from the audio_data argument will be used.
  • metadata: Some additional data to attach to the audio asset. Must be a JSON-encodable dict.
  • overwrite: if True will overwrite all existing assets with the same name.
  • copy_to_tmp Deprecated - If audio_data is a numpy array, then this flag determines if the WAV file is first copied to a temporary file before upload. If copy_to_tmp is False, then it is sent directly to the cloud.
  • step: Optional. Used to associate the audio asset to a specific step.

OfflineExperiment.log_code

log_code(file_name=None, folder=None, code=None, code_name=None,
    overwrite=False)

Logs additional source code files. This method allows logging code in different ways:

  • Without parameters, logs the code from the file calling this method.
  • With file_name, logs the code from the specified file.
  • Using folder, logs all files' code in that folder.
  • Using code, logs the content as source code.

Args:

  • file_name str, optional - File path to log.
  • folder str, optional - Folder path where the code files are stored.
  • code str, optional - Source code, either as text or a file-like object (such as StringIO). If passed, code_name is mandatory.
  • code_name str, optional - Name of the source code file.
  • overwrite bool, optional - If True, overwrites existing code with the same name.
>>> experiment.log_code()  # Logs code from the file using this method.
>>> experiment.log_code(file_name="dataset.py")  # Logs code from the file 'dataset.py'.
>>> experiment.log_code(folder="training/")  # Logs all files code in the 'training/' folder.
>>> experiment.log_code(code=DYNAMIC_CODE_SOURCE, code_name="user_training_code.py")  # Logs any other code.

OfflineExperiment.log_confusion_matrix

log_confusion_matrix(y_true=None, y_predicted=None, matrix=None, labels=None,
    title="Confusion Matrix", row_label="Actual Category",
    column_label="Predicted Category", max_examples_per_cell=25,
    max_categories=25, winner_function=None,
    index_to_example_function=None, cache=True,
    file_name="confusion-matrix.json", overwrite=False, step=None,
    epoch=None, images=None, selected=None, **kwargs)

Logs a confusion matrix.

Args:

  • y_true: (optional) list of vectors representing the targets, or a list of integers representing the correct label. If not provided, then matrix may be provided.
  • y_predicted: (optional) list of vectors representing predicted values, or a list of integers representing the output. If not provided, then matrix may be provided.
  • images: (optional) a list of data that can be passed to Experiment.log_image().
  • labels: (optional) a list of strings that name of the columns and rows, in order. By default, it will be "0" through the number of categories (e.g., rows/columns).
  • matrix: (optional) the confusion matrix (list of lists). Must be square, if given. If not given, then it is possible to provide y_true and y_predicted.
  • title: (optional) a custom name to be displayed. By default, it is "Confusion Matrix".
  • row_label: (optional) label for rows. By default, it is "Actual Category".
  • column_label: (optional) label for columns. By default, it is "Predicted Category".
  • max_examples_per_cell: (optional) maximum number of examples per cell. By default, it is 25.
  • max_categories: (optional) max number of columns and rows to use. By default, it is 25.
  • winner_function: (optional) a function that takes in an entire list of rows of patterns, and returns the winning category for each row. By default, it is argmax.
  • index_to_example_function: (optional) a function that takes an index and returns either a number, a string, a URL, or a {"sample": str, "assetId": str} dictionary. See below for more info. By default, the function returns a number representing the index of the example.
  • cache: (optional) should the results of index_to_example_function be cached and reused? By default, cache is True.
  • selected: (optional) None, or list of selected category indices. These are the rows/columns that will be shown. By default, select is None. If the number of categories is greater than max_categories, and selected is not provided, then selected will be computed automatically by selecting the most confused categories.
  • kwargs: (optional) any extra keywords and their values will be passed onto the index_to_example_function.
  • file_name: (optional) logging option, by default is "confusion-matrix.json",
  • overwrite: (optional) logging option, by default is False
  • step: (optional) logging option, by default is None
  • epoch: (optional) logging option, by default is None

See the executable Jupyter Notebook tutorial at Comet Confusion Matrix.

Notes:

Uses winner_function to compute winning categories for y_true and y_predicted, if they are vectors.

Examples:

>>> experiment = Experiment()

# If you have a y_true and y_predicted:
>>> y_predicted = model.predict(x_test)
>>> experiment.log_confusion_matrix(y_true, y_predicted)

# Or, if you have already computed the matrix:
>>> experiment.log_confusion_matrix(labels=["one", "two", "three"],
                                    matrix=[[10, 0, 0],
                                            [ 0, 9, 1],
                                            [ 1, 1, 8]])

# Or, if you have the categories for y_true or y_predicted
# you can just pass those in:
>>> experiment.log_confusion_matrix([0, 1, 2, 3],
                                    [2, 2, 2, 2]) # guesses 2 for all

# However, if you want to reuse examples from previous runs,
# you can reuse a ConfusionMatrix instance.

>>> from comet_ml import ConfusionMatrix

>>> cm = ConfusionMatrix()
>>> y_predicted = model.predict(x_test)
>>> cm.compute_matrix(y_true, y_predicted)
>>> experiment.log_confusion_matrix(matrix=cm)

# Log again, using previously cached values:
>>> y_predicted = model.predict(x_test)
>>> cm.compute_matrix(y_true, y_predicted)
>>> experiment.log_confusion_matrix(matrix=cm)

For more details and example uses, please see: https://www.comet.com/docs/python-sdk/Comet-Confusion-Matrix/

Also, for more low-level information, see comet_ml.utils.ConfusionMatrix

OfflineExperiment.log_current_epoch

log_current_epoch(value)

Deprecated.

OfflineExperiment.log_curve

log_curve(name, x, y, overwrite=False, step=None)

Log timeseries data.

Args:

  • name: (str) name of data
  • x: list of x-axis values
  • y: list of y-axis values
  • overwrite: (optional, bool) if True, overwrite previous log
  • step: (optional, int) the step value

Examples:

>>> experiment.log_curve("my curve", x=[1, 2, 3, 4, 5],
                                     y=[10, 20, 30, 40, 50])
>>> experiment.log_curve("my curve", [1, 2, 3, 4, 5],
                                     [10, 20, 30, 40, 50])

OfflineExperiment.log_dataframe_profile

log_dataframe_profile(dataframe: Any, name: Optional[str] = "dataframe",
    minimal: bool = False, log_raw_dataframe: bool = True,
    dataframe_format: str = "json",
    **format_kwargs: Any) -> Optional[Dict[str, Optional[Dict[str,
    str]]]]

Log a pandas DataFrame profile as an asset. Optionally, can also log the dataframe.

Args:

  • dataframe: the dataframe to profile and/or log
  • name (optional, default "dataframe"): the basename (without extension) of the dataframe assets
  • minimal (optional, default False): if True, create a minimal profile. Useful for large datasets.
  • log_raw_dataframe: (optional, default True), log the dataframe as an asset (same as calling log_table())
  • dataframe_format: (optional, default "json"), the format for optionally logging the dataframe.
  • format_kwargs: (optional), keyword args for dataframe logging as an asset.

Example:

>>> from comet_ml import Experiment
>>> import pandas as pd
>>> experiment = Experiment()
>>> df = pd.read_csv("https://data.nasa.gov/api/views/gh4g-9sfh/rows.csv?accessType=DOWNLOAD",
...     parse_dates=['year'], encoding='UTF-8')
>>> experiment.log_dataframe_profile(df)

See also: Experiment.log_table(pandas_dataframe)

OfflineExperiment.log_dataset_hash

log_dataset_hash(data)

Used to log the hash of the provided object. This is a best-effort hash computation which is based on the md5 hash of the underlying string representation of the object data. Developers are encouraged to implement their own hash computation that's tailored to their underlying data source. That could be reported as experiment.log_parameter("dataset_hash", your_hash).

data: Any object that when casted to string (e.g str(data)) returns a value that represents the underlying data.

OfflineExperiment.log_dataset_info

log_dataset_info(name=None, version=None, path=None)

Used to log information about your dataset.

Args:

  • name: Optional string representing the name of the dataset.
  • version: Optional string representing a version identifier.
  • path: Optional string that represents the path to the dataset. Potential values could be a file system path, S3 path or Database query.

At least one argument should be included. The logged values will show on the Other tab.

OfflineExperiment.log_dependency

log_dependency(name, version)

Reports name,version to the Installed Packages tab on Comet.ml. Useful to track dependencies.

Args:

  • name: Any type of key (str,int,float..)
  • version: Any type of value (str,int,float..)

Returns: None

OfflineExperiment.log_embedding

log_embedding(vectors, labels, image_data=None, image_size=None,
    image_preprocess_function=None, image_transparent_color=None,
    image_background_color_function=None, title="Comet Embedding",
    template_filename=None, group=None)

Logging embedding is only supported for Online Experiment at the moment

OfflineExperiment.log_epoch_end

log_epoch_end(epoch_cnt, step=None)

Logs that the epoch finished. Required for progress bars.

Args:

  • epoch_cnt: integer
  • step: integer

Returns: None

OfflineExperiment.log_figure

log_figure(figure_name=None, figure=None, overwrite=False, step=None)

Log a visual representation of the provided figure to Comet in SVG format.

Args:

  • figure_name: str (optional), A descriptive name for the figure. Defaults to None.
  • figure: object (optional), The figure to be logged. Accepts Plotly Figures, Matplotlib Figures, or Seaborn simple plots. If not provided, the function will log the current global Matplotlib Pyplot figure. Defaults to None.
  • overwrite: bool (optional), Determines whether to overwrite an existing figure with the same name. Defaults to False.
  • step: int (optional), Associates the figure asset with a specific step in the Comet experiment. Defaults to None.

OfflineExperiment.log_histogram_3d

log_histogram_3d(values, name=None, step=None, epoch=None, metadata=None,
    **kwargs)

Logs a histogram of values for a 3D chart as an asset for this experiment. Calling this method multiple times with the same name and incremented steps will add additional histograms to the 3D chart on Comet.ml.

Args:

  • values: a list, tuple, array (any shape) to summarize, or a Histogram object
  • name: str (optional), name of summary
  • step: Required. Used as the Z axis when plotting on Comet.ml.
  • epoch: Optional. Used as the Z axis when plotting on Comet.ml.
  • metadata: Optional: Used for items like prefix for histogram name.
  • kwargs: Optional. Additional keyword arguments for histogram.

Notes:

This method requires that step is either given here, or has been set elsewhere. For example, if you are using an auto- logger that sets step then you don't need to set it here.

OfflineExperiment.log_html

log_html(html, clear=False)

Reports any HTML blob to the HTML tab on Comet.ml. Useful for creating your own rich reports. The HTML will be rendered as an Iframe. Inline CSS/JS supported.

Args:

  • html: Any html string. for example:
  • clear: Default to False. when setting clear=True it will remove all previous html.
    experiment.log_html('<a href="www.comet.com"> I love Comet</a>')
    

Returns: None

OfflineExperiment.log_html_url

log_html_url(url, text=None, label=None)

Easy to use method to add a link to a URL in the HTML tab on Comet.ml.

Args:

  • url: a link to a file or notebook, for example
  • text: text to use a clickable word or phrase (optional; uses url if not given)
  • label: text that precedes the link

Examples:

>>> experiment.log_html_url("https://my-company.com/file.txt")

Adds html similar to:

<a href="https://my-company.com/file.txt">
  https://my-company.com/file.txt
</a>
>>> experiment.log_html_url("https://my-company.com/file.txt",
                            "File")

Adds html similar to:

<a href="https://my-company.com/file.txt">File</a>
>>> experiment.log_html_url("https://my-company.com/file.txt",
                            "File", "Label")

Adds html similar to:

Label: <a href="https://my-company.com/file.txt">File</a>

OfflineExperiment.log_image

log_image(image_data: Any, name: Optional[str] = None,
    overwrite: bool = False, image_format: str = "png",
    image_scale: float = 1.0, image_shape: Optional[Tuple[int,
    int]] = None, image_colormap: Optional[str] = None,
    image_minmax: Optional[Tuple[int, int]] = None,
    image_channels: str = "last", copy_to_tmp: bool = True,
    step: Optional[int] = None, annotations: Optional[List[Dict[str,
    Any]]] = None, metadata: Optional[Dict[str,
    Any]] = None) -> Optional[Dict[str, str]]

Logs the image. Images are displayed on the Graphics tab on Comet.ml.

Args:

  • image_data: Required. image_data is one of the following:

    • a path (string) to an image

    • a file-like object containing an image

    • a numpy matrix

    • a TensorFlow tensor

    • a PyTorch tensor

    • a list or tuple of values

    • a PIL Image

  • name: String - Optional. A custom name to be displayed on the dashboard. If not provided the filename from the image_data argument will be used if it is a path.

  • overwrite: Optional. Boolean - If another image with the same name exists, it will be overwritten if overwrite is set to True.
  • image_format: Optional. String. Default: 'png'. If the image_data is actually something that can be turned into an image, this is the format used. Typical values include 'png' and 'jpg'.
  • image_scale: Optional. Float. Default: 1.0. If the image_data is actually something that can be turned into an image, this will be the new scale of the image.
  • image_shape: Optional. Tuple. Default: None. If the image_data is actually something that can be turned into an image, this is the new shape of the array. Dimensions are (width, height) or (width, height, colors) where colors is 3 (RGB) or 1 (grayscale).
  • image_colormap: Optional. String. If the image_data is actually something that can be turned into an image, this is the colormap used to colorize the matrix.
  • image_minmax: Optional. (Number, Number). If the image_data is actually something that can be turned into an image, this is the (min, max) used to scale the values. Otherwise, the image is autoscaled between (array.min, array.max).
  • image_channels: Optional. Default 'last'. If the image_data is actually something that can be turned into an image, this is the setting that indicates where the color information is in the format of the 2D data. 'last' indicates that the data is in (rows, columns, channels) where 'first' indicates (channels, rows, columns).
  • copy_to_tmp Deprecated - If image_data is not a file path, then this flag determines if the image is first copied to a temporary file before upload. If copy_to_tmp is False, then it is sent directly to the cloud.
  • step: Optional. Used to associate the image asset to a specific step.
  • annotations: Optional. Annotations contains a list of annotation layers, where each layer contains {"name": layer_name, "data": list_of_annotations}. Each annotation can support bounding boxes, regions or both. annotations = [ { "name": "Predictions", "data": [ { "boxes": [[x, y, w, h]], "points": [[x1, y1, x2, y2, x3, y3 ...], ...], "label": // (str, required), "score": // (number, may be None) "id": // (str, optional), "metadata": {}, } ] } ]
  • metadata: Optional. Additional metadata to be associated with logged image.

OfflineExperiment.log_metric

log_metric(name, value, step=None, epoch=None, include_context=True)

Logs a general metric (i.e accuracy, f1).

e.g.

y_pred_train = model.predict(X_train)
acc = compute_accuracy(y_pred_train, y_train)
experiment.log_metric("accuracy", acc)

See also log_metrics

Args:

  • name: String - name of your metric
  • value: Float/Integer
  • step: Optional. Used as the X axis when plotting on comet.com
  • epoch: Optional. Used as the X axis when plotting on comet.com
  • include_context: Optional. If set to True (the default), the current context will be logged along the metric.

Returns: None

Down sampling metrics: Comet guarantees to store 15,000 data points for each metric. If more than 15,000 data points are reported we perform a form of reservoir sub sampling - https://en.wikipedia.org/wiki/Reservoir_sampling.

OfflineExperiment.log_metrics

log_metrics(dic, prefix=None, step=None, epoch=None)

Logs a key,value dictionary of metrics. See also log_metric

OfflineExperiment.log_model

log_model(name, file_or_folder, file_name=None, overwrite=False, metadata=None,
    copy_to_tmp=True, prepend_folder_name=True)

Logs the model data under the name. Data can be a file path, a folder path or a file-like object.

Args:

  • name: string (required), the name of the model
  • file_or_folder: the model data (required); can be a file path, a folder path or a file-like object.
  • file_name: (optional) the name of the model data. Used with file-like objects or files only.
  • overwrite: boolean, if True, then overwrite previous versions Does not apply to folders.
  • metadata: Some additional data to attach to the the data. Must be a JSON-encodable dict.
  • copy_to_tmp Deprecated - for file name or file-like; if True copy to temporary location before uploading; if False, then upload from current location
  • prepend_folder_name: boolean, default True. If True and logging a folder, prepend file path by the folder name.

Returns: dictionary of model URLs

OfflineExperiment.log_notebook

log_notebook(filename, overwrite=False)

Log a Jupyter Notebook file as an asset.

Args:

  • filename: (str) the path and name of notebook
  • overwrite: (optional, bool) if True, overwrite previous notebook

Example:

>>> experiment = Experiment()
# Save the notebook, if currently editing the notebook
>>> experiment.log_notebook("~/Untitled2394.ipynb")

OfflineExperiment.log_optimization

log_optimization(optimization_id: Optional[str] = None,
    metric_name: Optional[str] = None, metric_value: Any = None,
    parameters: Optional[Dict] = None,
    objective: Optional[str] = None) -> None

Logs an existing optimization result.

Args:

  • optimization_id: String - The id of the optimization result.
  • metric_name: String - name of your metric
  • metric_value: Float/Integer - The value of the given metric.
  • parameters: Optional Dict - Additional parametrs to be logged.
  • objective: Optional. The objective of the optimization, could be either maximum/minimum.

Returns: None

OfflineExperiment.log_other

log_other(key, value)

Reports a key and value to the Other tab on Comet.ml. Useful for reporting datasets attributes, datasets path, unique identifiers etc.

See related methods: log_parameter and log_metric

Other metadata can also be defined as environment variables. All environment variables that starts with COMET_LOG_OTHER_ will be logged automatically to the Experiment.

Args:

  • key: Any type of key (str,int,float..)
  • value: Any type of value (str,int,float..)

Returns: None

OfflineExperiment.log_others

log_others(dictionary)

Reports dictionary of key/values to the Other tab on Comet.ml. Useful for reporting datasets attributes, datasets path, unique identifiers etc.

See log_other

Args:

  • key: dict of key/values where value is Any type of value (str,int,float..)

Returns: None

OfflineExperiment.log_parameter

log_parameter(name, value, step=None)

Logs a single hyperparameter. For additional values that are not hyper parameters it's encouraged to use log_other.

See also log_parameters.

If the same key is reported multiple times only the last reported value will be saved.

If this method is called inside a context, like test, train, validate or context_manager, the parameter will be stored with the current context name as a prefix.

For example, the following code:

experiment = Experiment(api_key="MY_API_KEY")

with experiment.train():
    experiment.log_parameter("batch_size", 64)

Will logs the hyper-parameter train_batch_size.

Args:

  • name: String - name of your parameter
  • value: Float/Integer/Boolean/String/List
  • step: Optional. Used as the X axis when plotting on Comet.ml
  • source: Optional. Used to define source where parameter value was originated from.

Returns: None

OfflineExperiment.log_parameters

log_parameters(parameters, prefix=None, step=None, nested_support=True)

Logs a dictionary (or dictionary-like object) of multiple parameters. See also log_parameter.

e.g:

experiment = Experiment(api_key="MY_API_KEY")
params = {
    "batch_size":64,
    "layer1":"LSTM(128)",
    "layer2":"LSTM(128)",
    "MAX_LEN":200
}

experiment.log_parameters(params)

If you call this method multiple times with the same keys your values would be overwritten. For example:

experiment.log_parameters({"key1":"value1","key2":"value2"})
On Comet.com you will see the pairs of key1 and key2.

If you then call:

experiment.log_parameters({"key1":"other value"})l
On the UI you will see the pairs key1: other value, key2: value2

If this method is called inside a context, like test, train, validate or context_manager, the parameters will be stored with the current context name as a prefix.

OfflineExperiment.log_points_3d

log_points_3d(scene_name: str, points: Optional[List[Point3D]] = None,
    boxes: Optional[List[Dict[str, Any]]] = None,
    step: Optional[int] = None, metadata: Optional[Dict] = None)

Log 3d points and bounding boxes as an asset. You can visualize the asset with the following panel: see 3D Points.

Args:

  • scene_name: a string identifying the 3d scene to render. A same scene name could be logged across different steps.
  • points (optional, default None): a list of points, each point being a list (or equivalent like Numpy array). Each point length should be either 3, if only the position is given: [X, Y, Z]. The length could also be 6, if color is passed as well: [X, Y, Z, R, G, B]. Red, Green and Blue should be a number between 0 and 1. Either points or boxes are required.
  • step: Optional. Used to associate the asset to a specific step.
  • epoch: Optional. Used to associate the asset to a specific epoch.
  • boxes (optional, default None): a list of box definition Dict. Each box should match the following format:
    {
        "segments": [[[1, 2, 3], ...], ...], # Required, [[[x, y, z], [x, y, z], ...], ...]
        "name": "prediction", # Required
        "color": [255, 128, 0], # Optional, [R, G, B], values between 0 and 255.
        "score": 0.96, # Optional, value between 0 and 1.
        "label": "person", # Optional
    }

Note: Either points or boxes are required.

OfflineExperiment.log_remote_asset

log_remote_asset(uri, remote_file_name=None, overwrite=False,
    asset_type="asset", step=None, metadata=None)

Logs a Remote Asset identified by an URI. A Remote Asset is an asset but its content is not uploaded and stored on Comet. Rather a link for its location is stored, so you can identify and distinguish between two experiment using different version of a dataset stored somewhere else.

Args:

  • uri: String - the remote asset location, there is no imposed format, and it could be a private link.
  • remote_file_name: String, Optional. The "name" of the remote asset, could be a dataset name, a model file name.
  • overwrite: if True will overwrite all existing assets with the same name.
  • step: Optional. Used to associate the asset to a specific step.
  • metadata: Some additional data to attach to the remote asset. Must be a JSON-encodable dict.

Examples:

>>> experiment.log_remote_asset("s3://bucket/folder/file")

>>> experiment.log_remote_asset("dataset:701bd06b43b7423296fb626027d02198")

OfflineExperiment.log_remote_model

log_remote_model(model_name: str, uri: str, metadata: Any = None,
    sync_mode: bool = True, max_synced_objects: int = 10000) -> None

Logs the model data from the URI provided under the name of the model. If the URI begins with s3:// or gs://, Comet attempts to list all objects in the given bucket and path and log them individually.

Args:

  • model_name: string (required), the name of the model.
  • uri: string (required), the remote model location, there is no imposed format and it could be a private link. Can log a single file and a folder.
  • metadata: Some additional data to attach to the the data. Must be a JSON-encodable dict.
  • sync_mode: Bool, If True and the URI begins with s3:// or gs://, Comet attempts to list all objects in the given bucket and path. All the objects will be logged under the given model name. If False, Comet just logs a single remote model with the provided URI as the remote URI. Default is True.
  • max_synced_objects: int, When sync_mode is True and the URI begins with s3:// or gs://, set the maximum number of S3/GCP objects to log. If there are more matching S3/GCP objects than max_synced_objects, a warning will be displayed and the provided URI will be logged as a single remote model.

OfflineExperiment.log_system_info

log_system_info(key, value)

Reports the key and value to the System Metric tab on Comet.ml. Useful to track general system information. This information can be added to the table on the Project view. You can retrieve this information via the Python API.

Args:

  • key: Any type of key (str,int,float..)
  • value: Any type of value (str,int,float..)

Returns: None

Example:

# Can also use ExistingExperiment here instead of Experiment:
>>> from comet_ml import Experiment, APIExperiment
>>> e = Experiment()
>>> e.log_system_info("info-about-system", "debian-based")
>>> e.end()

>>> apie = APIExperiment(previous_experiment=e.id)
>>> apie.get_system_details()['logAdditionalSystemInfoList']
[{"key": "info-about-system", "value": "debian-based"}]

OfflineExperiment.log_table

log_table(filename, tabular_data=None, headers=False, **format_kwargs)

Log tabular data, including data, csv files, tsv files, and Pandas dataframes.

Args:

  • filename: str (required), a filename ending in ".csv", or ".tsv" (for tablular data) or ".json", ".csv", ".md", or ".html" (for Pandas dataframe data).
  • tabular_data: (optional) data that can be interpreted as 2D tabular data or a Pandas dataframe).
  • headers: bool or list, if True, will add column headers automatically if tabular_data is given; if False, no headers will be added; if list then it will be used as headers. Only useful with tabular data (csv, or tsv).
  • format_kwargs: (optional keyword arguments), when passed a Pandas dataframe these keyword arguments are used in the conversion to "json", "csv", "md", or "html". See Pandas Dataframe conversion methods (like to_json()) for more information.

See also:

Examples:

>>> experiment.log_table("vectors.tsv",
...                      [["one", "two", "three"],
...                       [1, 2, 3],
...                       [4, 5, 6]],
...
>>> experiment.log_table("dataframe.json", pandas_dataframe)

See also: Experiment.log_panadas_profile()

OfflineExperiment.log_tensorflow_folder

log_tensorflow_folder(folder)

Logs all the tensorflow log files located in the given folder as assets.

Args:

  • folder: String - the path to the folder you want to log.

Use APIExperiment.download_tensorflow_folder() to get the files.

Example:

>>> experiment = comet_ml.Experiment()
>>> experiment.log_tensorboard_folder("logs")
>>> api = comet_ml.API()
>>> api_experiment = api.get_experiment_by_id(experiment.id)
>>> api_experiment.download_tensorflow_folder()

OfflineExperiment.log_text

log_text(text, step=None, metadata=None)

Logs the text. These strings appear on the Text Tab in the Comet UI.

Args:

  • text: string to be stored
  • step: Optional. Used to associate the asset to a specific step.
  • metadata: Some additional data to attach to the text. Must be a JSON-encodable dict.

OfflineExperiment.log_video

log_video(file: Union[str, IO], name: Optional[str] = None,
    overwrite: bool = False, format: Optional[str] = None,
    step: Optional[int] = None, epoch: Optional[int] = None,
    metadata: Optional[Dict[str, Any]] = None) -> Optional[Dict[str,
    str]]

Logs the video to Comet. Videos are displayed on the assets tab in Comet and support the following formats: MP4, MOV, WMV, and GIF.

Args:

  • file: Required. The path to the video or a file-like object that contains the video.
  • name: String - Optional. A custom name can be provided to be displayed on the assets tab. If not provided, the filename from the file argument will be used if it is a path.
  • overwrite: Boolean - Optional. If another video with the same name exists, it will be overwritten if overwrite is set to True.
  • format: String - Optional. The extension of the file is used to check if the asset is of a supported format. If the extension of the file is not correct or if you are uploading a file-like object, you can indicate the format of the video through this parameter.
  • step: Number - Optional. This is used to associate the video asset with a specific step.
  • epoch: Integer - Optional. Used to associate the asset to a specific epoch.
  • metadata: Optional. Additional custom metadata can be associated with the logged video.

OfflineExperiment.register_model

register_model(model_name: str, version: Optional[str] = None,
    workspace: Optional[str] = None,
    registry_name: Optional[str] = None, public: Optional[bool] = None,
    description: Optional[str] = None, comment: Optional[str] = None,
    tags: Optional[list] = None, stages: Optional[list] = None,
    status: Optional[str] = None, sync: Optional[bool] = False,
    timeout: float = 10.0) -> None

Register an experiment model in the workspace registry.

Args:

  • model_name: the name of the experiment model
  • workspace Deprecated - optional, the name of workspace; defaults to current workspace
  • version: a proper semantic version string; defaults to "1.0.0"
  • registry_name: optional, the name of the registered workspace model, if not provided the model_name will be used instead.
  • public: optional, if True, then the model will be publicly viewable
  • description: optional, a textual description of the model
  • comment: optional, a textual comment about the model
  • tags: optional, a list of textual tags such as ["tag1", "tag2"], etc.
  • stages Deprecated - optional, equivalent to tags, with newer backend versions
  • status: optional, a string describing the status of this model version.
  • sync: Whether this function is synchronous and will be finished only once the model was registered.
  • timeout: Maximum time (In seconds) before the function would end if called with sync = True.

OfflineExperiment.send_notification

send_notification(title, status=None, additional_data=None)

With an Experiment, this method will send you a notification through email when an experiment ends. However, with other experiment types, this method does nothing.

OfflineExperiment.set_cmd_args

set_cmd_args()

OfflineExperiment.set_code

set_code(code=None, overwrite=False, filename=None)

Sets the current experiment script's code. Should be called once per experiment.

Deprecated: Use Experiment.log_code()

Args:

  • code: optional, string: experiment source code.
  • overwrite: optional, bool: if True, send the code
  • filename: optional, str: name of file to get source code from

OfflineExperiment.set_epoch

set_epoch(epoch)

Sets the current epoch in the training process. In Deep Learning each epoch is an iteration over the entire dataset provided. This is used to generate plots on comet.com. You can also pass the epoch directly when reporting log_metric.

Args:

  • epoch: Integer value

Returns: None

OfflineExperiment.set_filename

set_filename(fname)

Sets the current experiment filename.

Args:

  • fname: String. script's filename.

OfflineExperiment.set_model_graph

set_model_graph(graph, overwrite=False)

Sets the current experiment computation graph.

Args:

  • graph: String or Google Tensorflow Graph Format.
  • overwrite: Bool, if True, send the graph again

OfflineExperiment.set_name

set_name(name)

Set a name for the experiment. Useful for filtering and searching on Comet.ml. Will shown by default under the Other tab.

Args:

  • name: String. A name for the experiment.

OfflineExperiment.set_os_packages

set_os_packages()

Reads the installed os packages and reports them to server as a message. Returns: None

OfflineExperiment.set_pip_packages

set_pip_packages()

Get the installed pip packages using pkg resources and reports them to server as a message. Returns: None

OfflineExperiment.set_step

set_step(step)

Sets the current step in the training process. In Deep Learning each step is after feeding a single batch into the network. This is used to generate correct plots on Comet.ml. You can also pass the step directly when reporting log_metric, and log_parameter.

Args: step: Integer value

Returns: None

OfflineExperiment.stop_early

stop_early(epoch)

Should the experiment stop early?

OfflineExperiment.test

test()

A context manager to mark the beginning and the end of the testing phase. This allows you to provide a namespace for metrics/params. For example:

with experiment.test():
    pred = model.predict(x_test)
    test_acc = compute_accuracy(pred, y_test)
    experiment.log_metric("accuracy", test_acc)
    # this will be logged as test accuracy
    # based on the context.

OfflineExperiment.train

train()

A context manager to mark the beginning and the end of the training phase. This allows you to provide a namespace for metrics/params. For example:

experiment = Experiment(api_key="MY_API_KEY")
with experiment.train():
    model.fit(x_train, y_train)
    accuracy = compute_accuracy(model.predict(x_train),y_train)
    # returns the train accuracy
    experiment.log_metric("accuracy",accuracy)
    # this will be logged as train accuracy based on the context.

OfflineExperiment.url

url()

Get the url of the experiment.

Example:

>>> experiment.url
"https://www.comet.com/username/34637643746374637463476"

OfflineExperiment.validate

validate()

A context manager to mark the beginning and the end of the validating phase. This allows you to provide a namespace for metrics/params. For example:

with experiment.validate():
    pred = model.predict(x_validation)
    val_acc = compute_accuracy(pred, y_validation)
    experiment.log_metric("accuracy", val_acc)
    # this will be logged as validation accuracy
    # based on the context.
Mar. 27, 2024