Log Remote Data¶
This guide provides an overview of how to log remote data, including assets and models, to your Comet experiments. Leveraging remote logging can help you manage large datasets and models efficiently without needing to store them directly within Comet.
Registering Remote Artifacts¶
For a detailed explanation of registering remote artifacts, please refer to the official Comet ML documentation: Register Remote Artifacts.
This linked documentation covers the fundamentals of Comet's Artifacts system, which is crucial for managing and versioning your data and models.
Remote Assets¶
Comet allows you to log references to assets that are stored remotely (e.g., in cloud storage like Amazon S3 or Google Cloud Storage) rather than uploading the entire file to Comet's servers. This is particularly useful for very large files.
The primary method for logging a remote asset is Experiment.log_remote_asset().
This method enables you to log a URL that points to an asset. Comet will then store this URL, allowing you to access the asset from your experiment. Comet ML enables you to log various types of data, such as model files, datasets, images, and other custom assets. Crucially, you can enrich all logged data with associated metadata using the metadata parameter, allowing you to attach key-value pairs that describe the data's origin, version, characteristics, or any other relevant information for better organization and traceability.
Example:
from comet_ml import Experiment
experiment = Experiment(project_name="my-remote-project")
# Log a remote image from an S3 bucket
experiment.log_remote_asset(
uri="s3://remote-assets-demo/images/classification_image.img",
metadata={"size": "5.2 MB"},
asset_type="image",
step=50
)
# Log a remote dataset from S3
experiment.log_remote_asset(
uri="s3://remote-assets-demo/data/training_data.csv",
metadata={"source": "s3"},
asset_type="dataset"
)
experiment.end()
Remote Models¶
Similar to remote assets, you can log metadata about a model that is stored in a remote location, such as AWS S3, Google Cloud Storage, or Azure Blob Storage, without requiring you to upload the actual model files to Comet. This is particularly useful for managing large models or when you prefer to keep your model files in your existing storage infrastructure.
The primary method for logging a remote model is Experiment.log_remote_model().
This method allows you to track different remote versions of your models and enrich the logged metadata with relevant information like training details, performance metrics, or any other custom data for enhanced organization and traceability.
from comet_ml import Experiment
experiment = Experiment(project_name="my-remote-project")
# Log metadata for a model stored on AWS S3
experiment.log_remote_model(
model_name="production_model_v1",
uri="s3://my-model-bucket/models/prod_model_2023-01-15.pth",
metadata={"version": "1.0", "framework": "PyTorch", "dataset": "ImageNet"},
sync_mode=False
)
experiment.end()
You can then view this model under your "Assets & Artifacts" tab within your Comet experiment, categorized by the asset_type you provided.
Synced Mode with S3/GCS Integration¶
When you use Experiment.log_remote_model() with S3 or GCS URLs, you can set sync_mode=True to sync your remote model. Comet's synced mode enhances the logging of remote models by integrating directly with your S3 or GCS buckets, allowing you to keep track of model metadata without moving the actual model files into Comet. Once the model is registered, you can then download the model with Model.download(). You Can find more on how to register your remote model here: Register Remote Model.
Syncing is enabled by default but requires the user to have configured the authentication credentials following the cloud providers recommendation. You can learn more at: