Integrate with Snowflake¶

Comet's integration with Snowflake's scalable cloud data platform provides developers building Machine Learning applications with a comprehensive overview of their model development process by allowing them to track and version their Snowflake queries and datasets. As a result, developers can trace the lineage of their models and their performance all the way back to their input datasets and the queries that created those datasets.

Start logging¶

Seamlessly upload a SnowPark DataFrame as a Comet Artifact.

import os
import comet_ml
from comet_ml.integration import snowflake as comet_snowflake

from snowflake.snowpark import Session

# Configure Snowflake and Comet
connection_parameters = {
    "account": "Your account name",
    "user": "Your user name",
    "password": "Your Snowflake Password",
    "database": "Your database name",
}

session = Session.builder.configs(connection_parameters).create()
df = session.sql("select * from TPCH_SF10.ORDERS limit 1000")

experiment = comet_ml.Experiment(project_name="comet-snowflake")
comet_snowflake.log_artifact_v1(
    experiment, "my-snowflake-dataframe", df, sample_size=50
)

experiment.end()

Log Automatically¶

Comet will log the following information from your Snowflake Dataframes as a Comet Artifact

The query used to create the dataset
A sample of the dataset, the size of which can be configured through the SDK
Any error messages that are raised while querying the data.
A link back to the Snowflake instance where the data is stored

You can find the logged dataset information in the Comet Artifacts Tab

Fetching Query Information from Logged Snowflake Artifacts¶

You can retreive the queries used to create your Snowflake datasets directly from the logged Artifacts.

import os
import comet_ml
from comet_ml.integration import snowflake as comet_snowflake

from snowflake.snowpark import Session

# Configure Snowflake and Comet
connection_parameters = {
    "account": "Your account name",
    "user": "Your user name",
    "password": "Your Snowflake Password",
    "database": "Your database name",
}

session = Session.builder.configs(connection_parameters).create()

experiment = comet_ml.Experiment(project_name="comet-snowflake")
query = comet_snowflake.get_artifact_v1(
    experiment, "my-snowflake-dataframe", "team-comet-ml"
)
df = session.sql(query)

Try it out!¶

Try out a full example of using Comet with Snowflake in this Colab Notebook.

Apr. 25, 2024