Integrate with Langchain¶

LangChain is a framework for developing applications powered by language models. Langchain provides APIs to build data aware, agentic applications that allow language models to incorporate external data sources and interact with their compute environments.

Start logging¶

Import the CometCallbackHandler into your script
Add the callback to Langchain's CallbackManager.
Pass the callback_manager parameter to your language model, chain, or agent. Also make sure to pass the parameter verbose=True to enable logging.

from langchain.callbacks import CometCallbackHandler
from langchain.llms import OpenAI

comet_callback = CometCallbackHandler(
    project_name="comet-example-langchain",
    complexity_metrics=True,
    stream_logs=True,
    tags=["llm"],
    visualizations=["dep"],
)
llm = OpenAI(temperature=0.9, callbacks=[comet_callback], verbose=True)

llm_result = llm.generate(["Tell me a joke", "Tell me a poem", "Tell me a fact"])
print("LLM result", llm_result)
comet_callback.flush_tracker(llm, finish=True)

Log automatically¶

Comet will log the following information from your Langchain Runs

Language Model Parameters
Input Text Prompts
Generated Output Text
A Session Dataframe that maps input prompts to their generated outputs
Action Records Log. Comet will log every action step executed in the chain, which can then be visualized using Comet's LLM Panels.
Source Code
The Chain saved as a model.json file.

End to End Example¶

If you can't wait, check out the results of this example Langchain project for a preview of what's to come.

Install Dependencies¶

pip install comet_ml "langchain>=0.0.40" openai google-search-results spacy textstat pandas

python -m spacy download en_core_web_sm

Set your Comet Credentials¶

You can grab your Comet API key here

export COMET_API_KEY="<Your Comet API Key>"

Set Third Party Credentials¶

You will need an OpenAI API Key and a SerpAPI API Key to run the following examples

export OPENAI_API_KEY="Your OpenAI API Key"
export SERPAPI_API_KEY="Your SerpAPI API Key"

Run the example¶

from langchain.agents import initialize_agent, load_tools
from langchain.callbacks import CometCallbackHandler
from langchain.llms import OpenAI

comet_callback = CometCallbackHandler(
    project_name="comet-example-langchain",
    complexity_metrics=True,
    stream_logs=True,
    tags=["agent"],
)
callbacks = [comet_callback]
llm = OpenAI(temperature=0.9, callbacks=callbacks, verbose=True)

tools = load_tools(["serpapi", "llm-math"], llm=llm, callbacks=callbacks, verbose=True)
agent = initialize_agent(
    tools,
    llm,
    agent="zero-shot-react-description",
    callbacks=callbacks,
    verbose=True,
)
agent.run(
    "Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?"
)
comet_callback.flush_tracker(agent, finish=True)

Try it out!¶

Try out some examples for using Comet with Langchain in this Colab Notebook.

Configure the `CometCallbackHandler` for Langchain¶

The Comet callback can be further customized to log spacy visualizations, text complexity metrics, and custom metrics.

Log Spacy Visualizations¶

To log visualizations of the generated text with Spacy, pass in a list of visualization names to the visualizations argument of the callback handler.

comet_callback = CometCallbackHandler(
    project_name="comet-example-langchain",
    stream_logs=True,
    tags=["llm"],
    visualizations=["dep", "ent"],
)

Log Text Complexity Metrics¶

Enable Logging Text Complexity Metrics to Comet via Textstat by setting the complexity_metrics argument to True

comet_callback = CometCallbackHandler(
    project_name="comet-example-langchain",
    complexity_metrics=True,
    stream_logs=True,
    tags=["llm"],
)

Log Custom Metrics¶

Comet supports logging custom metrics from Langchain by passing a callable to the custom_metrics argument. The callable must accept the following arguments:

A Langchain Generation object.
An integer argument for the prompt index, which tracks which input prompt was used to create the current output generation.
An integer argument for the generation index, which tracks the index of the generated output when creating multiple generations per input prompt.

from rouge_score import rouge_scorer

from langchain.callbacks import CometCallbackHandler
from langchain.chains import LLMChain
from langchain.llms import OpenAI
from langchain.prompts import PromptTemplate


class Rouge:
    def __init__(self, reference):
        self.reference = reference
        self.scorer = rouge_scorer.RougeScorer(["rougeLsum"], use_stemmer=True)

    def compute_metric(self, generation, prompt_idx, gen_idx):
        prediction = generation.text
        results = self.scorer.score(target=self.reference, prediction=prediction)

        return {
            "rougeLsum_score": results["rougeLsum"].fmeasure,
            "reference": self.reference,
        }


reference = """
The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building.
It was the first structure to reach a height of 300 metres.

It is now taller than the Chrysler Building in New York City by 5.2 metres (17 ft)
Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France .
"""
rouge_score = Rouge(reference=reference)

template = """Given the following article, it is your job to write a summary.
Article:
{article}
Summary: This is the summary for the above article:"""
prompt_template = PromptTemplate(input_variables=["article"], template=template)

comet_callback = CometCallbackHandler(
    project_name="comet-example-langchain",
    complexity_metrics=False,
    stream_logs=True,
    tags=["custom_metrics"],
    custom_metrics=rouge_score.compute_metric,
)
callbacks = [comet_callback]
llm = OpenAI(temperature=0.9, callbacks=callbacks, verbose=True)

synopsis_chain = LLMChain(llm=llm, prompt=prompt_template, callbacks=callbacks, verbose=True)

test_prompts = [
    {
        "article": """
                 The tower is 324 metres (1,063 ft) tall, about the same height as
                 an 81-storey building, and the tallest structure in Paris. Its base is square,
                 measuring 125 metres (410 ft) on each side.
                 During its construction, the Eiffel Tower surpassed the
                 Washington Monument to become the tallest man-made structure in the world,
                 a title it held for 41 years until the Chrysler Building
                 in New York City was finished in 1930.

                 It was the first structure to reach a height of 300 metres.
                 Due to the addition of a broadcasting aerial at the top of the tower in 1957,
                 it is now taller than the Chrysler Building by 5.2 metres (17 ft).

                 Excluding transmitters, the Eiffel Tower is the second tallest
                 free-standing structure in France after the Millau Viaduct.
                 """
    }
]
print(synopsis_chain.apply(test_prompts))
comet_callback.flush_tracker(synopsis_chain, finish=True)

May. 24, 2023