Skip to content

Integrate with Langchain

LangChain is a framework for developing applications powered by language models. Langchain provides APIs to build data aware, agentic applications that allow language models to incorporate external data sources and interact with their compute environments.

Open In Colab

Start logging

  1. Import the CometCallbackHandler into your script
  2. Add the callback to Langchain's CallbackManager.
  3. Pass the callback_manager parameter to your language model, chain, or agent. Also make sure to pass the parameter verbose=True to enable logging.
from langchain.callbacks import CometCallbackHandler
from langchain.llms import OpenAI

comet_callback = CometCallbackHandler(
    project_name="comet-example-langchain",
    complexity_metrics=True,
    stream_logs=True,
    tags=["llm"],
    visualizations=["dep"],
)
llm = OpenAI(temperature=0.9, callbacks=[comet_callback], verbose=True)

llm_result = llm.generate(["Tell me a joke", "Tell me a poem", "Tell me a fact"])
print("LLM result", llm_result)
comet_callback.flush_tracker(llm, finish=True)

Log automatically

Comet will log the following information from your Langchain Runs

  1. Language Model Parameters
  2. Input Text Prompts
  3. Generated Output Text
  4. A Session Dataframe that maps input prompts to their generated outputs
  5. Action Records Log. Comet will log every action step executed in the chain, which can then be visualized using Comet's LLM Panels.
  6. Source Code
  7. The Chain saved as a model.json file.

End to End Example

If you can't wait, check out the results of this example Langchain project for a preview of what's to come.

Install Dependencies

pip install comet_ml "langchain>=0.0.40" openai google-search-results spacy textstat pandas
python -m spacy download en_core_web_sm

Set your Comet Credentials

You can grab your Comet API key here

export COMET_API_KEY="<Your Comet API Key>"

Set Third Party Credentials

You will need an OpenAI API Key and a SerpAPI API Key to run the following examples

export OPENAI_API_KEY="Your OpenAI API Key"
export SERPAPI_API_KEY="Your SerpAPI API Key"

Run the example

from langchain.agents import initialize_agent, load_tools
from langchain.callbacks import CometCallbackHandler
from langchain.llms import OpenAI

comet_callback = CometCallbackHandler(
    project_name="comet-example-langchain",
    complexity_metrics=True,
    stream_logs=True,
    tags=["agent"],
)
callbacks = [comet_callback]
llm = OpenAI(temperature=0.9, callbacks=callbacks, verbose=True)

tools = load_tools(["serpapi", "llm-math"], llm=llm, callbacks=callbacks, verbose=True)
agent = initialize_agent(
    tools,
    llm,
    agent="zero-shot-react-description",
    callbacks=callbacks,
    verbose=True,
)
agent.run(
    "Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?"
)
comet_callback.flush_tracker(agent, finish=True)

Try it out!

Try out some examples for using Comet with Langchain in this Colab Notebook.

Open In Colab

Configure the CometCallbackHandler for Langchain

The Comet callback can be further customized to log spacy visualizations, text complexity metrics, and custom metrics.

Log Spacy Visualizations

To log visualizations of the generated text with Spacy, pass in a list of visualization names to the visualizations argument of the callback handler.

comet_callback = CometCallbackHandler(
    project_name="comet-example-langchain",
    stream_logs=True,
    tags=["llm"],
    visualizations=["dep", "ent"],
)

Log Text Complexity Metrics

Enable Logging Text Complexity Metrics to Comet via Textstat by setting the complexity_metrics argument to True

comet_callback = CometCallbackHandler(
    project_name="comet-example-langchain",
    complexity_metrics=True,
    stream_logs=True,
    tags=["llm"],
)

Log Custom Metrics

Comet supports logging custom metrics from Langchain by passing a callable to the custom_metrics argument. The callable must accept the following arguments:

  1. A Langchain Generation object.
  2. An integer argument for the prompt index, which tracks which input prompt was used to create the current output generation.
  3. An integer argument for the generation index, which tracks the index of the generated output when creating multiple generations per input prompt.
from rouge_score import rouge_scorer

from langchain.callbacks import CometCallbackHandler
from langchain.chains import LLMChain
from langchain.llms import OpenAI
from langchain.prompts import PromptTemplate


class Rouge:
    def __init__(self, reference):
        self.reference = reference
        self.scorer = rouge_scorer.RougeScorer(["rougeLsum"], use_stemmer=True)

    def compute_metric(self, generation, prompt_idx, gen_idx):
        prediction = generation.text
        results = self.scorer.score(target=self.reference, prediction=prediction)

        return {
            "rougeLsum_score": results["rougeLsum"].fmeasure,
            "reference": self.reference,
        }


reference = """
The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building.
It was the first structure to reach a height of 300 metres.

It is now taller than the Chrysler Building in New York City by 5.2 metres (17 ft)
Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France .
"""
rouge_score = Rouge(reference=reference)

template = """Given the following article, it is your job to write a summary.
Article:
{article}
Summary: This is the summary for the above article:"""
prompt_template = PromptTemplate(input_variables=["article"], template=template)

comet_callback = CometCallbackHandler(
    project_name="comet-example-langchain",
    complexity_metrics=False,
    stream_logs=True,
    tags=["custom_metrics"],
    custom_metrics=rouge_score.compute_metric,
)
callbacks = [comet_callback]
llm = OpenAI(temperature=0.9, callbacks=callbacks, verbose=True)

synopsis_chain = LLMChain(llm=llm, prompt=prompt_template, callbacks=callbacks, verbose=True)

test_prompts = [
    {
        "article": """
                 The tower is 324 metres (1,063 ft) tall, about the same height as
                 an 81-storey building, and the tallest structure in Paris. Its base is square,
                 measuring 125 metres (410 ft) on each side.
                 During its construction, the Eiffel Tower surpassed the
                 Washington Monument to become the tallest man-made structure in the world,
                 a title it held for 41 years until the Chrysler Building
                 in New York City was finished in 1930.

                 It was the first structure to reach a height of 300 metres.
                 Due to the addition of a broadcasting aerial at the top of the tower in 1957,
                 it is now taller than the Chrysler Building by 5.2 metres (17 ft).

                 Excluding transmitters, the Eiffel Tower is the second tallest
                 free-standing structure in France after the Millau Viaduct.
                 """
    }
]
print(synopsis_chain.apply(test_prompts))
comet_callback.flush_tracker(synopsis_chain, finish=True)
May. 24, 2023