Integrate with Langchain¶
LangChain is a framework for developing applications powered by language models. Langchain provides APIs to build data aware, agentic applications that allow language models to incorporate external data sources and interact with their compute environments.
Start logging¶
- Import the
CometCallbackHandler
into your script - Add the callback to Langchain's
CallbackManager
. - Pass the
callback_manager
parameter to your language model, chain, or agent. Also make sure to pass the parameterverbose=True
to enable logging.
from langchain.callbacks import CometCallbackHandler
from langchain.llms import OpenAI
comet_callback = CometCallbackHandler(
project_name="comet-example-langchain",
complexity_metrics=True,
stream_logs=True,
tags=["llm"],
visualizations=["dep"],
)
llm = OpenAI(temperature=0.9, callbacks=[comet_callback], verbose=True)
llm_result = llm.generate(["Tell me a joke", "Tell me a poem", "Tell me a fact"])
print("LLM result", llm_result)
comet_callback.flush_tracker(llm, finish=True)
Log automatically¶
Comet will log the following information from your Langchain Runs
- Language Model Parameters
- Input Text Prompts
- Generated Output Text
- A Session Dataframe that maps input prompts to their generated outputs
- Action Records Log. Comet will log every action step executed in the chain, which can then be visualized using Comet's LLM Panels.
- Source Code
- The Chain saved as a
model.json
file.
End to End Example¶
If you can't wait, check out the results of this example Langchain project for a preview of what's to come.
Install Dependencies¶
pip install comet_ml "langchain>=0.0.40" openai google-search-results spacy textstat pandas
python -m spacy download en_core_web_sm
Set your Comet Credentials¶
You can grab your Comet API key here
export COMET_API_KEY="<Your Comet API Key>"
Set Third Party Credentials¶
You will need an OpenAI API Key and a SerpAPI API Key to run the following examples
export OPENAI_API_KEY="Your OpenAI API Key"
export SERPAPI_API_KEY="Your SerpAPI API Key"
Run the example¶
from langchain.agents import initialize_agent, load_tools
from langchain.callbacks import CometCallbackHandler
from langchain.llms import OpenAI
comet_callback = CometCallbackHandler(
project_name="comet-example-langchain",
complexity_metrics=True,
stream_logs=True,
tags=["agent"],
)
callbacks = [comet_callback]
llm = OpenAI(temperature=0.9, callbacks=callbacks, verbose=True)
tools = load_tools(["serpapi", "llm-math"], llm=llm, callbacks=callbacks, verbose=True)
agent = initialize_agent(
tools,
llm,
agent="zero-shot-react-description",
callbacks=callbacks,
verbose=True,
)
agent.run(
"Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?"
)
comet_callback.flush_tracker(agent, finish=True)
Try it out!¶
Try out some examples for using Comet with Langchain in this Colab Notebook.
Configure the CometCallbackHandler
for Langchain¶
The Comet callback can be further customized to log spacy visualizations, text complexity metrics, and custom metrics.
Log Spacy Visualizations¶
To log visualizations of the generated text with Spacy, pass in a list of visualization names to the visualizations
argument of the callback handler.
comet_callback = CometCallbackHandler(
project_name="comet-example-langchain",
stream_logs=True,
tags=["llm"],
visualizations=["dep", "ent"],
)
Log Text Complexity Metrics¶
Enable Logging Text Complexity Metrics to Comet via Textstat by setting the complexity_metrics
argument to True
comet_callback = CometCallbackHandler(
project_name="comet-example-langchain",
complexity_metrics=True,
stream_logs=True,
tags=["llm"],
)
Log Custom Metrics¶
Comet supports logging custom metrics from Langchain by passing a callable to the custom_metrics
argument. The callable must accept the following arguments:
- A Langchain
Generation
object. - An integer argument for the prompt index, which tracks which input prompt was used to create the current output generation.
- An integer argument for the generation index, which tracks the index of the generated output when creating multiple generations per input prompt.
from rouge_score import rouge_scorer
from langchain.callbacks import CometCallbackHandler
from langchain.chains import LLMChain
from langchain.llms import OpenAI
from langchain.prompts import PromptTemplate
class Rouge:
def __init__(self, reference):
self.reference = reference
self.scorer = rouge_scorer.RougeScorer(["rougeLsum"], use_stemmer=True)
def compute_metric(self, generation, prompt_idx, gen_idx):
prediction = generation.text
results = self.scorer.score(target=self.reference, prediction=prediction)
return {
"rougeLsum_score": results["rougeLsum"].fmeasure,
"reference": self.reference,
}
reference = """
The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building.
It was the first structure to reach a height of 300 metres.
It is now taller than the Chrysler Building in New York City by 5.2 metres (17 ft)
Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France .
"""
rouge_score = Rouge(reference=reference)
template = """Given the following article, it is your job to write a summary.
Article:
{article}
Summary: This is the summary for the above article:"""
prompt_template = PromptTemplate(input_variables=["article"], template=template)
comet_callback = CometCallbackHandler(
project_name="comet-example-langchain",
complexity_metrics=False,
stream_logs=True,
tags=["custom_metrics"],
custom_metrics=rouge_score.compute_metric,
)
callbacks = [comet_callback]
llm = OpenAI(temperature=0.9, callbacks=callbacks, verbose=True)
synopsis_chain = LLMChain(llm=llm, prompt=prompt_template, callbacks=callbacks, verbose=True)
test_prompts = [
{
"article": """
The tower is 324 metres (1,063 ft) tall, about the same height as
an 81-storey building, and the tallest structure in Paris. Its base is square,
measuring 125 metres (410 ft) on each side.
During its construction, the Eiffel Tower surpassed the
Washington Monument to become the tallest man-made structure in the world,
a title it held for 41 years until the Chrysler Building
in New York City was finished in 1930.
It was the first structure to reach a height of 300 metres.
Due to the addition of a broadcasting aerial at the top of the tower in 1957,
it is now taller than the Chrysler Building by 5.2 metres (17 ft).
Excluding transmitters, the Eiffel Tower is the second tallest
free-standing structure in France after the Millau Viaduct.
"""
}
]
print(synopsis_chain.apply(test_prompts))
comet_callback.flush_tracker(synopsis_chain, finish=True)