{"id":9197,"date":"2024-02-21T06:00:44","date_gmt":"2024-02-21T14:00:44","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=9197"},"modified":"2025-04-24T17:03:12","modified_gmt":"2025-04-24T17:03:12","slug":"exploring-generative-artificial-intelligence-with-pandasai-openai-and-comet-llm","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/exploring-generative-artificial-intelligence-with-pandasai-openai-and-comet-llm\/","title":{"rendered":"Exploring Generative Artificial Intelligence with PandasAI, OpenAI, and Comet LLM"},"content":{"rendered":"\n<section class=\"section section--body\">\n<div class=\"section-divider\"><\/div>\n<div class=\"section-content\">\n<div class=\"section-inner sectionLayout--insetColumn\">\n<figure class=\"graf graf--figure\"><img loading=\"lazy\" decoding=\"async\" class=\"graf-image alignnone\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/1*iT6ZjzY7gN9DcyQDdfIMLQ.jpeg\" alt=\"pink and blue feathers\" width=\"800\" height=\"500\" data-image-id=\"1*iT6ZjzY7gN9DcyQDdfIMLQ.jpeg\" data-width=\"2400\" data-height=\"1500\" data-is-featured=\"true\"><figcaption class=\"imageCaption\">Photo by <a class=\"markup--anchor markup--figure-anchor\" href=\"https:\/\/unsplash.com\/@orwhat?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/unsplash.com\/@orwhat?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText\">Richard Horvath<\/a> on&nbsp;<a class=\"markup--anchor markup--figure-anchor\" href=\"https:\/\/unsplash.com\/photos\/WOA3QKFjlo8?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/unsplash.com\/photos\/WOA3QKFjlo8?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText\">Unsplash<\/a><\/figcaption><\/figure>\n<p>&nbsp;<\/p>\n<p class=\"graf graf--p\">Python&#8217;s Pandas library has long been the trusted companion of data analysts and data scientists. This library is known for its data manipulation, transformation, and wrangling capabilities. But what if I told you there&#8217;s a way to bring the power of Generative Artificial Intelligence to the Pandas library?<\/p>\n<p class=\"graf graf--p\">While researching LLMs (large language models), I came across PandasAI, a library that adds Generative AI capabilities to Pandas. This library will revolutionize how we can work with data by allowing us to turn complex queries into simple conversations. This article will dive into PandasAI, explore its capabilities, and how you can keep track of your model with Comet. We will also explore CometLLM.<\/p>\n<h3 class=\"graf graf--h3\">Overview of&nbsp;PandasAI<\/h3>\n<p class=\"graf graf--p\">PandasAI is a powerful Python library that uses generative AI models to make Pandas conversational. This lets you use natural language to execute Pandas&#8217; commands, like manipulating data frames and plotting a graph.<\/p>\n<p class=\"graf graf--p\">It is essential to know that while PandasAI is a game changer, it is designed to be used with Pandas and not replace it.<\/p>\n<h3 class=\"graf graf--h3\">Tools to be&nbsp;Used<\/h3>\n<ul class=\"postList\">\n<li class=\"graf graf--li\">Jupyter Notebook or Google Colab for experimentation<\/li>\n<\/ul>\n<h3 class=\"graf graf--h3\">Installation<\/h3>\n<p class=\"graf graf--p\">To get started, we will need to install the PandasAI. Run the command below:<\/p>\n<pre>pip install pandasai<\/pre>\n<p class=\"graf graf--p\">You might run into this error: &#8220;<em class=\"markup--em markup--p-em\">ERROR: Could not find a version that satisfies the requirement pandasai (from versions: none) ERROR: No matching distribution found for pandasai<\/em>&#8220;.<\/p>\n<p class=\"graf graf--p\">This is because PandasAI requires Python &gt;=3.9, &lt;4.0. To solve this, you need to upgrade your Python.<\/p>\n<figure class=\"graf graf--figure\"><img loading=\"lazy\" decoding=\"async\" class=\"graf-image alignnone\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/0*aXMFaBIRRH4jVk-I\" alt=\"code of installing PandasAI, generative artificial intelligence\" width=\"800\" height=\"259\" data-image-id=\"0*aXMFaBIRRH4jVk-I\" data-width=\"1600\" data-height=\"518\"><\/figure>\n<h3 class=\"graf graf--h3\">Getting Started with&nbsp;PandasAI<\/h3>\n<p class=\"graf graf--p\">LLMs power PandasAI and support several large language models (LLMs), from OpenAI, Azure OpenAI, and Google PaLM to HuggingFace&#8217;s Starcoder and Falcon models. These models are essential to give PandasAI its natural language query capabilities.<\/p>\n<p class=\"graf graf--p\">We must use OpenAI LLM API Wrapper for this tutorial to power PandasAI&#8217;s generative artificial intelligence capabilities. We will need to set up an <a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/auth0.openai.com\/u\/signup\/identifier?state=hKFo2SBYanJCYndwNWZSN3RrQUlfNFBzbTJVU01MbjlScEFTVKFur3VuaXZlcnNhbC1sb2dpbqN0aWTZIE45ZGpYS2Jna2FDYl9ta3BPNVM1em52Rm9DeG1FZVBuo2NpZNkgRFJpdnNubTJNdTQyVDNLT3BxZHR3QjNOWXZpSFl6d0Q\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/auth0.openai.com\/u\/signup\/identifier?state=hKFo2SBYanJCYndwNWZSN3RrQUlfNFBzbTJVU01MbjlScEFTVKFur3VuaXZlcnNhbC1sb2dpbqN0aWTZIE45ZGpYS2Jna2FDYl9ta3BPNVM1em52Rm9DeG1FZVBuo2NpZNkgRFJpdnNubTJNdTQyVDNLT3BxZHR3QjNOWXZpSFl6d0Q\">OpenAI<\/a> account and generate an OpenAI API token key, which you can find on your <a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/platform.openai.com\/account\/api-keys\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/platform.openai.com\/account\/api-keys\">account here<\/a>. You should <a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/platform.openai.com\/docs\/quickstart\/pricing\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/platform.openai.com\/docs\/quickstart\/pricing\">set up billing<\/a> since OpenAI access is a paid service; I will show you how to monitor your billing later in the article.<\/p>\n<pre>from pandasai import PandasAI\nfrom pandasai.llm import OpenAI\nllm = OpenAI(api_token=\"*******\")<\/pre>\n<blockquote class=\"graf graf--blockquote graf--hasDropCapModel\"><p><strong class=\"markup--strong markup--blockquote-strong\">Note: <\/strong>******* represents your OpenAI API token key. You can find on your <a class=\"markup--anchor markup--blockquote-anchor\" href=\"https:\/\/platform.openai.com\/account\/api-keys\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/platform.openai.com\/account\/api-keys\">account here<\/a>.<\/p><\/blockquote>\n<h3 class=\"graf graf--h3\">Our Dataset<\/h3>\n<p class=\"graf graf--p\">I will use the <a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/www.kaggle.com\/datasets\/iamsouravbanerjee\/house-rent-prediction-dataset\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/www.kaggle.com\/datasets\/iamsouravbanerjee\/house-rent-prediction-dataset\">house rent prediction dataset<\/a> I found on Kaggle for this article. The dataset contains information like the number of bedrooms, kitchens, the rent, city, area, and furnishing status of over 4,000 houses, apartments, and flats in India.<\/p>\n<p class=\"graf graf--p\">We will now load our dataset using the Pandas library.<\/p>\n<pre>import pandas as pd\n# House rent dataset link: https:\/\/www.kaggle.com\/datasets\/iamsouravbanerjee\/house-rent-prediction-dataset\ndf = pd. read_csv('\/content\/House_Rent_Dataset.csv')\ndf.head(5)<\/pre>\n<h3 class=\"graf graf--h3\">Exploring Our Dataset with PandasAI and&nbsp;OpenAI<\/h3>\n<p class=\"graf graf--p\">We can now explore the dataset with PandasAI&#8217;s generative artificial intelligence capabilities. Just as Pandas has <a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/pandas.pydata.org\/docs\/reference\/api\/pandas.DataFrame.html\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/pandas.pydata.org\/docs\/reference\/api\/pandas.DataFrame.html\">dataframes<\/a>, PandasAI has SmartDataframes.<\/p>\n<p class=\"graf graf--p\">SmartDataframe has the same properties as <code class=\"markup--code markup--p-code\">pd.DataFrame<\/code> but with conversational features.<\/p>\n<pre>from pandasai import SmartDataframe\nsdf = SmartDataframe(df, config={\"llm\": llm})<\/pre>\n<p class=\"graf graf--p\">Now, let&#8217;s explore PandasAI.<\/p>\n<pre class=\"graf graf--pre graf--preV2\" spellcheck=\"false\" data-code-block-mode=\"2\" data-code-block-lang=\"python\"><span class=\"pre--content\">sdf.impute_missing_values()<\/span><\/pre>\n<p class=\"graf graf--p\">This command will impute missing values in your data frame.<\/p>\n<figure class=\"graf graf--figure\"><img loading=\"lazy\" decoding=\"async\" class=\"graf-image alignnone\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/0*nNvfXlrTlIXuGUEn\" alt=\"data of different rents\" width=\"800\" height=\"289\" data-image-id=\"0*nNvfXlrTlIXuGUEn\" data-width=\"1485\" data-height=\"537\"><\/figure>\n<p class=\"graf graf--p\">To ask your data questions using PandasAI, you use <code class=\"markup--code markup--p-code\">sdf.chat<\/code><\/p>\n<pre class=\"graf graf--pre graf--preV2\" spellcheck=\"false\" data-code-block-mode=\"2\" data-code-block-lang=\"python\"><span class=\"pre--content\">sdf.chat(\u201cwho <span class=\"hljs-keyword\">is<\/span> the ideal tenant <span class=\"hljs-keyword\">for<\/span> a <span class=\"hljs-number\">3<\/span> bedroom <span class=\"hljs-keyword\">in<\/span> kolkata\u201d)<\/span><\/pre>\n<figure class=\"graf graf--figure\"><img decoding=\"async\" class=\"graf-image\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/0*cjo-KkFk8rV4F5pz\" data-image-id=\"0*cjo-KkFk8rV4F5pz\" data-width=\"725\" data-height=\"63\"><\/figure>\n<pre class=\"graf graf--pre graf--preV2\" spellcheck=\"false\" data-code-block-mode=\"2\" data-code-block-lang=\"python\"><span class=\"pre--content\">sdf.chat(\u201cReturn the top <span class=\"hljs-number\">5<\/span> expensive city by rent\u201d)<\/span><\/pre>\n<figure class=\"graf graf--figure\"><img loading=\"lazy\" decoding=\"async\" class=\"graf-image alignnone\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/0*zzAXt4TqhAxAx_ig\" alt=\"table of city and rents\" width=\"621\" height=\"277\" data-image-id=\"0*zzAXt4TqhAxAx_ig\" data-width=\"621\" data-height=\"277\"><\/figure>\n<pre class=\"graf graf--pre graf--preV2\" spellcheck=\"false\" data-code-block-mode=\"2\" data-code-block-lang=\"python\"><span class=\"pre--content\">sdf.chat(\u201cIn a table show me the average rent <span class=\"hljs-keyword\">in<\/span> the various cities each month <span class=\"hljs-keyword\">and<\/span> group this data by the BHK\u201d)\n<span class=\"hljs-comment\">#BHK here represent numbers of bedrooms, hall, and kitchen<\/span><\/span><\/pre>\n<figure class=\"graf graf--figure\"><img loading=\"lazy\" decoding=\"async\" class=\"graf-image alignnone\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/0*cB41Y5rKuyS8UCKq\" alt=\"chart of cities and months\" width=\"800\" height=\"120\" data-image-id=\"0*cB41Y5rKuyS8UCKq\" data-width=\"1600\" data-height=\"240\"><\/figure>\n<pre class=\"graf graf--pre graf--preV2\" spellcheck=\"false\" data-code-block-mode=\"2\" data-code-block-lang=\"python\"><span class=\"pre--content\">sdf.chat(\u201cIn a table show me the number of Point of Contact <span class=\"hljs-keyword\">in<\/span> the various cities each month <span class=\"hljs-keyword\">and<\/span> group this data by the Point of Contact\u201d)<\/span><\/pre>\n<figure class=\"graf graf--figure\"><img loading=\"lazy\" decoding=\"async\" class=\"graf-image alignnone\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/0*XO8gZ5wv-2GQAOwI\" alt=\"chart of contacts by year\" width=\"800\" height=\"166\" data-image-id=\"0*XO8gZ5wv-2GQAOwI\" data-width=\"1558\" data-height=\"323\"><\/figure>\n<p class=\"graf graf--p\">You can also generate charts with PandasAI:<\/p>\n<pre class=\"graf graf--pre graf--preV2\" spellcheck=\"false\" data-code-block-mode=\"2\" data-code-block-lang=\"python\"><span class=\"pre--content\">sdf.chat(\u201cPlot a chart of the average rent by city\u201d)<\/span><\/pre>\n<figure class=\"graf graf--figure\"><img loading=\"lazy\" decoding=\"async\" class=\"graf-image alignnone\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/0*04vtoaXxTRQPIYyr\" alt=\"bar graph of rent by city\" width=\"800\" height=\"447\" data-image-id=\"0*04vtoaXxTRQPIYyr\" data-width=\"1110\" data-height=\"620\"><\/figure>\n<pre class=\"graf graf--pre graf--preV2\" spellcheck=\"false\" data-code-block-mode=\"2\" data-code-block-lang=\"python\"><span class=\"pre--content\">sdf.chat(\u201cCreate a line chart to show the trend of rent by city <span class=\"hljs-keyword\">in<\/span> the last few months\u201d)<\/span><\/pre>\n<figure class=\"graf graf--figure\"><img loading=\"lazy\" decoding=\"async\" class=\"graf-image alignnone\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/0*HwrU09t-qh5TQIPM\" alt=\"graph of rent by city\" width=\"800\" height=\"338\" data-image-id=\"0*HwrU09t-qh5TQIPM\" data-width=\"986\" data-height=\"416\"><\/figure>\n<pre class=\"graf graf--pre graf--preV2\" spellcheck=\"false\" data-code-block-mode=\"2\" data-code-block-lang=\"python\"><span class=\"pre--content\">sdf.plot_correlation_heatmap()<\/span><\/pre>\n<figure class=\"graf graf--figure\"><img loading=\"lazy\" decoding=\"async\" class=\"graf-image alignnone\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/0*UE4l4xypH1hp8wi0\" alt=\"correlation heat map\" width=\"800\" height=\"475\" data-image-id=\"0*UE4l4xypH1hp8wi0\" data-width=\"1163\" data-height=\"691\"><\/figure>\n<pre class=\"graf graf--pre graf--preV2\" spellcheck=\"false\" data-code-block-mode=\"2\" data-code-block-lang=\"python\"><span class=\"pre--content\">sdf.chat(\u201cCalculate the average cost of rent <span class=\"hljs-keyword\">in<\/span> Delhi\u201d)<\/span><\/pre>\n<figure class=\"graf graf--figure\"><img decoding=\"async\" class=\"graf-image\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/0*hvZjAaYb2IPeJ48K\" data-image-id=\"0*hvZjAaYb2IPeJ48K\" data-width=\"741\" data-height=\"67\"><\/figure>\n<p class=\"graf graf--p\">Here is a GitHub gist of the code snippets.<\/p>\n<pre># This command will impute missing values in your data frame.\nsdf.impute_missing_values()\n\n# To ask your data questions using PandaAI, you use sdf.chat\n\nsdf.chat(\"who is the ideal tenant for a 3 bedroom in kolkata\")\n\nsdf.chat(\"Return the top 5 expensive city by rent\")\n\nsdf.chat(\"In a table show me the average rent in the various cities each month and group this data by the BHK\") #BHK here represent number of bedrooms, hall, and kitchen\n\nsdf.chat(\"In a table show me the number of Point of Contact in the various cities each month and group this data by the Point of Contact\")\n\n# You can also generate charts with PandaAI;\n\nsdf.chat(\"Plot a chart of the average rent by city\")\n\nsdf.plot_correlation_heatmap()\n\nsdf.chat(\"Create a line chart to show the trend of rent  by city in the last few months\")\n\nsdf.chat(\"Calculate the average cost of rent in Delhi\")<\/pre>\n<h3 class=\"graf graf--h3\">Worried about the Billing? Count Your&nbsp;Tokens<\/h3>\n<p class=\"graf graf--p\">OpenAI&#8217;s API process and break down your prompts into tokens. You can think of them as pieces of words and characters. You should visit the <a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/help.openai.com\/en\/articles\/4936856-what-are-tokens-and-how-to-count-them\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/help.openai.com\/en\/articles\/4936856-what-are-tokens-and-how-to-count-them\">official documentation<\/a> to learn more.<\/p>\n<p class=\"graf graf--p\">You can analyze and count the number of tokens your prompt uses using the command below.<\/p>\n<pre>from pandasai.helpers.openai_info import get_openai_callback\nwith get_openai_callback() as cb:\nresponse = sdf.chat(\"Calculate the average cost of rent in Delhi\")\nprint(response)\nprint(cb)<\/pre>\n<figure class=\"graf graf--figure\"><img decoding=\"async\" class=\"graf-image\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/0*6SFyZ3gxP1Zby19l\" data-image-id=\"0*6SFyZ3gxP1Zby19l\" data-width=\"738\" data-height=\"158\"><\/figure>\n<p class=\"graf graf--p\">You can also check out your usage on OpenAI <a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/platform.openai.com\/account\/usage\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/platform.openai.com\/account\/usage\">here<\/a>.<\/p>\n<h3 class=\"graf graf--h3\">Logging your <strong class=\"markup--strong markup--h3-strong\">Artifacts on&nbsp;<\/strong>Comet<\/h3>\n<p class=\"graf graf--p\">Once done, you should log the dataset (Comet calls these <strong class=\"markup--strong markup--p-strong\">Artifacts) <\/strong>to Comet. This way, when working and building machine learning models, you can automate and keep track of your code and artifacts.<\/p>\n<p class=\"graf graf--p\">To get started, we will need to create a <a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/www.comet.com\/site\/\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/www.comet.com\/site\/\">Comet account<\/a>. This will enable us to log our Artifacts on the Comet platform through an &#8220;experiment.&#8221; Let&#8217;s log a new experiment with our <a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/www.kaggle.com\/datasets\/iamsouravbanerjee\/house-rent-prediction-dataset\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/www.kaggle.com\/datasets\/iamsouravbanerjee\/house-rent-prediction-dataset\">house rent prediction dataset<\/a> using the code below:<\/p>\n<pre class=\"graf graf--pre graf--preV2\" spellcheck=\"false\" data-code-block-mode=\"1\" data-code-block-lang=\"python\"><span class=\"pre--content\"><span class=\"hljs-keyword\">import<\/span> comet_ml\n<span class=\"hljs-keyword\">from<\/span> comet_ml <span class=\"hljs-keyword\">import<\/span> Artifact, Experiment\n<span class=\"hljs-comment\">#Initialize comet instance for API Key<\/span>\ncomet_ml.login()<\/span><\/pre>\n<p class=\"graf graf--p\">Here, we imported and initialized the CometML library. Once you run this, you will be prompted to pass your <a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/www.comet.com\/account-settings\/apiKeys\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/www.comet.com\/account-settings\/apiKeys\">Comet API key<\/a> into Colab or Jupyter.<\/p>\n<p class=\"graf graf--p\">Next, we create an Experiment object by giving it a name and the workspace it should belong to. You can get all your available workspaces <a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/www.comet.com\/account-settings\/workspaces\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/www.comet.com\/account-settings\/workspaces\">here<\/a>. After that, we will create an Artifact instance by giving it a name, Artifact type and specifying the file path with the <code class=\"markup--code markup--p-code\">artifact.add()<\/code>. Then, we end the experiment.<\/p>\n<p class=\"graf graf--p\">You can do that using the code below:<\/p>\n<pre># Install and import the Comet Library:\nimport comet_ml\nfrom comet_ml import Artifact, Experiment\n\n# Initialize comet instance for API Key. You can access your API Key here: https:\/\/www.comet.com\/account-settings\/apiKeys\ncomet_ml.login()\n\n# Create an Experiment object by giving it a name and the workspace it should belong to.\n# You can get all your available workspaces here: https:\/\/www.comet.com\/account-settings\/workspaces\n\nexperiment = Experiment(project_name=\"Rent Experiment\", workspace=\"bennykillua\")\n\n# Initialize Artifact by giving it a name, artifact_type, and specifying the file path with the artifact.add().\n\nartifact = Artifact(name=\"HouseRent\", artifact_type=\"dataset\")\n\n#Specify the path of the artifact\n\nartifact.add(r\"\/content\/House_Rent_Dataset.csv\")\nWe can now log the artifact to the Comet platform.\n\n#log artifact to experiment\nexperiment.log_artifact(artifact)\n\n#end the experiment\nexperiment.end()<\/pre>\n<h3 class=\"graf graf--h3\">Logging with&nbsp;CometLLM<\/h3>\n<p class=\"graf graf--p\">However, since this is an LLM project, it will be best to log our prompt using <a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/github.com\/comet-ml\/comet-llm\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/github.com\/comet-ml\/comet-llm\">CometLLM<\/a>.<\/p>\n<blockquote class=\"graf graf--blockquote\"><p>CometLLM is a new suite of LLMOps tools designed to help you effortlessly track and visualize your LLM prompts and chains.<\/p><\/blockquote>\n<p class=\"graf graf--p\">First, we will need to install the Comet Library:<\/p>\n<pre class=\"graf graf--pre graf--preV2\" spellcheck=\"false\" data-code-block-mode=\"2\" data-code-block-lang=\"python\"><span class=\"pre--content\">pip install comet_llm<\/span><\/pre>\n<p class=\"graf graf--p\">Then, we will also need our API key, which we can get from our <a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/www.comet.com\/site\/\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/www.comet.com\/site\/\">Comet account<\/a>. Once you have that, we can log our prompts and their outputs to Comet.<\/p>\n<p class=\"graf graf--p\">We can test CometLLM by running the code below:<\/p>\n<pre class=\"graf graf--pre graf--preV2\" spellcheck=\"false\" data-code-block-mode=\"2\" data-code-block-lang=\"python\"><span class=\"pre--content\"><span class=\"hljs-keyword\">import<\/span> comet_llm\n\ncomet_llm.log_prompt(\n    prompt=<span class=\"hljs-string\">\"What is your name?\"<\/span>,\n    output=<span class=\"hljs-string\">\" My name is Benny Ifeanyi\"<\/span>,\n    api_key=<span class=\"hljs-string\">\"YOUR_COMET_API_KEY\"<\/span>,\n    project = <span class=\"hljs-string\">\"MY_Project_\"<\/span>,\n)<\/span><\/pre>\n<p class=\"graf graf--p\">This is the output. You can view it <a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/www.comet.com\/bennykillua\/MY_Project_\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/www.comet.com\/bennykillua\/MY_Project_\">on the Comet platform<\/a> as well.<\/p>\n<figure class=\"graf graf--figure\"><img loading=\"lazy\" decoding=\"async\" class=\"graf-image alignnone\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/1*CidQfiJDPXGd4XGtN93w3g.png\" alt=\"comet screenshot platform\" width=\"800\" height=\"390\" data-image-id=\"1*CidQfiJDPXGd4XGtN93w3g.png\" data-width=\"1363\" data-height=\"665\"><\/figure>\n<p class=\"graf graf--p\">Now, let&#8217;s log our prompt and their output. We can do that using the code below:<\/p>\n<pre class=\"graf graf--pre graf--preV2\" spellcheck=\"false\" data-code-block-mode=\"2\" data-code-block-lang=\"python\"><span class=\"pre--content\"><span class=\"hljs-keyword\">import<\/span> comet_llm\n<span class=\"hljs-keyword\">import<\/span> os\n\n\n<span class=\"hljs-comment\"># Define your questions<\/span>\nquestions = [\n    <span class=\"hljs-string\">\"Return the top 5 expensive city by rent\"<\/span>\n]\n\n<span class=\"hljs-comment\"># Create a list to store question-response pairs<\/span>\nquestion_response_pairs = []\n\n<span class=\"hljs-comment\"># Log the questions and responses to CometLLM and store them<\/span>\n<span class=\"hljs-keyword\">for<\/span> question <span class=\"hljs-keyword\">in<\/span> questions:\n    <span class=\"hljs-comment\"># Generate the response using sdf.chat (this is the PandaAI model we created earlier)<\/span>\n    response = sdf.chat(question)  <span class=\"hljs-comment\"># Response is a string<\/span>\n\n    <span class=\"hljs-comment\"># Store the question and response in the list<\/span>\n    question_response_pairs.append({<span class=\"hljs-string\">\"question\"<\/span>: question, <span class=\"hljs-string\">\"response\"<\/span>: response})\n\n<span class=\"hljs-comment\"># Save the question-response pairs to a CSV file<\/span>\ncsv_file_path = <span class=\"hljs-string\">'\/content\/question_response_pairs.csv'<\/span>\npairs_df = pd.DataFrame(question_response_pairs)\npairs_df.to_csv(csv_file_path, index=<span class=\"hljs-literal\">False<\/span>)\n\n<span class=\"hljs-keyword\">with<\/span> <span class=\"hljs-built_in\">open<\/span>(csv_file_path, <span class=\"hljs-string\">'r'<\/span>) <span class=\"hljs-keyword\">as<\/span> csv_file:\n    csv_content = csv_file.read()\n\n<span class=\"hljs-comment\"># Log the entire CSV content as the response to CometLLM<\/span>\ncomet_llm.log_prompt(\n    prompt=<span class=\"hljs-string\">\"Question-Response Pairs\"<\/span>,\n    output=csv_content,\n    api_key=<span class=\"hljs-string\">\"YOUR_COMET_API_KEY\"<\/span>,\n    project=<span class=\"hljs-string\">\"MY_Project_\"<\/span>,\n)<\/span><\/pre>\n<p class=\"graf graf--p\">This is the output. You can also view it <a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/www.comet.com\/bennykillua\/my-project\/prompts\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/www.comet.com\/bennykillua\/my-project\/prompts\">on the Comet Platform<\/a>.<\/p>\n<figure class=\"graf graf--figure\"><img loading=\"lazy\" decoding=\"async\" class=\"graf-image alignnone\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/1*vw_thT4aMeQnLD0O5Q7dJw.png\" alt=\"comet platform screenshot\" width=\"800\" height=\"400\" data-image-id=\"1*vw_thT4aMeQnLD0O5Q7dJw.png\" data-width=\"1449\" data-height=\"725\"><\/figure>\n<p class=\"graf graf--p\">However, we can&#8217;t log our dataset with CometLLM right now because, at the moment, this isn&#8217;t supported.<\/p>\n<h3 class=\"graf graf--h3\">What&#8217;s next?<\/h3>\n<p class=\"graf graf--p\">From here, we can do much more, like using the Comet UI to score our prompt outputs or to search from specific prompts. This is helpful when working with thousands of prompts.<\/p>\n<p class=\"graf graf--p\">All of these and much more were covered in this article: <a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/heartbeat.comet.ml\/organize-your-prompt-engineering-with-cometllm-66e390ef6645\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/heartbeat.comet.ml\/organize-your-prompt-engineering-with-cometllm-66e390ef6645\">Organize Your Prompt Engineering with CometLLM<\/a>.<\/p>\n<h3 class=\"graf graf--h3\">Conclusion<\/h3>\n<p class=\"graf graf--p\">PandasAI has brought generative artificial intelligence capabilities to the Pandas library. You can explore these capabilities on your datasets and start exciting projects with other LLM models like Azure OpenAI and LangChain (<em class=\"markup--em markup--p-em\">which you can view in the GitHub gist below<\/em>). Lastly, remember to track, compare, and optimize your ML experiments as you work with Comet.<\/p>\n<p class=\"graf graf--p\"><strong class=\"markup--strong markup--p-strong\">P.S.<\/strong> If you prefer to learn by code, check out this <a class=\"markup--anchor markup--p-anchor\" href=\"https:\/\/gist.github.com\/Bennykillua\/1e8942e5728338456776cd65677f35ff\" target=\"_blank\" rel=\"noopener\" data-href=\"https:\/\/gist.github.com\/Bennykillua\/1e8942e5728338456776cd65677f35ff\">Github gist<\/a>, which hosts an informative Google Colab notebook with all the code snippets.<\/p>\n<\/div>\n<\/div>\n<\/section>\n","protected":false},"excerpt":{"rendered":"<p>Photo by Richard Horvath on&nbsp;Unsplash &nbsp; Python&#8217;s Pandas library has long been the trusted companion of data analysts and data scientists. This library is known for its data manipulation, transformation, and wrangling capabilities. But what if I told you there&#8217;s a way to bring the power of Generative Artificial Intelligence to the Pandas library? While [&hellip;]<\/p>\n","protected":false},"author":90,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[65,7],"tags":[],"coauthors":[187],"class_list":["post-9197","post","type-post","status-publish","format-standard","hentry","category-llmops","category-tutorials"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v25.9 (Yoast SEO v25.9) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Generative Artificial Intelligence with PandasAI &amp; CometLLM<\/title>\n<meta name=\"description\" content=\"Dive into generative artificial intelligence with PandasAI and how you can keep track of your model with CometLLM. Learn more.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/exploring-generative-artificial-intelligence-with-pandasai-openai-and-comet-llm\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Exploring Generative Artificial Intelligence with PandasAI, OpenAI, and Comet LLM\" \/>\n<meta property=\"og:description\" content=\"Dive into generative artificial intelligence with PandasAI and how you can keep track of your model with CometLLM. Learn more.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/exploring-generative-artificial-intelligence-with-pandasai-openai-and-comet-llm\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2024-02-21T14:00:44+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-24T17:03:12+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/cdn-images-1.medium.com\/max\/800\/1*iT6ZjzY7gN9DcyQDdfIMLQ.jpeg\" \/>\n<meta name=\"author\" content=\"Benny Ifeanyi Iheagwara\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Benny Ifeanyi Iheagwara\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"10 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Generative Artificial Intelligence with PandasAI & CometLLM","description":"Dive into generative artificial intelligence with PandasAI and how you can keep track of your model with CometLLM. Learn more.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/exploring-generative-artificial-intelligence-with-pandasai-openai-and-comet-llm","og_locale":"en_US","og_type":"article","og_title":"Exploring Generative Artificial Intelligence with PandasAI, OpenAI, and Comet LLM","og_description":"Dive into generative artificial intelligence with PandasAI and how you can keep track of your model with CometLLM. Learn more.","og_url":"https:\/\/www.comet.com\/site\/blog\/exploring-generative-artificial-intelligence-with-pandasai-openai-and-comet-llm","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2024-02-21T14:00:44+00:00","article_modified_time":"2025-04-24T17:03:12+00:00","og_image":[{"url":"https:\/\/cdn-images-1.medium.com\/max\/800\/1*iT6ZjzY7gN9DcyQDdfIMLQ.jpeg","type":"","width":"","height":""}],"author":"Benny Ifeanyi Iheagwara","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Benny Ifeanyi Iheagwara","Est. reading time":"10 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/exploring-generative-artificial-intelligence-with-pandasai-openai-and-comet-llm#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/exploring-generative-artificial-intelligence-with-pandasai-openai-and-comet-llm\/"},"author":{"name":"Benny Ifeanyi Iheagwara","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/71e200e652e19c10311a4dacf3891cb1"},"headline":"Exploring Generative Artificial Intelligence with PandasAI, OpenAI, and Comet LLM","datePublished":"2024-02-21T14:00:44+00:00","dateModified":"2025-04-24T17:03:12+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/exploring-generative-artificial-intelligence-with-pandasai-openai-and-comet-llm\/"},"wordCount":1038,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/exploring-generative-artificial-intelligence-with-pandasai-openai-and-comet-llm#primaryimage"},"thumbnailUrl":"https:\/\/cdn-images-1.medium.com\/max\/800\/1*iT6ZjzY7gN9DcyQDdfIMLQ.jpeg","articleSection":["LLMOps","Tutorials"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/exploring-generative-artificial-intelligence-with-pandasai-openai-and-comet-llm\/","url":"https:\/\/www.comet.com\/site\/blog\/exploring-generative-artificial-intelligence-with-pandasai-openai-and-comet-llm","name":"Generative Artificial Intelligence with PandasAI & CometLLM","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/exploring-generative-artificial-intelligence-with-pandasai-openai-and-comet-llm#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/exploring-generative-artificial-intelligence-with-pandasai-openai-and-comet-llm#primaryimage"},"thumbnailUrl":"https:\/\/cdn-images-1.medium.com\/max\/800\/1*iT6ZjzY7gN9DcyQDdfIMLQ.jpeg","datePublished":"2024-02-21T14:00:44+00:00","dateModified":"2025-04-24T17:03:12+00:00","description":"Dive into generative artificial intelligence with PandasAI and how you can keep track of your model with CometLLM. Learn more.","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/exploring-generative-artificial-intelligence-with-pandasai-openai-and-comet-llm#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/exploring-generative-artificial-intelligence-with-pandasai-openai-and-comet-llm"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/exploring-generative-artificial-intelligence-with-pandasai-openai-and-comet-llm#primaryimage","url":"https:\/\/cdn-images-1.medium.com\/max\/800\/1*iT6ZjzY7gN9DcyQDdfIMLQ.jpeg","contentUrl":"https:\/\/cdn-images-1.medium.com\/max\/800\/1*iT6ZjzY7gN9DcyQDdfIMLQ.jpeg"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/exploring-generative-artificial-intelligence-with-pandasai-openai-and-comet-llm#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Exploring Generative Artificial Intelligence with PandasAI, OpenAI, and Comet LLM"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/71e200e652e19c10311a4dacf3891cb1","name":"Benny Ifeanyi Iheagwara","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/f1d1f429361deaae05413115cd5914d4","url":"https:\/\/secure.gravatar.com\/avatar\/0b222798a274a8dee5e2394af804c5e76ba2fa8e5708ef6a35c5549a7d2794f2?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/0b222798a274a8dee5e2394af804c5e76ba2fa8e5708ef6a35c5549a7d2794f2?s=96&d=mm&r=g","caption":"Benny Ifeanyi Iheagwara"},"url":"https:\/\/www.comet.com\/site\/blog\/author\/iheifeanyigmail-com\/"}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/9197","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/90"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=9197"}],"version-history":[{"count":1,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/9197\/revisions"}],"predecessor-version":[{"id":15385,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/9197\/revisions\/15385"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=9197"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=9197"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=9197"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=9197"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}