{"id":11966,"date":"2024-11-21T14:33:47","date_gmt":"2024-11-21T22:33:47","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=11966"},"modified":"2025-11-17T21:20:19","modified_gmt":"2025-11-17T21:20:19","slug":"perplexity-for-llm-evaluation","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/perplexity-for-llm-evaluation\/","title":{"rendered":"Perplexity for LLM Evaluation"},"content":{"rendered":"\n
\n
Follow along with the Colab!<\/a><\/div>\n<\/div>\n\n\n\n

Perplexity is, historically speaking, one of the “standard” evaluation metrics for language models. And while recent years have seen a surge in more complex and robust metrics, including LLM-based evaluations, perplexity still has a lot of value as a component in your evaluation suite. If you want to build effective evaluation pipelines\u2014or just understand what researchers mean when they report perplexity scores\u2014you need to have a grasp on what perplexity is and how it can be implemented.<\/span><\/p>\n\n\n\n

\"Perplexity<\/figure>\n\n\n\n

Perplexity seeks to quantify the “uncertainty” a model experiences when when predicting the next token in a sequence. High uncertainty occurs when the model is unsure about the next word or token in a sequence. This can happen when the input is ambiguous or the model hasn’t seen similar examples during training. <\/span>Quantifying uncertainty in language models helps us judge when it might need human-in-the-loop<\/a> oversight or further training, allowing us to handle those cases differently.<\/b> This is especially critical in high-stakes situations, like with medical or legal advice, where an overconfident wrong answer could have serious consequences.<\/span><\/p>\n\n\n\n

But this is just scratching the surface of perplexity. In this article, I want to go in depth, covering perplexity’s mathematical basis, underlying intuitions, and limitations. I’ll show you how to implement perplexity from scratch in Python, and how to add perplexity to your evaluation suite using Opik<\/a>, our open-source LLM evaluation framework<\/a>.<\/span><\/p>\n\n\n\n

Let\u2019s dive in!<\/span><\/p>\n\n\n\n

A Little Background on Perplexity<\/h2>\n\n\n\n

Perplexity was first introduced 1977 by a team of IBM researchers working on speech recognition<\/span>. The team, led by Frederick Jelinek, was looking for a metric that could measure the difficulty a statistical model experienced while making a prediction. As an interesting aside, Jelinek is the original author of the famous quote \u201cEvery time I fire a linguist, the performance of the speech recognizer goes up.\u201d<\/span><\/p>\n\n\n\n

 <\/p>\n\n\n\n

\"Table
Table from the original paper: Perplexity\u2014a measure of the difficulty of speech recognition tasks<\/a><\/figcaption><\/figure>\n\n\n\n

 <\/p>\n\n\n\n

The key insight of the initial perplexity paper is that by applying concepts from information theory to a model\u2019s internal state, we can begin to quantify more subtle qualities of a model. While the original authors were looking for a metric to approximate the \u201cdifficulty\u201d of speech recognition tasks, researchers working on NLP quickly recognized perplexity as relevant to their work as well.<\/span><\/p>\n\n\n\n

Throughout the 1980s and 1990s, perplexity emerged as the key metric for evaluating the performance of n-gram models. Perplexity was used to measure how well these models captured linguistic patterns by quantifying the average uncertainty of predictions. \u201cUncertainty\u201d  was calculated using entropy and its close mathematical relative, cross-entropy, both of which we\u2019ll explore in more detail shortly.<\/span><\/p>\n\n\n\n

Perplexity remains a primary benchmark to this day and is a popular metric for evaluating sequential neural networks (including the GPT family of models). Its many advantages, and its historical role in benchmarking, make it common even in contemporary research. At the same time, its many limitations make it insufficient as a standalone evaluation metric, especially for modern LLMs.<\/span><\/p>\n\n\n\n

In order to gain a more intuitive understanding of perplexity and its pros and cons, we need to first explore the underlying mathematics. Namely, we need to understand entropy and cross-entropy. If you already feel comfortable with these topics, feel free to skip the following section.<\/span><\/p>\n\n\n\n

Entropy, Cross-entropy and Information Theory<\/h3>\n\n\n\n

Perplexity, as an evaluation metric, has its roots in information theory and probabilistic modeling, building on Claude Shannon\u2019s work on entropy in the 1940s. Shannon used language entropy to describe the amount of information in a message, specifically when converting from a programming language to raw binary and back to a programming language:<\/span><\/p>\n\n\n\n

\n

\u201cThe entropy is a statistical parameter which measures, in a certain sense, how much information is produced on the average for each letter of a text in the language. If the language is translated into binary digits (0 or 1) in the most efficient way, the entropy H is the average of binary digits required per letter of the original language.\u201d<\/span><\/p>\nClaude Shannon in <\/span><\/i>Prediction and Entropy of Printed English<\/span><\/a>, 1951<\/span><\/cite><\/blockquote>\n\n\n\n

As described by Shannon in the 40s and 50s, language entropy quantifies the average amount of information contained in a word or sequence of words, reflecting how unpredictable the next word is based on previous context. In other words, language entropy refers to the degree of uncertainty or unpredictability in a language\u2019s word distribution. <\/span><\/p>\n\n\n\n

In the experiment below, Shannon counted how many guesses it took a human being to correctly predict each letter (including spaces) in the sentence, given only the preceding letters in the sequence.<\/span><\/p>\n\n\n\n

 <\/p>\n\n\n\n

\"Graphic
Graphic from Prediction and Entropy of Printed English<\/a> by Claude Shannon, 1950<\/figcaption><\/figure>\n\n\n\n

 <\/p>\n\n\n\n

Entropy is calculated as H(P)<\/strong> where <\/span>p<\/i>(w<\/i>1<\/i><\/strong>)<\/strong> is the probability of the <\/span>i<\/i><\/strong>th word occurring in a given context, and the summation runs over all possible words in the vocabulary. The negative sign ensures that entropy is a non-negative value, as logp<\/em>(w)<\/strong> is negative.<\/span><\/p>\n\n\n\n

 <\/p>\n\n\n\n

\"Mathematical
Mathematical equation for entropy<\/figcaption><\/figure>\n\n\n\n

 <\/p>\n\n\n\n

Higher entropy values indicate lower predictability and greater diversity in word choice, while lower values suggest more predictable language patterns, reflecting the underlying complexity of the language being modeled.<\/span><\/p>\n\n\n\n

Because the output of a large language model is typically a probability distribution calculated across all possible output tokens, entropy is very straightforward to calculate. A natural next question is how we might use entropy to train a language model, and that is where entropy\u2019s close relative <\/span>cross-entropy<\/b> comes in.<\/span><\/p>\n\n\n\n

While entropy measures the average uncertainty in a <\/span>single<\/span><\/i> probability distribution, cross-entropy quantifies the difference between <\/span>two<\/span><\/i> probability distributions. In the case of language modeling, these would be the true distribution, <\/span>P,<\/span><\/i> and the model\u2019s predicted distribution, <\/span>Q. <\/span><\/i>In this way cross-entropy provides a way to assess how well the model\u2019s predictions approximate the actual distribution of the data.<\/span><\/p>\n\n\n\n

 <\/p>\n\n\n\n

\"Mathematical
Mathematical equation for cross-entropy<\/figcaption><\/figure>\n\n\n\n

Here, <\/span>p<\/i>(x<\/i>i<\/i><\/strong>)<\/strong> represents the true probability of outcome <\/span>x<\/i>i<\/i><\/strong> and <\/span>q<\/i>(x<\/i>i<\/i><\/strong>)<\/strong> denotes the predicted probability. Lower cross-entropy values indicate higher certainty, as they imply that the model’s probabilities are more closely aligned with the actual distribution.<\/span><\/p>\n\n\n\n

In the context of model training, cross-entropy is used as the backbone of many loss functions. It measures the difference between how likely the model is to select a given token, versus the true likelihood that a given token is correct. <\/span><\/p>\n\n\n\n

Once you have a grasp of entropy and cross-entropy, perplexity follows intuitively.<\/span><\/p>\n\n\n\n

So, what is perplexity, really?<\/h2>\n\n\n\n

Like entropy and cross-entropy, perplexity also quantifies a model\u2019s uncertainty in predicting the next token in a sequence. So, why not just use entropy or cross-entropy? It turns out perplexity is far more intuitive in explaining model behavior.<\/span><\/p>\n\n\n\n

 <\/p>\n\n\n\n

\"Mathematical
Mathematical equation for perplexity<\/figcaption><\/figure>\n\n\n\n

 <\/p>\n\n\n\n

Mathematically speaking, perplexity is defined as the exponentiated average log-likelihood of the predicted words in a sequence. Or, less verbosely, perplexity is cross-entropy with the exponential function applied. This transformation might seem somewhat arbitrary at first, but it actually makes a big difference, especially in terms of interpretability.  <\/span><\/p>\n\n\n\n

Because cross-entropy is a negative log measure, when we take its exponential, we “undo” the log, converting this measure back into a regular probability space. This value represents the tangible count of likely choices the model considers at each step, or the \u201ceffective <\/span>branching factor<\/span><\/a>.\u201d <\/span><\/p>\n\n\n\n

Perplexity, then, is essentially a measure of how many options the model finds plausible on average, with lower values indicating fewer options (more confident predictions) and higher values indicating more options (greater uncertainty).<\/b><\/p>\n\n\n\n

 <\/p>\n\n\n\n

Entropy<\/strong><\/td>Measures the average uncertainty in a single probability distribution P<\/em><\/span><\/td><\/tr>
Cross-entropy<\/strong><\/td>Measures how well a predicted distribution Q<\/em> approximates the true distribution P<\/em><\/span><\/td><\/tr>
Perplexity<\/strong><\/td>Exponentiation of cross-entropy; Measures how many likely candidate tokens the model is choosing between<\/span><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

 <\/p>\n\n\n\n

In summary, entropy<\/i> measures the inherent uncertainty in a true probability distribution, reflecting the average unpredictability of outcomes, such as words in a language. Cross-entropy<\/i> extends this concept by measuring the difference between the true distribution of the data and the predicted distribution from a model, penalizing inaccurate predictions. Perplexity<\/i> builds on cross-entropy by transforming it into a more interpretable form, using the exponential function to express how many equally likely word choices the model is effectively considering.<\/p>\n\n\n\n

Note that the perplexity score of a language model on a sequence of tokens is the average of the perplexity scores for each predicted token. This means that if a language model has a perplexity of 10, on average, the model is selecting between 10 equally likely options for the next word.<\/span><\/p>\n\n\n\n

\"An
The perplexity score of a language model on a sequence of tokens is the average of the perplexity scores for each predicted token.<\/figcaption><\/figure>\n\n\n\n

Using this intuition, a lower perplexity score is better because it indicates that a model is effectively \u201cchoosing\u201d between fewer viable options for the next word and is \u201cless surprised.\u201d A higher perplexity score, on the other hand, indicates more \u201cuncertainty.\u201d <\/span><\/p>\n\n\n\n

Of course, it is entirely possible for a language model to be \u201cconfident\u201d <\/span>and<\/span><\/i> \u201cincorrect,\u201d so perplexity should not be confused with an accuracy metric. But we\u2019ll dive into more of perplexity\u2019s limitations later on. First, let\u2019s explore some of its advantages.<\/span><\/p>\n\n\n\n

Advantages of Perplexity<\/h2>\n\n\n\n

As mentioned, one of the biggest advantages of perplexity is that it is highly intuitive and explainable in a field that is notoriously opaque. This is a notable advantage over learned metrics like BERTScore<\/a> and LLM-as-a-Judge<\/a> metrics like G-Eval<\/a>.<\/span><\/p>\n\n\n\n

Having an estimate of a model\u2019s certainty is also especially useful when using an LLM to plan or guide actions. While high certainty suggests the model has strong backing for a given prediction, low certainty can prompt further human oversight or additional checks before execution. <\/span><\/p>\n\n\n\n

Perplexity is also computationally straightforward to calculate, making it fast and efficient. This also allows practitioners to evaluate model performance in real-time during training, helping to identify issues and improvements promptly and leading to faster development cycles.<\/span><\/p>\n\n\n\n

As we\u2019ll see in the next section on perplexity\u2019s limitations, it is not an end-all evaluation metric for LLMs. However, given its explainability and low-overhead, perplexity is a quick and useful first-pass metric that works well when used in conjunction with other LLM evaluation metrics<\/a>.<\/span><\/p>\n\n\n\n

Limitations of Perplexity<\/h2>\n\n\n\n

The most important limitation of perplexity is that it does not convey a model\u2019s \u201cunderstanding.\u201d Perplexity is strictly a measure of uncertainty, and a model being uncertain doesn\u2019t mean it is right or wrong. A model may be correct but unconfident or wrong but confident. So, a perplexity score isn\u2019t a measure of accuracy, just of confidence.<\/span><\/p>\n\n\n\n

It is also difficult to use perplexity as a benchmark between models. Perplexity scores are influenced by various model-specific factors, such as tokenization method, dataset, pre-processing steps, vocabulary size, and context length. For example, a character-level model may have a lower perplexity than a word-level model, but that doesn\u2019t necessarily mean the character-level model is better. <\/span><\/p>\n\n\n\n

A model can also achieve a low perplexity score by assigning high probabilities to common words, like articles and conjunctions, leading to a misleadingly low score. Overfit models can show low perplexity but lack true understanding. <\/span>Research also suggests that perplexity doesn’t correlate well with an LLM’s long-term understanding<\/span><\/a>, likely because it fails to capture long-term dependencies. Additionally, <\/span>perplexity can be skewed by punctuation and repeated text spans<\/span><\/a>, which lower scores but don\u2019t necessarily improve text quality.<\/span><\/p>\n\n\n\n

While perplexity has limitations, it remains a valuable first-pass metric when combined with other task-specific LLM evaluation<\/a> metrics, offering both interpretability and efficiency.<\/span><\/p>\n\n\n\n

Implementing Perplexity From Scratch in Python<\/h2>\n\n\n\n

Using what we\u2019ve learned so far about perplexity, let\u2019s implement it from scratch in Python so we can apply it directly to our LLM outputs. Note that because perplexity is such a common evaluation metric, there are several pre-built modules to implement it in Python, including <\/span>Hugging Face\u2019s evaluate.metrics.perplexity<\/span><\/a> and <\/span>perplexed from Stability AI<\/span><\/a>. But coding the metric from scratch will help build intuition for what perplexity is actually doing under the hood. Later, we\u2019ll test our function out on GPT-2 and learn how to automatically track the perplexity scores of our LLM using a custom metric in Opik.<\/span><\/p>\n\n\n\n

Throughout our implementation, we\u2019ll be using PyTorch and HuggingFace\u2019s Transformers library.<\/span><\/p>\n\n\n\n

Our basic perplexity function will take <\/span>logits<\/b> and <\/span>target<\/b> labels as inputs. <\/span><\/p>\n\n\n\n

Logits<\/b> are the raw scores output by the model for each token in the <\/span>vocabulary<\/b> for a given position in the <\/span>input sequence<\/b>. For each position in the sequence, the model outputs a vector of logits, where each entry in that vector corresponds to a token in the vocabulary.<\/span><\/p>\n\n\n\n

\"Shape
Shape of logits vector in LLM output<\/figcaption><\/figure>\n\n\n\n

The <\/span>targets<\/b> will be the ground truth label tensors.<\/span><\/p>\n\n\n\n

To calculate perplexity, we\u2019ll need to:<\/span><\/p>\n\n\n\n

    \n
  1. Convert the logits to probabilities using the log_softmax function, which normalizes the scores.<\/span><\/li>\n\n\n\n
  2. Gather the log probabilities of the correct target tokens.<\/span><\/li>\n\n\n\n
  3. As these results will be negatives, we\u2019ll multiply them by -1 to convert the probabilities into the range of 0-1. This value represents the <\/span>entropy<\/b>.<\/span><\/li>\n\n\n\n
  4. Take the mean entropy of all tokens in the sequence. This value represents the <\/span>cross-entropy<\/b>.<\/span><\/li>\n\n\n\n
  5. Take the exponential of the <\/span>cross-entropy<\/b>. This value represents the <\/span>perplexity<\/b>, or effective branching factor of each token in the sequence.<\/span><\/li>\n<\/ol>\n\n\n\n
    \nimport torch\n\ndef calculate_perplexity(logits, target):\n    \"\"\"\n    Calculate perplexity from logits and target labels.\n\n    Args:\n    - logits (torch.Tensor): Logits output from the model (batch_size, seq_length, vocab_size).\n    - target (torch.Tensor): Ground truth labels (batch_size, seq_length).\n\n    Returns:\n    - perplexity (float): The perplexity score.\n    \"\"\"\n\n    # Convert logits to log probabilities\n    log_probs = torch.nn.functional.log_softmax(logits, dim=-1)\n\n    # Gather the log probabilities for the correct target tokens\n    # log_probs has shape (batch_size, seq_length, vocab_size)\n    # target has shape (batch_size, seq_length)\n    # The gather method will pick the log probabilities of the true target tokens\n    target_log_probs = log_probs.gather(dim=-1, index=target.unsqueeze(-1)).squeeze(-1)\n\n    # Calculate the negative log likelihood\n    negative_log_likelihood = -target_log_probs\n\n    # Calculate the mean negative log likelihood over all tokens\n    mean_nll = negative_log_likelihood.mean()\n\n    # Calculate perplexity as exp(mean negative log likelihood)\n    perplexity = torch.exp(mean_nll)\n\n    return perplexity.item()\n\n# Example usage\n# Simulate a batch of logits (batch_size=2, seq_length=4, vocab_size=10)\nlogits = torch.randn(2, 4, 10)\n# Simulate ground truth target tokens\ntarget = torch.tensor([[1, 2, 3, 4], [4, 3, 2, 1]])\n\n# Calculate perplexity\nperplexity = calculate_perplexity(logits, target)\nprint(f'Perplexity: {perplexity}')\n<\/code><\/pre>\n\n\n\n
    The function above calculates perplexity from a mathematical perspective, but it requires some adjustments to handle raw text, as you would encounter in real-world scenarios. <\/span><\/p>\n\n\n\n
    Now that we’ve covered the math behind perplexity, let’s modify the function to work with the inputs and outputs of a large language model.For this version of our function we\u2019ll want to:<\/span><\/p>\n\n\n\n
      \n
    1. Shift the logits and target tensors<\/b> so that each model prediction (logit) matches the actual token in the sequence (target\/input_ids). Since each token is predicted based on the previous tokens, the prediction for token \ud835\udc61 should be compared to the actual token at \ud835\udc61 + 1.<\/span><\/li>\n\n\n\n
    2. Add batching<\/b> to handle texts longer than the model\u2019s context length by splitting them into smaller chunks for parallel processing.<\/span><\/li>\n\n\n\n
    3. Use padding tokens<\/b> to standardize input lengths across sentences of varying lengths.<\/span><\/li>\n\n\n\n
    4. Apply an attention mask<\/b> to exclude padding tokens from perplexity calculations.<\/span><\/li>\n\n\n\n
    5. Average the perplexity scores across tokens for a <\/span>sequence-level score<\/b>.<\/span><\/li>\n\n\n\n
    6. Average the sequence-level scores for an overall batch-level perplexity score<\/b>.<\/span><\/li>\n<\/ol>\n\n\n\n
      \"An
      Since each token is predicted based on the previous tokens, the prediction for token \ud835\udc61 should be compared to the actual token at \ud835\udc61 + 1.<\/figcaption><\/figure>\n\n\n\n
      \nimport torch\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\n# Load the model and tokenizer (e.g., GPT-2)\nmodel_name = \"gpt2\"\ntokenizer = AutoTokenizer.from_pretrained(model_name)\nmodel = AutoModelForCausalLM.from_pretrained(model_name)\n\n# Assign the EOS token as the padding token\ntokenizer.pad_token = tokenizer.eos_token\n\ndef calculate_batch_perplexity(input_texts):\n    \"\"\"\n    Calculate perplexity for a batch of input texts using a pretrained language model.\n\n    Args:\n    - input_texts (List[str]): A list of input texts to evaluate.\n\n    Returns:\n    - List[float]: A list of perplexity scores, one for each input text.\n    \"\"\"\n    # Tokenize the batch of texts with padding for uniform length\n    inputs = tokenizer(\n        input_texts, return_tensors=\"pt\", padding=True, truncation=True\n    )\n\n    input_ids = inputs[\"input_ids\"]\n    attention_mask = inputs[\"attention_mask\"]\n\n    # Pass the input batch through the model to get logits\n    with torch.no_grad():\n        outputs = model(input_ids, attention_mask=attention_mask)\n        logits = outputs.logits\n\n    # Shift the logits and input_ids to align targets correctly\n    # Logits dimensions are: (batch_size, seq_length, vocab_size)\n    shift_logits = logits[:, :-1, :]  # Ignore the last token's logits\n    shift_labels = input_ids[:, 1:]   # Skip the first token in the labels\n\n    # Compute log probabilities\n    log_probs = torch.nn.functional.log_softmax(shift_logits, dim=-1)\n\n    # Gather the log probabilities for the correct tokens\n    target_log_probs = log_probs.gather(dim=-1, index=shift_labels.unsqueeze(-1)).squeeze(-1)\n\n    # Mask out positions corresponding to padding tokens\n    target_log_probs = target_log_probs * attention_mask[:, 1:].to(log_probs.dtype)\n\n    # Compute the mean negative log-likelihood for each sequence\n    negative_log_likelihood = -target_log_probs.sum(dim=-1) \/ attention_mask[:, 1:].sum(dim=-1)\n\n    # Compute perplexity for each sequence\n    perplexities = torch.exp(negative_log_likelihood)\n    perplexities = perplexities.tolist()\n\n# Take mean of perplexities of each batch\n    mean_perplexity_score = torch.mean(perplexities)\n\n    return {\"perplexities\": perplexities, \"mean_perplexity\": mean_perplexity_score}\n\n# Example usage\ntexts = [\n    \"The quick brown fox jumps over the lazy dog.\",\n    \"A journey of a thousand miles begins with a single step.\"\n]\nprint(f\"Perplexity scores: {calculate_batch_perplexity(texts)}\")\n<\/code><\/pre>\n\n\n\n
      This function takes as input a list of texts, and outputs a dictionary containing a list of perplexity scores for each text in the input list, as well as the average perplexity score of text sequences in the input list.  <\/span><\/p>\n\n\n\n
      Note that taking the average of perplexity scores across texts of different lengths can lead to a skewed overall perplexity score for a couple of reasons. <\/span><\/p>\n\n\n\n
      First, perplexity scores tend to be more stable for longer sequences, while shorter sequences may have higher variance, leading to outliers. Second, taking a simple arithmetic mean across scores for texts of varying lengths can give disproportionate weight to tokens in shorter sequences. Nevertheless, using the arithmetic mean is currently the most common approach to calculating overall perplexity, so we use it here for the sake of consistency. <\/span><\/p>\n\n\n\n
      Implementing Perplexity in Opik<\/h2>\n\n\n\n
      In the real world, you\u2019ll likely want to use an evaluation framework to implement LLM metrics. In this section, we\u2019ll implement perplexity in <\/span>Opik<\/span><\/a>, Comet\u2019s open source LLM evaluation framework.<\/span><\/p>\n\n\n\n
      Here, we use our original perplexity function and modify it slightly to implement it as a custom Opik metric with a `score` method that returns a `ScoreResult` object:<\/span><\/p>\n\n\n\n
      \nfrom opik.evaluation.metrics import base_metric, score_result\n\nclass Perplexity(base_metric.BaseMetric):\n    \"\"\"\n    Perplexity (PPL) is a common LLM evaluation metric defined as the exponentiated average\n    negative log-likelihood of a sequence.\n\n    For more information on perplexity, see:\n    https://en.wikipedia.org\/wiki\/Perplexity\n\n    Args:\n        name: The name of the metric, perplexity.\n    \"\"\"\n\n    def __init__(\n        self,\n        name: str = \"Perplexity\",\n    ):\n        super().__init__(name=name)\n\n    def score(\n        self, input_ids: torch.Tensor, logits: torch.Tensor, attention_mask: torch.Tensor\n    ) -> score_result.ScoreResult:\n        \"\"\"\n        Calculate the perplexity score of each token give the previous tokens in the sequence.\n\n        Args:\n            input_ids: input ids of the text sequence input to the model (torch.Tensor)\n            logits: output logits of the model (torch.Tensor)\n            attention_mask: attention mask\n\n        Returns:\n            score_result.ScoreResult: A ScoreResult object\n        \"\"\"\n\n        # Shift the logits and input_ids to align targets correctly\n        shift_logits = logits[:, :-1, :]  # Ignore the last token's logits\n        shift_labels = input_ids[:, 1:]   # Skip the first token in the labels\n\n        # Compute log probabilities\n        log_probs = torch.nn.functional.log_softmax(shift_logits, dim=-1)\n\n        # Gather the log probabilities for the correct tokens\n        target_log_probs = log_probs.gather(dim=-1, index=shift_labels.unsqueeze(-1)).squeeze(-1)\n\n        # Mask out positions corresponding to padding tokens\n        target_log_probs = target_log_probs * attention_mask[:, 1:].to(log_probs.dtype)\n\n        # Compute the mean negative log-likelihood for each sequence\n        negative_log_likelihood = -target_log_probs.sum(dim=-1) \/ attention_mask[:, 1:].sum(dim=-1)\n\n        # Take the exp(negative_log_likelihood)\n        perplexities = torch.exp(negative_log_likelihood)\n\n        # Take the mean of perplexity scores\n        mean_perplexity_score = torch.mean(perplexities)\n\n        return score_result.ScoreResult(value=mean_perplexity_score, name=self.name)\n\nperplexity = Perplexity()\n<\/code><\/pre>\n\n\n\n
      After defining perplexity as a custom metric, we can use it by:<\/span><\/p>\n\n\n\n