{"id":12303,"date":"2024-12-19T09:10:38","date_gmt":"2024-12-19T17:10:38","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=12303"},"modified":"2025-11-13T19:58:08","modified_gmt":"2025-11-13T19:58:08","slug":"bertscore-for-llm-evaluation","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/bertscore-for-llm-evaluation\/","title":{"rendered":"BERTScore For LLM Evaluation"},"content":{"rendered":"\n
\n
Follow along with the Colab!<\/a><\/div>\n<\/div>\n\n\n\n

Introduction<\/h2>\n\n\n\n

BERTScore represents a pivotal shift in LLM evaluation<\/a>, moving beyond traditional heuristic-based metrics like BLEU and ROUGE to a learned approach that captures complex linguistic nuances. Unlike older n-gram-based methods, BERTScore excels at evaluating paraphrasing, coherence, relevance, and polysemy\u2014essential features for modern AI applications.<\/span><\/p>\n\n\n\n

\"BERTScore<\/figure>\n\n\n\n

BERTScore leverages transformer-based contextual embeddings and compares them using cosine similarity to assess the quality of model outputs. Its popularity endures due to its relatively low computational cost and greater interpretability compared to black-box methods like LLM-as-a-judge<\/a> metrics.<\/span><\/p>\n\n\n\n

In this article, I\u2019ll explore how BERTScore improves upon traditional evaluation methods, explain its key components, and discuss its role in the broader hierarchy of language model evaluation metrics. Finally, I\u2019ll guide you through implementing BERTScore in Python and show how to integrate it into your evaluation suite using Opik, our open-source LLM evaluation framework<\/a>.<\/span><\/p>\n\n\n\n

The Basics of BERTScore<\/h2>\n\n\n\n

On the surface, BERTScore is, pretty easy to explain: it measures the similarity between tokens in two text sequences by representing them as BERT embeddings and calculating their cosine similarity.  For example, given the target sentence, \u201cThe red shoes cost $20.00,\u201d BERTScore would rate the candidate sentence “The rouge slippers cost $20” as more similar than “The blue socks cost $20,” even though they have roughly the same number of incorrect tokens.<\/span><\/p>\n\n\n\n

 <\/p>\n\n\n\n

\"Illustration
Source: BERTScore: Evaluating Text Generation with BERT<\/a><\/figcaption><\/figure>\n\n\n\n

 <\/p>\n\n\n\n

What makes BERTScore particularly compelling is how it combines different approaches to evaluation. Broadly, LLM evaluation metrics<\/a> can generally be broken down into three hierarchical categories: heuristic metrics, learned metrics, and LLM-as-a-judge metrics, with BERTScore occupying a unique position within this framework.<\/span><\/p>\n\n\n\n

\"Diagram
Broadly speaking, LLM evaluation metrics can generally be broken down into three hierarchical categories: heuristic metrics, learned metrics, and LLM-as-a-judge metrics, with BERTScore occupying a unique position within this framework.<\/figcaption><\/figure>\n\n\n\n

<\/h3>\n\n\n\n

Heuristic Metrics<\/h3>\n\n\n\n

Heuristic metrics are evaluation <\/span>measures<\/span><\/a> that are based on <\/span>predefined<\/b>, <\/span>rules-based formulas<\/b> that <\/span>quantify<\/b> specific aspects of model outputs. They are deterministic, interpretable, and computationally efficient. But because they rely on measurable surface-level features like token overlap or exact matches, they often fail to account for the more complex aspects of language, like context, complex semantics, or creativity.<\/span><\/p>\n\n\n\n

Heuristic metrics include distance metrics, statistical metrics, and overlap or n-gram-based metrics. Popular examples include accuracy, <\/span>perplexity<\/span><\/a>, <\/span>BLEU<\/span><\/a>, <\/span>ROUGE<\/span><\/a>, <\/span>Levenshtein distance<\/span><\/a>, and <\/span>cosine similarity<\/b>.<\/span><\/p>\n\n\n\n

Learned Metrics<\/h3>\n\n\n\n

While heuristic metrics rely on fixed, rules-based formulas, <\/span>learned metrics use machine learning models to score text quality<\/b>. Typically, these models will represent the evaluated text as some kind of learned embedding. <\/span><\/p>\n\n\n\n

Because embeddings capture semantic and contextual information, learned metrics provide more depth and nuance than heuristic metrics alone and are effectively able to capture aspects like paraphrasing, coherence, and relevance. <\/span><\/p>\n\n\n\n

Learned metrics tend to be more aligned with human judgment, but are also more computationally expensive and less interpretable. Examples of learned metrics include BERTScore, <\/span>BLEURT<\/span><\/a>, <\/span>COMET<\/span><\/a>, and <\/span>UniEval<\/span><\/a>.<\/span><\/p>\n\n\n\n

LLM-as-a-judge Metrics<\/h3>\n\n\n\n

LLM-as-a-judge metrics are probably the most popular evaluation metrics for evaluating generative language models, and are able to capture the deepest levels of nuance in language. However, they are also the most computationally expensive and present unique interpretability challenges.<\/span><\/p>\n\n\n\n

LLM-as-a-judge metrics use large language models themselves to act as a “judge” and provide feedback or a quality score based on an evaluation criteria. They are especially useful for open-ended and complex tasks, such as creative writing or reasoning, where predefined metrics may fall short.<\/span><\/p>\n\n\n\n

BERTScore has the robustness of a learned metric, as it uses BERT\u2019s learned embeddings, but because it is \u201conly\u201d measuring the cosine similarity of token embeddings, it also benefits from the computational efficiency and repeatability of heuristic metrics. If you\u2019re not sure what any of this means, don\u2019t worry, we\u2019ll cover it in the next section! <\/span><\/p>\n\n\n\n

Theory Behind BERTScore<\/h2>\n\n\n\n

As we established earlier, BERTScore evaluates the similarity between a reference (ground truth) sentence and a candidate (prediction) sentence by representing their tokens with contextual embeddings and comparing them using cosine similarity. Let\u2019s break that down, starting with a little background.<\/span><\/p>\n\n\n\n

Prior to BERTScore, the most popular evaluation metrics for text generation were heuristic metrics like n-gram or overlap-based metrics. <\/span><\/p>\n\n\n\n

The Problem With N-grams<\/h3>\n\n\n\n

N-grams count the number of continuous sequences of n<\/em> tokens that occur in both the reference and candidate sentences. It\u2019s highly intuitive, but poses some major challenges. Smaller n values often fail to capture context, such as word order, while larger n values quickly become overly restrictive. <\/span><\/p>\n\n\n\n

More critically, n-grams cannot account for linguistic nuances like paraphrasing, dependencies, and polysemy. This means they score words with multiple meanings identically and fail to recognize synonyms or paraphrases with similar meaning. These limitations make n-grams inadequate for evaluating the depth and complexity of modern language models.<\/span><\/p>\n\n\n\n

 <\/p>\n\n\n\n

\"n-gram
N-grams cannot account for linguistic nuances like paraphrasing, dependencies, and polysemy<\/a>.<\/figcaption><\/figure>\n\n\n\n

[br][br]<\/p>\n\n\n\n

Contextual Embeddings<\/h3>\n\n\n\n

To address these issues, BERTScore leverages contextual embeddings. Unlike static embeddings, such as those from Word2Vec or GloVe, contextual embeddings are generated by transformer models, which use attention mechanisms to capture the relationships between all words in a sentence. This approach provides greater flexibility and nuance, making it more suitable for complex tasks like language understanding.<\/span><\/p>\n\n\n\n

While a deeper exploration of embeddings is beyond the scope of this article, in simple terms, embeddings are vectors of floating-point numbers that capture the semantic context of individual tokens.<\/span><\/p>\n\n\n\n

Cosine Similarity<\/h3>\n\n\n\n

After BERTScore has used a transformer-based model (originally a BERT model) to generate contextual embeddings, how does it use them to quantify sentence similarity?<\/span><\/p>\n\n\n\n

As mentioned, contextual embeddings are high-dimensional vectors of floating-point numbers. To measure their similarity, we apply basic linear algebra by calculating the cosine of the angle between two vectors. Vectors that are closer in the embedding space have a higher semantic similarity. This measure is the cosine similarity<\/a>.<\/span><\/p>\n\n\n\n

\"A
The cosine of any two normalized vectors is equal to their dot product<\/a>.<\/figcaption><\/figure>\n\n\n\n

Once the cosine similarity has been calculated between each token in the candidate sentence and each token in the reference sentence, greedy matching is used to select the highest cosine similarity score for each token.<\/span><\/p>\n\n\n\n

The core benefit of BERTScore is that it gives you the richness of a learned metric via contextual embeddings, with the computational efficiency of a heuristic metric like cosine similarity.<\/span><\/p>\n\n\n\n

The final steps include using the maximum similarity scores of each token to compute BERTRecall, BERTPrecision, and BERTF1, and optional importance weighting and baseline rescaling, which we\u2019ll cover in the next section.<\/span><\/p>\n\n\n\n

Implement BERTScore From Scratch in Python<\/h2>\n\n\n\n

Using what we\u2019ve learned so far about BERTScore, let\u2019s implement it from scratch in Python to help build intuition for what it\u2019s actually doing under the hood. Later, we\u2019ll implement BERTScore as a custom metric in Opik, and test it out on an image-captioning dataset. <\/span><\/p>\n\n\n\n

First, we\u2019ll need to choose our BERT-based model. Here we choose a medium-sized BERT model for English texts and load its accompanying tokenizer:<\/span><\/p>\n\n\n\n

import torch\nfrom transformers import BertTokenizer, BertModel\n\n# Load BERT model and tokenizer\nMODEL_NAME = \"bert-base-uncased\"\n\ntokenizer = BertTokenizer.from_pretrained(MODEL_NAME)\nmodel = BertModel.from_pretrained(MODEL_NAME, device_map=\"auto\")\n<\/code><\/pre>\n\n\n\n
You can find a full list of supported models, along with their performance scores and best representation layers here<\/a>.<\/p>\n\n\n\n
BERT Embeddings and Cosine Similarity<\/h3>\n\n\n\n
Next, to calculate BERTScore, we\u2019ll define several functions to help out with each step of the process. Leaving out the optional steps mentioned above, this includes:<\/span><\/p>\n\n\n\n
    \n
  1. Getting embeddings of each sentence <\/span><\/li>\n\n\n\n
  2. Calculating cosine similarity between embeddings<\/span><\/li>\n\n\n\n
  3. Using greedy matching to select highest score<\/span><\/li>\n\n\n\n
  4. Calculating BERTRecall, BERTPrecision, BERTF1 and return in a dictionary.<\/span><\/li>\n<\/ol>\n\n\n\n
    Let\u2019s start with a function to compute the embeddings of each reference and candidate sentence. We\u2019ll need to tokenize the text, create embeddings of the tokens, and return the first dimension of the model\u2019s output, which corresponds to the last hidden state.<\/span><\/p>\n\n\n\n
    def get_embeddings(text):\n    \"\"\"\n    Generate token embeddings for the input text using BERT.\n\n    Args:\n        text (str): Input text or batch of sentences.\n\n    Returns:\n        torch.Tensor: Token embeddings with shape (batch_size, seq_len, hidden_dim).\n    \"\"\"\n    # Tokenize input text\n    inputs = tokenizer(text, return_tensors=\"pt\", padding=True, truncation=True)\n    # Move inputs to GPU if available\n    device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n    inputs = inputs.to(device)\n\n    # Compute embeddings without gradient calculation\n    with torch.no_grad():\n        outputs = model(**inputs, output_hidden_states=True)\n    # Return last hidden states (token-level embeddings)\n    return outputs.last_hidden_state<\/code><\/pre>\n\n\n\n
    Next, we\u2019ll create a function to calculate the cosine similarity between generated embeddings. These embeddings will need to be reshaped and then normalized. Once normalized the cosine similarity between two vectors equals their dot product, so we can use basic matrix multiplication to create a matrix of cosine similarity scores.<\/span><\/p>\n\n\n\n
    def cosine_similarity(generated_embeddings, reference_embeddings):\n    \"\"\"\n    Compute cosine similarity between two sets of embeddings.\n\n    Args:\n        generated_embeddings (torch.Tensor): Embeddings of candidate tokens with shape (batch_size, seq_len, hidden_dim).\n        reference_embeddings (torch.Tensor): Embeddings of reference tokens with shape (batch_size, seq_len, hidden_dim).\n\n    Returns:\n        torch.Tensor: Cosine similarity matrix with shape (seq_len_generated, seq_len_reference).\n    \"\"\"\n    # Normalize embeddings along the hidden dimension\n    generated_embeddings = torch.nn.functional.normalize(generated_embeddings, dim=-1)\n    reference_embeddings = torch.nn.functional.normalize(reference_embeddings, dim=-1)\n\n    # Compute similarity using batched matrix multiplication\n    return torch.bmm(generated_embeddings, reference_embeddings.transpose(1, 2))\n<\/code><\/pre>\n\n\n\n
    We now have matrices containing the cosine similarity scores of each token in the candidate sentence with each token in the reference sentence. But we need a way to aggregate these values for a sentence-level representation of similarity. The original authors of the BERTScore paper proposed three measures: BERTRecall, BERTPrecision, and BERTF1 to do just this.<\/span><\/p>\n\n\n\n
    \"Similarity
    BERTScore provides a convenient function to visualize the cosine similarity matrix of any two sentences.<\/a><\/figcaption><\/figure>\n\n\n\n
    BERTPrecision, BERTRecall, BERTF1<\/h3>\n\n\n\n
    Traditionally, precision, recall, and F1 scores evaluate a classifier\u2019s ability to distinguish between positive and negative samples. While BERTScore isn\u2019t designed for classification, its authors adapted these metrics for evaluating LLM-generated text, preserving their original intent in this new context. <\/span><\/p>\n\n\n\n
    \"A
    To calculate BERTPrecision, BERTRecall, and BERTF1, we first use greedy matching to gather the maximum similarity scores for each candidate with each reference, as well as for each reference with each candidate.<\/figcaption><\/figure>\n\n\n\n
    Let\u2019s start with precision. Precision measures a model\u2019s accuracy in identifying true positives. In BERTScore, \u201ctrue positives\u201d are candidate tokens that align with reference tokens. <\/span>BERTprecision quantifies how much of the candidate\u2019s content is semantically meaningful relative to the reference<\/b>. It is calculated as the average of the maximum cosine similarities between each candidate token’s embedding and the embeddings of all reference tokens. A high BERTPrecision indicates that the candidate is concise and relevant.<\/span><\/p>\n\n\n\n
    \"Mathematical
    Mathematical equation for BERTPrecision<\/figcaption><\/figure>\n\n\n\n
    def get_precision(similarity_matrix):\n    \"\"\"\n    Calculate BERT precision as the mean of the maximum similarity scores from the candidate to the reference.\n\n    Args:\n        similarity_matrix (torch.Tensor): Cosine similarity matrix.\n\n    Returns:\n        torch.Tensor: Precision score.\n    \"\"\"\n    return similarity_matrix.max(dim=2)[0].mean()\n<\/code><\/pre>\n\n\n\n
    Next, let\u2019s define BERTRecall. Recall measures the proportion of actual positive instances that a model identifies correctly. For BERTScore, <\/span>BERTRecall reflects how much of the reference\u2019s meaning is captured by the candidate<\/b>. It is calculated as the average of the maximum cosine similarity scores between each reference token\u2019s embedding and the embeddings of all candidate tokens. \u200b\u200bA high BERTRecall suggests that the candidate does not miss key information present in the reference.<\/span><\/p>\n\n\n\n
    \"Mathematical
    Mathematical formula for BERTRecall<\/figcaption><\/figure>\n\n\n\n
    def get_recall(similarity_matrix):\n    \"\"\"\n    Calculate BERT recall as the mean of the maximum similarity scores from the reference to the candidate.\n\n    Args:\n        similarity_matrix (torch.Tensor): Cosine similarity matrix.\n\n    Returns:\n        torch.Tensor: Recall score.\n    \"\"\"\n    return similarity_matrix.max(dim=1)[0].mean()\n<\/code><\/pre>\n\n\n\n
    The BERTF1 score is the harmonic mean of BERTPrecision and BERTRecall, balancing these two metrics when there is a trade-off. It provides a single summary value of overall semantic alignment between the candidate and the reference.<\/span><\/p>\n\n\n\n
    \"Mathematical
    Mathematical formula for BERTF1<\/figcaption><\/figure>\n\n\n\n
    def get_f1_score(precision, recall):\n    \"\"\"\n    Compute the F1 score given precision and recall.\n\n    Args:\n        precision (torch.Tensor): Precision score.\n        recall (torch.Tensor): Recall score.\n\n    Returns:\n        torch.Tensor: F1 score.\n    \"\"\"\n    return 2 * (precision * recall) \/ (precision + recall)\n<\/code><\/pre>\n\n\n\n
    Finally, BERTScore outputs BERTPrecision, BERTRecall, and BERTF1 as a dictionary, which we\u2019ll cover in the next coding section. <\/span><\/p>\n\n\n\n
    Importance Weighting and Baseline Rescaling with BERTScore<\/h3>\n\n\n\n
    Additionally, BERTScore includes some optional processes including importance weighting with IDF and baseline rescaling. <\/span><\/p>\n\n\n\n
    Since rare words are often more indicative of sentence meaning than common words or stop words<\/a>, BERTScore allows for frequency penalization using Inverse Document Frequency (IDF) of the test corpus (body of reference sentences). <\/span><\/p>\n\n\n\n
    IDF is based on the principle that words appearing in many documents are less informative than words that appear in fewer documents. It\u2019s calculated by taking the logarithm of the total number of documents, N<\/code>, divided by the number of documents containing a given term, <\/span>t<\/span><\/i><\/code>.<\/span><\/p>\n\n\n\n
    \"Mathematical
    Mathematical formula for Inverse Document Frequency (IDF)<\/figcaption><\/figure>\n\n\n\n
    Additionally, to normalize the scores to a range of -1 to 1 and make them more readable, the original BERTScore paper suggests rescaling BERTScore with respect to its empirical lower bound <\/span>b<\/span><\/i> as a baseline. This calculation does not affect score ranking however, and is solely meant to improve readability. A full list of baseline scores for BERT models in 12 languages can be found in <\/span>this directory<\/span><\/a>.<\/span><\/p>\n\n\n\n
    \"Mathematical
    Mathematical formula for BERTScore baseline rescoring<\/figcaption><\/figure>\n\n\n\n
    \"A
    After baseline rescaling, the cosine similarity scores range from -1 to 1.<\/figcaption><\/figure>\n\n\n\n
    Both of these processes are set to <\/span>False<\/b> by default in Hugging Face\u2019s implementation of BERTScore, so we won\u2019t include them when we code BERTScore from scratch. Each can be set to <\/span>True<\/b> with the <\/span>idf<\/b> and <\/span>rescale_with_baseline<\/b> parameters of <\/span>evaluate.bertscore<\/b>, respectively.<\/span><\/p>\n\n\n\n
    Putting It All Together<\/h3>\n\n\n\n
    Now that we have all of our helper functions, let\u2019s put them together to create our BERTScore function. In this function we will:<\/span><\/p>\n\n\n\n
      \n
    1. Generate embeddings of the candidate and reference tokens using the model we instantiated above.<\/span><\/li>\n\n\n\n
    2. Calculate the cosine similarity matrix (or, after normalization, the dot product) between each candidate embedding and each reference embedding.<\/span><\/li>\n\n\n\n
    3. Using greedy matching, calculate the precision, recall, and f1 scores of each sentence<\/span><\/li>\n\n\n\n
    4. Return the dictionary of values.<\/span><\/li>\n<\/ol>\n\n\n\n
      def bert_score(candidate, reference):\n    \"\"\"\n    Compute BERTScore (Precision, Recall, F1) between a candidate and a reference sentence.\n\n    Args:\n        candidate (str): Candidate sentence.\n        reference (str): Reference sentence.\n\n    Returns:\n        dict: Dictionary containing precision, recall, and F1 scores.\n    \"\"\"\n    # Get token embeddings for candidate and reference\n    candidate_embeddings = get_embeddings(candidate)\n    reference_embeddings = get_embeddings(reference)\n\n    # Compute cosine similarity matrix\n    similarity_matrix = cosine_similarity(candidate_embeddings, reference_embeddings)\n\n    # Calculate precision, recall, and F1 scores\n    precision = get_precision(similarity_matrix)\n    recall = get_recall(similarity_matrix)\n    f1_score = get_f1_score(precision, recall)\n\n    # Return scores as a dictionary\n    return {\n        \"precision\": precision.item(),\n        \"recall\": recall.item(),\n        \"f1_score\": f1_score.item(),\n    }\n\n# Example usage\nif __name__ == \"__main__\":\n    candidate_sentence = \"The cat sat on the mat.\"\n    reference_sentence = \"A cat rested on a mat.\"\n\n    scores = bert_score(candidate_sentence, reference_sentence)\n    print(\"BERTScore:\", scores)\n<\/code><\/pre>\n\n\n\n
      Feel free to test this function out for yourself! Note that the intention of this exercise is to help build intuition around what BERTScore does under the hood, and it is significantly simplified from the HuggingFace.evaluate version, which incorporates IDF, baseline rescaling, batching, padding, attention mask shifting, and more.<\/span><\/p>\n\n\n\n
      For these reasons, we will be using the Hugging Face implementation of BERTScore to build a custom metric in Opik below. <\/span><\/p>\n\n\n\n
      Implement BERTScore in Opik<\/h2>\n\n\n\n
      Now let’s try a real-life end-to-end example. If you aren’t already, you can follow along with the Colab here<\/a>.<\/p>\n\n\n\n
      In this section, we\u2019ll use BLIP, an image-captioning model, along with a small subset of the <\/span>Conceptual Captions<\/span><\/a> dataset from Google Research, which pairs images with captions sourced from the internet. Notably, image captioning and machine translation were the original use cases proposed by BERTScore\u2019s authors.<\/span><\/p>\n\n\n\n
      To do this, we\u2019ll implement BERTScore in Opik, Comet\u2019s open-source LLM evaluation framework<\/a>. We\u2019ll leverage Hugging Face\u2019s evaluate implementation of BERTScore, modifying it slightly to create a custom Opik metric with a score method that returns a ScoreResult object:<\/span><\/p>\n\n\n\n
      from evaluate import load\n\nbertscore = load(\"bertscore\")\n\nfrom opik.evaluation.metrics import base_metric, score_result\nfrom typing import List, Union\n\nclass BERTScore(base_metric.BaseMetric):\n    \"\"\"\n    BERTScore is a semantic similarity evaluation metric for text generation tasks.\n    It measures the similarity between predicted (candidate) and reference texts\n    by comparing their contextual embeddings using a pre-trained language model.\n\n    This implementation leverages the Hugging Face Evaluate library for computing BERTScore.\n\n    For more details:\n    - Original BERTScore paper: https:\/\/arxiv.org\/abs\/1904.09675\n    - Hugging Face implementation: https:\/\/github.com\/huggingface\/evaluate\/blob\/main\/metrics\/bertscore\/README.md\n\n    Args:\n        name (str): The name of the metric, defaults to \"BERTScore\".\n        language (str): The language of the model, defaults to \"en\" (English).\n    \"\"\"\n\n    def __init__(\n        self,\n        name: str = \"BERTScore\",\n        language: str = \"en\"\n    ):\n        self.name=name\n        self.language = language\n\n    def score(\n        self, candidate: str, reference: str, **kwargs\n    ) -> List[score_result.ScoreResult]:\n        \"\"\"\n        Computes the BERTScore between a candidate (predicted) text and a reference (ground truth) text.\n\n        This method calculates recall, precision, and F1 score based on token-level\n        contextual embeddings, using a pre-trained transformer model.\n\n        Args:\n            candidate (str or List[str]): The candidate text or list of texts to evaluate.\n                Must not be empty or contain only whitespace.\n            reference (str or List[str]): The reference text or list of texts to compare against.\n                Must not be empty or contain only whitespace.\n            **kwargs: Additional keyword arguments for the Hugging Face BERTScore computation.\n\n        Returns:\n            List[score_result.ScoreResult]: A list of `ScoreResult` objects containing:\n                - BERTRecall: The BERTScore recall score.\n                - BERTPrecision: The BERTScore precision score.\n                - BERTF1: The BERTScore F1 score.\n\n        Raises:\n            ValueError: If candidate or reference inputs are empty strings or lists.\n            TypeError: If candidate or reference inputs are not strings or lists of strings.\n        \"\"\"\n        # Validate and normalize inputs\n        def validate_and_normalize(text: Union[str, List[str]]) -> List[str]:\n            if isinstance(text, str):\n                if not text.strip():\n                    raise ValueError(\"Input text cannot be empty or whitespace.\")\n                return [text]\n            if isinstance(text, list):\n                if not all(isinstance(t, str) and t.strip() for t in text):\n                    raise ValueError(\"All elements in the input list must be non-empty strings.\")\n                return text\n            raise TypeError(\"Input must be a string or a list of strings.\")\n\n        candidate = validate_and_normalize(candidate)\n        reference = validate_and_normalize(reference)\n\n        results_dict = bertscore.compute(predictions=candidate, references=reference, lang=self.language)\n\n        # Create score results\n        return [\n            score_result.ScoreResult(value=results_dict[\"recall\"][0], name=\"BERTRecall\"),\n            score_result.ScoreResult(value=results_dict[\"precision\"][0], name=\"BERTPrecision\"),\n            score_result.ScoreResult(value=results_dict[\"f1\"][0], name=\"BERTF1\"),\n        ]\n\nbscore = BERTScore()\n<\/code><\/pre>\n\n\n\n
      After defining BERTScore as a custom metric, we can use it by:<\/span><\/p>\n\n\n\n