LLM-as-a-judge is an evaluation method for assessing the output quality of AI apps. Think of it as a mechanism that lets you know whether your AI agent is producing useful work or slop.
LLM-as-a-judge uses one language model to assess the outputs of another. One model is the app model that users interact with — this is the model that you want to evaluate. The other model is the judge model, which performs the evaluation.
The practical payoff is that you can automate quality checks that would otherwise require human review. A judge can evaluate thousands of outputs in minutes, flag hallucinations or off-topic responses, and give you a written explanation for every score. That turns evaluation from a manual bottleneck into something you can run on every deployment.
If your first reaction is “wait, isn’t using an LLM to grade an LLM circular?” — that’s a reasonable question, and we’ll get into why it actually works in more detail later in this blog post. But the short version is that verifying an answer is easier than generating one, and the research shows that you can tune LLM judges to match human judgements. We’ll also cover the known biases in LLM judges and how to mitigate them, walk through the major judging architectures, and show you how to implement evaluation in practice with Opik.
What LLM-as-a-Judge Actually Means
LLM-as-a-judge is the name for the entire judge-model AI evaluation method. You give the judge model output from the app model, plus an evaluation prompt with specific criteria — for example, you could prompt it to assess the app model’s output for helpfulness, accuracy, tone, or whatever matters for your use case — and the judge model then scores the app model’s output, typically with a written rationale explaining its reasoning.
If you’re coming from traditional software development, this solves a problem you’ve probably already noticed: LLM outputs aren’t deterministic, so you can’t write conventional unit tests against expected values. The same prompt can produce different (but equally valid) responses, and “correct” is often dependent on multiple factors. An LLM judge handles this by evaluating qualities rather than exact matches — did the response actually answer the question? Is the information grounded in the context you provided? Does it sound like your brand?
The LLM-as-a-judge concept was formalized in a 2023 NeurIPS paper by Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” The researchers introduced MT-Bench (a multi-turn question set) and the Chatbot Arena (a crowdsourced comparison platform), then systematically studied whether strong LLMs could stand in for human evaluators. Their central finding: GPT-4 achieved over 80% agreement with human preferences — roughly the same level of agreement that human evaluators reached with each other.
That 80% number deserves some context. Human inter-annotator agreement is often lower than people assume. That means that when two humans look at the same material and use the same evaluation methods, they’ll agree with each other about 80% of the time. The Zheng et al. study found that agreement between individual human raters hovered in the same range, meaning the LLM judge was performing at human-level consistency, not exceeding some perfect gold standard. The benchmark for LLM-as-a-judge is parity with the realistic noise floor of human evaluation, not parity with some imagined perfect evaluator.
Why Deterministic ML Metrics Hit a Wall for GenAI Output Evaluation
If you’re not interested in the history of Natural Language Processing (NLP) evaluation metrics, feel free to skip to the next section. To summarize, older automated metrics compare word overlap between generated outputs and correct reference answers. This deterministic approach fails for open-ended generation, because many phrasings can be equally correct. LLM judges solve this by evaluating meaning instead of matching words.
Before LLM-as-a-judge, the standard approach to automated evaluation for generated text was comparing the generated text against a correct or reference answer, and measuring how many words or phrases matched. Metrics like BLEU and ROUGE work this way. They’re fast, deterministic, and work reasonably well when there’s a single correct answer — machine translation with a known target, for instance.
These metrics fall apart for open-ended generation. Consider a support chatbot that’s asked “How do I reset my password?” A helpful response might say “Go to Settings, click Security, then select ‘Reset Password’” while the reference answer says “Navigate to your account preferences and choose the password reset option.” A human sees these as equivalent, with the same intent and the same outcome. But because the phrasing has zero word overlap, traditional metrics score it poorly.
This is where correlation with human judgment comes in. When researchers evaluate a new metric, they check whether it ranks outputs the same way humans do. If humans prefer Response A over Response B, does the metric agree? The G-Eval paper from Microsoft Research (Liu et al., EMNLP 2023) found that BLEU and ROUGE showed low correlation with human preferences, especially for tasks requiring creativity. G-Eval with GPT-4 achieved substantially higher correlation. Correlation means that when the metric says a response is good, humans tend to agree that it’s good, and when the metric says a response is bad, humans agree that it’s bad.
This matters practically because optimizing for a metric that doesn’t match human judgment actively misleads your development process. You make changes that improve the score while making the actual output worse.
That said, deterministic checks are still useful for many relevant things, like format validation. For example, if your agent is supposed to return valid JSON or include a required legal disclaimer, a deterministic check is faster, cheaper, and more reliable than an LLM judge. The right approach layers both: deterministic metrics for structural requirements, LLM judges for semantic quality.
The Practical Case for LLM Judges
Three properties make LLM-as-a-judge essential for teams building production LLM applications: speed, explainability, and consistency at scale.
- Speed and cost. Evaluating 1,000 model responses with human reviewers can take days or weeks and cost thousands of dollars. An LLM judge does it in minutes for a fraction of the cost. With this efficiency gain, you can run evaluations on every pull request, every prompt change, every model swap, turning evaluation from a periodic audit into continuous regression testing.
- Explainability. Unlike BLEU, which returns an opaque float, an LLM judge can articulate why it scored a response the way it did. If a response gets penalized, the judge can point to the specific claim that contradicted the context or which part of the user query went unaddressed. This transforms evaluation from a measurement step into a debugging tool. Instead of knowing that your new prompt “decreased performance by 3%,” you can see that the new prompt is causing the model to hallucinate details from retrieved documents or ignore the user’s actual question.
- Consistency at scale. Human reviewers drift. They interpret rubrics differently on Monday morning versus Friday afternoon. Different annotators bring different priors. The Zheng et al. paper documented this: human inter-annotator agreement was around 80%, with some tasks falling lower. An LLM judge, given the same prompt and criteria, applies the same standards to every output. This consistency makes longitudinal comparisons meaningful. You can track quality over weeks and months with confidence that score changes reflect actual quality changes, not evaluator mood.
A Caveat About LLM Judge Consistency
LLM judges aren’t perfectly deterministic either. Temperature settings introduce variation, and even at temperature 0, some providers introduce small sampling differences. Practitioners should be aware of potential “score drift” where a judge may produce slightly different distributions across evaluation runs. Running evaluations with low temperature and averaging across multiple judgments for high-stakes decisions helps stabilize the results.
Known LLM-as-a-Judge Biases and How to Mitigate Them
The early criticism of LLM-as-a-judge was pointed: if you use a model to evaluate another model, aren’t you just encoding the judge’s preferences and blind spots into your quality signal? This concern was legitimate, and the research community has since identified specific, measurable biases that affect LLM judges. The good news is that these are engineering problems with known mitigation strategies.
Position Bias
When an LLM judge compares two responses side by side (pairwise evaluation), it often favors whichever response appears first or last in the prompt. This positional preference has nothing to do with quality. Zheng et al. documented this in the original MT-Bench study, and a more recent systematic analysis by Shi et al. (“Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge,” 2024) confirmed that the effect persists across model families, though its direction (favoring first vs. last) varies by model.
Mitigation: Run each pairwise comparison twice with the response order swapped, then average the results. This is called position-consistency checking. If the judge picks Response A when it appears first and Response B when it appears first, that’s a signal the comparison is too close to call, and you can flag it as a tie. Most LLM evaluation frameworks support this automatically.
Verbosity Bias
LLM judges tend to assign higher scores to longer responses, even when the extra length is padding or repetition. This is a real problem if your application values conciseness, because a tightly written three-sentence answer might score lower than a five-paragraph response that says the same thing with filler.
Mitigation: Explicitly include conciseness in your evaluation rubric. You can also normalize for length in your scoring criteria (e.g., “Penalize responses that include unnecessary repetition or filler content”). Hamel Husain, in his widely-referenced guide to LLM-as-a-judge, recommends using binary pass/fail judgments over Likert scales partly for this reason, because a pass/fail verdict is harder to game with verbosity.
Self-Preference Bias (Self-Enhancement)
Models tend to rate their own outputs, or outputs from the same model family, higher than outputs from other models. This self-enhancement effect has been documented across several model families and is thought to be related to perplexity: models find text that matches their own training distribution more “natural” and therefore score it higher.
Mitigation: Use a judge model from a different family than the model being evaluated. If you’re evaluating outputs from a GPT-based pipeline, consider using Claude or an open-source model as the judge. Alternatively, use purpose-built evaluation models (more on this below). Another approach is “LLM juries” — aggregating scores from multiple judge models to reduce any single judge’s bias.
Leniency and Central Tendency
Some LLM judges cluster their scores in the middle of the scale, avoiding extreme ratings. On a 1–10 scale, you’ll see a lot of 6s and 7s but very few 1s or 10s. This reduces the evaluation’s ability to distinguish between mediocre and excellent outputs.
Mitigation: Use narrower scales (1–5 or even binary pass/fail), provide calibration examples that demonstrate what each score level looks like, or use few-shot prompting with examples of low, medium, and high-quality responses and their corresponding scores. Husain argues against multi-point scales entirely — in his experience across 30+ companies, domain expert pass/fail judgments correlate better with actual quality than granular numeric scores.
Judge Architectures: Choosing the Right Approach
There are several ways to structure an LLM judge, each with different tradeoffs around cost, granularity, and reliability.
Pointwise Evaluation (Direct Scoring)
Pointwise evaluation is the most common architecture in production. The judge receives a single prompt-response pair and scores it against a rubric. This is fast, doesn’t require a reference model, and works well for monitoring live traffic.
Pointwise scoring is the backbone of production evaluation because you can run it on every trace without needing to compare against anything else. The downside is that absolute scores are harder to calibrate than relative comparisons. A “7 out of 10” means different things to different judges (and to the same judge on different days).
You can stabilize pointwise scoring with reference-based evaluation: provide the judge with a “gold standard” answer as an anchor, then ask it to score the candidate response relative to that reference. This grounds the judgment and reduces drift.
Pairwise Comparison
Show the judge two responses to the same input and ask it to pick the better one. This is the format behind Arena and MT-Bench. Relative comparisons are inherently easier than absolute scoring — both for humans and for models. You don’t need a calibrated scale; you just need to decide which response is better.
The tradeoff is cost. Comparing five model variants requires ten pairwise battles to cover all combinations. With ten variants, that’s 45 comparisons. The scaling is quadratic. Pairwise comparison is best suited for periodic A/B tests between model versions or prompt variants, not for continuous production monitoring.
G-Eval: Chain-of-Thought Evaluation
The G-Eval framework (Liu et al., EMNLP 2023) adds a structured reasoning step before scoring. Instead of asking the judge to immediately output a number, you first prompt it to generate a chain of detailed evaluation steps. For example, “1. Check for factual accuracy against the context. 2. Verify that all parts of the user question are addressed. 3. Assess whether the tone matches the specified persona.”. Have the judge execute those steps before producing a final score.
This “show your work” approach has two benefits. First, it produces better-calibrated scores: the G-Eval paper showed improved correlation with human judgment compared to direct scoring. Second, the intermediate reasoning is useful for debugging. When a response scores poorly, you can read the judge’s rationale to understand exactly what failed.
Opik implements G-Eval natively (we’ll show code examples below), and it uses the probability-weighted scoring approach recommended in the original paper — requesting log probabilities from the judge model and computing a weighted average for more robust scores.
Using Dedicated Evaluation Models
General-purpose chat models carry all their biases into the judging task. An alternative is to use models specifically fine-tuned for evaluation. The research community has produced several, and the broader trend is toward purpose-built judges that separate the evaluation capability from general chat ability.
The key advantage of dedicated evaluation models is that they can be independently validated against human judgments, creating a more transparent evaluation pipeline. The key risk is that they may not generalize as well to novel tasks or domains outside their training distribution. For most production teams, using a strong general-purpose model (GPT-5.2, Claude) as the judge with well-crafted evaluation prompts remains the most practical starting point.
Implementing LLM-as-a-Judge with Opik
Enough theory; let’s talk about how to implement. Here’s how to set up LLM-as-a-judge GenAI evaluation in Opik, covering both offline evaluation during development and online scoring in production.
The Evaluation Workflow
Opik’s LLM evaluation follows a five-step process:
- Prepare a dataset. Create a set of inputs (and optionally, expected outputs) that represent the scenarios your application should handle well. This is your “golden set” — the baseline for all tests.
- Instrument your application. Use Opik’s
@trackdecorator to log every interaction to the platform. This captures inputs, outputs, and intermediate steps for later analysis. - Apply heuristic checks first. Before running expensive LLM judges, use deterministic metrics. For example, use Equals for exact matches,
RegexMatchfor format validation, andIsJsonfor structured output checks. These are fast, cheap, and reliable for structural requirements. - Run LLM-as-a-judge metrics. Apply semantic LLM evaluation metrics —
Hallucination, AnswerRelevance, ContextPrecision, G-Eval, and others — to assess the qualitative dimensions that heuristic checks can’t capture. - Compare and iterate. Use Opik’s experiment comparison to see how different versions of your application stack up: “Version A has 85% relevance, but Version B trades 3% relevance for a 15% reduction in hallucination rate.”
Using Built-in Metrics: Hallucination and Answer Relevance
Opik provides more than 20 pre-built LLM-as-a-judge eval metrics that you can use out of the box. Here’s how to run a hallucination check and a relevance evaluation against a test dataset:
from opik import Opik, track
from opik.integrations.openai import track_openai
from opik.evaluation import evaluate
from opik.evaluation.metrics import Hallucination, AnswerRelevance
import openai
openai_client = track_openai(openai.OpenAI())
opik_client = Opik()
# Your LLM application — the thing being evaluated
@track
def support_bot(input: str) -> dict:
# In a real app, you'd retrieve context from a vector store
context = ["Returns are accepted within 30 days of purchase.",
"Refunds are processed to the original payment method."]
response = openai_client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": f"Answer using this context: {context}"},
{"role": "user", "content": input}
],
)
return {
"input": input,
"output": response.choices[0].message.content,
"context": context,
}
# Define the metrics
hallucination_metric = Hallucination()
relevance_metric = AnswerRelevance()
# Run the evaluation
dataset = opik_client.get_dataset("support_questions_golden_set")
results = evaluate(
experiment_name="support_bot_v1",
dataset=dataset,
task=support_bot,
scoring_metrics=[hallucination_metric, relevance_metric],
)
A few things to note about this code. The Hallucination metric returns a binary score: 0 means no hallucination detected, 1 means the judge found unsupported claims. The AnswerRelevance metric checks whether the output actually addresses the input question. Together, they give you a “safety-utility” balance — you want responses that are helpful (relevant) and grounded in your context (not hallucinated).
By default, Opik uses GPT-5-nano as the judge model, but you can swap in any model supported by LiteLLM by setting the model parameter:
hallucination_metric = Hallucination(model="anthropic/claude-sonnet-4-20250514")
relevance_metric = AnswerRelevance(model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0")This flexibility is important for mitigating self-preference bias. If your application runs on GPT-4o, using Claude as the judge avoids the self-enhancement effect where the model scores its own family’s outputs more favorably.
Building Custom Metrics with G-Eval
The built-in metrics cover common evaluation dimensions, but production applications often have domain-specific requirements. A healthcare chatbot needs to be evaluated for clinical accuracy. A financial assistant needs compliance checking. A brand chatbot needs tone adherence.
Opik’s GEval metric lets you define custom evaluation criteria without building a judge from scratch:
from opik.evaluation.metrics import GEval
tone_metric = GEval(
task_introduction="You are an expert judge evaluating whether a customer support response maintains a professional, empathetic tone.",
evaluation_criteria="The response should be polite and considerate. It should avoid jargon, slang, or dismissive language. It should acknowledge the customer's frustration when appropriate.",
)
Score a single output
result = tone_metric.score(
output="INPUT: I've been waiting 3 weeks for my refund!\nOUTPUT: I completely understand your frustration with the delay. Let me look into this right away and get you an update within 24 hours."
)
print(result.value) # 0.0 to 1.0 (normalized from 0-10)
print(result.reason) # The judge's written rationaleUnder the hood, G-Eval first expands your task description into step-by-step evaluation instructions (the chain-of-thought), then uses those steps as a rubric when scoring. The model outputs an integer between 0 and 10, which Opik normalizes to the 0–1 range. When the judge model exposes log probabilities, Opik computes a probability-weighted average of scores for more robust results, following the approach recommended in the original G-Eval paper.
Opik also ships opinionated G-Eval presets for common evaluation scenarios:
ComplianceRiskJudgeflags non-compliant or risky statements (finance, healthcare, legal).SummarizationConsistencyJudgechecks if summaries are faithful to source material.DialogueHelpfulnessJudgerates how helpful an assistant reply is in context.AgentTaskCompletionJudgeevaluates whether an agent fulfilled its assigned task.AgentToolCorrectnessJudgeassesses whether an agent invoked tools appropriately.
Each of these inherits from GEval and can be customized with different judge models and temperature settings.
Evaluating RAG Pipelines: Context Precision and Context Recall
If you’re building a retrieval-augmented generation (RAG) application, you need to evaluate both the retrieval quality and the generation quality. Opik provides metrics for each:
ContextPrecisionchecks whether the retrieved context is actually relevant to the question, scored against an expected output. High precision means the retriever is surfacing useful documents, not noise.ContextRecallmeasures how well the generated answer uses the available supporting context, scored against an expected output. Low recall suggests the model is ignoring relevant retrieved information.Hallucinationcatches cases where the model generates claims that aren’t supported by the retrieved context.
These three metrics together give you a diagnostic view of your RAG pipeline. If hallucination rates spike, you can look at context precision to determine whether the problem is bad retrieval (garbage in, garbage out) or bad generation (the model is ignoring good context).
Online Evaluation: Scoring Production Traffic
Evaluation shouldn’t stop after deployment. In production, you’re dealing with real user inputs that may differ dramatically from your test dataset. Opik’s Online Evaluation Rules let you define LLM-as-a-judge metrics that automatically score a subset of live production traces.
You configure these rules in the Opik dashboard by specifying a judge model, a prompt template with variable mappings to your trace data, and a scoring definition. Opik comes pre-configured with three online evaluation metrics: Hallucination, Moderation, and Answer Relevance.
This creates a continuous feedback loop. When a judge flags a production response as a hallucination, that trace can be routed to a manual review queue. A human reviewer verifies the issue, adds the problematic input to the golden dataset, and uses it to retest the application after a fix. This closed-loop process is how you maintain quality as your application scales to thousands or millions of interactions.
Evaluating Agents: Beyond Input-Output Scoring
As LLM applications evolve from simple chatbots into multi-step agents that browse the web, write code, and call APIs, evaluation needs to evolve too. Evaluating an agent requires looking at the entire decision-making trajectory, not just the final output.
An agent might produce a correct final answer but get there through an inefficient path, such as calling the wrong tool three times before finding the right one. Or it might produce the wrong answer because its retrieval step failed, even though its generation step worked perfectly on the (bad) context it received. Final-output-only evaluation can’t distinguish between these failure modes.
Opik supports trajectory-level evaluation through several specialized metrics:
TrajectoryAccuracyscores how closely an agent’s actual steps match expected steps.AgentTaskCompletionJudgeevaluates whether the overall goal was achieved.AgentToolCorrectnessJudgechecks whether tools were selected and invoked correctly.
Combined with Opik’s LLM tracing infrastructure, these metrics let you inspect specific spans within an agent trace to pinpoint where things went wrong. If an agent failed to answer a question, you can look at the retrieval span to determine whether the search step failed to find relevant documents — exonerating the generation step — or whether the model had good context and still produced a bad answer.
Tips for Building Effective LLM Judges
Based on the research literature and practical experience from teams shipping LLM applications, here are guidelines that consistently improve judge quality:
Start with binary pass/fail before scaling to numeric scores. A pass/fail verdict forces you to define what “acceptable” means before worrying about what distinguishes a 3 from a 4. Once you have a clear pass/fail boundary, you can add granularity.
Include few-shot examples in your judge prompt. Adding a few examples of outputs with appropriate scores to the judge prompt increases scoring consistency significantly.
Validate your judge against human labels. Before trusting an LLM judge in production, build a small calibration set (30–50 examples, annotated by domain experts) and measure agreement. If the judge disagrees with your experts more than 20% of the time on clear-cut cases, iterate on the prompt before deploying.
Limit evaluation criteria to 3–5 dimensions per judge call. As noted in multiple practical guides, including the Towards Data Science walkthrough, evaluating too many dimensions at once dilutes the judge’s focus and reduces scoring quality. Run separate judge calls for separate concerns.
Use separate judge models for separate concerns. Hallucination detection and tone evaluation are different skills. Using specialized judge prompts (or specialized metrics like Opik’s built-in ones) for each concern produces better results than a single monolithic evaluation prompt.
Where LLM-as-a-Judge is Heading
The LLM-as-a-judge paradigm has moved from research curiosity to production necessity in about two years. The core insight — that verifying a solution is easier than generating one — has proven robust across use cases, model families, and application types.
The active frontiers are in agent evaluation (trajectory-level judging that can diagnose complex multi-step failures), multi-modal evaluation (judging outputs that include images, code, and structured data), and judge efficiency (getting reliable evaluations from smaller, faster, cheaper models). Opik is tracking all of these with recent additions like multimodal evaluation support and conversation-level metrics for multi-turn agents.
If you’re building LLM applications today, you should think about how quickly you can get an automated feedback loop in place. A hallucination metric and a relevance metric running against a 50-example golden set will teach you more about your application’s behavior in an afternoon than weeks of manual spot-checking.
Free, Open-Source LLM-as-a-Judge Evaluation with Opik
Opik comes with everything you need to run LLM-as-a-judge evaluations against your LLM application or AI agents in development, testing, and production. The complete LLM observability and evaluation featureset is available in both the open-source version and the free cloud version, so you can start building and testing today with no strings attached. Choose your version and visit our QuickStart Guide to get up and running in minutes.
