Why AI systems need automated evaluation
Agent 02 can retrieve products and generate answers — but how do you know if the answers are actually good? Without a systematic evaluation framework, quality is subjective and regressions go undetected. When a prompt changes, a model updates, or the catalogue grows, you need to know immediately whether the system got better or worse.
The objective was to build an automated eval pipeline that runs on demand, scores responses against known-correct answers, and produces a structured report — the same way engineering teams run test suites before shipping code.
The Problem
Without evals, AI quality is anecdotal. One good demo does not mean the system works reliably across all question types, edge cases, and failure modes.
The Objective
An automated LLM-as-Judge pipeline that scores Agent 02's RAG responses on three dimensions and produces a quantified pass-rate report.
The same pipeline, built in n8n
The eval pipeline is orchestrated in n8n — a webhook triggers a loop over all 20 benchmark questions. For each question, the workflow embeds the query via OpenAI, retrieves the top-3 matching products from Pinecone, generates Agent 02's answer using Claude, then passes the question, retrieved context, and answer to a second Claude instance acting as judge. Scores are aggregated across all questions and returned as a structured pass-rate report.
Claude evaluating Claude
Traditional rule-based evaluation checks for keyword matches — brittle, unable to handle paraphrasing, and blind to reasoning quality. LLM-as-Judge uses a second model instance as an impartial evaluator, assessing responses the way a human expert would.
The full RAG pipeline runs — embed query → retrieve from Pinecone → generate with Claude. The agent produces a natural language answer grounded in retrieved products.
A second Claude instance receives the original question, the ground-truth expected answer, and Agent 02's actual response. It judges purely on answer quality.
Each response is scored 1–5 on Accuracy, Completeness, and Grounding. The judge returns structured JSON with scores and one-sentence reasoning.
Pass/Partial/Fail verdicts are tallied. Average scores per dimension are charted. Results can be filtered by category and exported as JSON.
How each response is scored
Every response is evaluated on three independent dimensions. A response can be factually accurate but incomplete, or complete but contain hallucinated details — each dimension captures a different failure mode.
| Score | Accuracy | Completeness | Grounding |
|---|---|---|---|
| 5 | All facts correct — prices, SKUs, materials exact | Fully addresses question with all key details | Entirely from retrieved products, nothing invented |
| 4 | Minor inaccuracies that do not mislead | Main point covered, minor details missing | Mostly grounded, trivial additions |
| 3 | Mix of correct and incorrect facts | Partially addresses the question | Mix of grounded and invented content |
| 2 | Mostly incorrect facts | Significant gaps in the answer | Significant hallucinations present |
| 1 | Completely wrong or hallucinated | Does not address the question | Mostly or entirely hallucinated |
Verdict thresholds: PASS = average ≥ 4.0 · PARTIAL = average ≥ 2.5 · FAIL = average < 2.5
20 questions across 6 failure modes
The eval dataset is designed to stress-test different aspects of the RAG pipeline — not just easy lookups. Each category targets a specific failure mode that production systems commonly encounter.
Specific Lookup (7) -- price, SKU, materials, stock, colours Category Browsing (4) -- "what bedding for hot sleepers?" Cross-Category (3) -- "what sustainable products do you sell?" Care & Maintenance (3) -- "how do I care for the berber rug?" Edge Cases (2) -- products not in catalogue (tests honest refusal) Comparison (2) -- "difference between jute and sheepskin rug?" Difficulty: Easy (9) · Medium (5) · Hard (4) · Edge (2)
What this agent teaches
Using a second model instance to evaluate responses — the same pattern used by Anthropic, OpenAI, and enterprise AI teams to automate quality measurement.
20 questions across 6 categories and 4 difficulty levels — intentionally targeting different failure modes rather than just easy happy-path queries.
Accuracy, Completeness, and Grounding scored independently — because a response can fail on one dimension while passing others.
Aggregated pass/partial/fail metrics by category and difficulty — the kind of report a PM shares with engineering to prioritise improvements.
Each result shows the agent's answer, the expected answer, scores per dimension, and the judge's reasoning — making failures actionable, not just countable.
Results export as timestamped JSON — shareable with engineering teams, trackable over time, and importable into dashboards or notebooks.
What I would do differently
Edge cases reveal the most important failure modes. The two “products not in catalogue” questions are the most revealing — they test whether the agent honestly says “I don't have that” or hallucinates a product. In production, honest refusal is far better than a confident wrong answer.
The judge prompt is a product decision, not a technical one. How you define the scoring rubric — what counts as a 4 vs a 5 on Accuracy — directly shapes what the system optimises for. This is a PM decision that should be made with the business, not left to engineers.
Evals should run automatically on every change. Running evals manually is useful for demos — but production value comes from running them in CI/CD. Every prompt change, model update, or catalogue expansion should trigger an automatic eval run with regression alerting.
Track scores over time, not just point-in-time. A single eval run tells you current quality. A trend over 10 runs tells you whether the system is improving or degrading. Time-series eval tracking is what separates mature AI teams from teams that just run demos.
Human-in-the-loop review on borderline cases is essential. PARTIAL verdicts are where the most learning happens — a human expert can identify exactly what the model got right and wrong, and use that to improve the prompt or retrieval strategy.