LLM Evaluation

The flow

1. 1.Dataset
2. 2.Versioned run
3. 3.Score
4. 4.Inspect failures
5. 5.Release gate

Start with behavior, not a single score

Write cases for known answers, missing evidence, private data, malformed input and tool failure. Record expected behavior and source IDs. Keep a held-out set and version datasets to make comparisons meaningful.

Score structure and permission violations deterministically where possible. Use a human rubric for answer quality; if using an LLM judge, calibrate it against human labels and inspect disagreements. A judge is not an independent truth oracle.

Inspect slices and traces

A high average can hide complete failure on Turkish inputs or access-denial cases. Track per-slice results, retrieval recall, groundedness, tool success and appropriate abstention. Safety failures may need a hard gate instead of averaging.

Record prompt/model/dataset versions plus latency and usage. Repeated runs expose variance; do not present three synthetic examples as a production benchmark or a statistically reliable success rate.

Use failures to guide the next change

Inspect failed cases before changing the model. Missing evidence calls for retrieval work; invalid JSON calls for contract handling; forbidden actions call for policy enforcement. Compare the candidate to the previous release and retain a rollback path.

Try it

import assert from 'node:assert/strict';
const cases = [
  { id: 'known', expected: 'ANSWER', actual: 'ANSWER', ms: 200, tokens: 80 },
  { id: 'unknown', expected: 'ABSTAIN', actual: 'ANSWER', ms: 300, tokens: 90 },
  { id: 'private', expected: 'ABSTAIN', actual: 'ABSTAIN', ms: 250, tokens: 40 },
];
const successRate = cases.filter(c => c.actual === c.expected).length / cases.length;
const failures = cases.filter(c => c.actual !== c.expected).map(c => c.id);
assert.equal(successRate, 2 / 3);
assert.deepEqual(failures, ['unknown']);
assert.ok(successRate < 0.9, 'A 90% release gate should reject this fixture run');
console.log({ successRate, failures, meanLatency: cases.reduce((n, c) => n + c.ms, 0) / cases.length });
// Synthetic results: not a benchmark, groundedness judge or live provider eval.

Run with Node.js 22: node examples/generative-ai/llm-evaluation.mjs. This is a deterministic teaching example, not a live model call.

Interview questions

Official references

• Spring AI — evaluation testing

Continue learning

• RAG Fundamentals →
• Model Evaluation →