Sections in this lesson

Testing & Evaluation for Spring AI

Separate unit behavior, provider contracts, database retrieval and live answer quality.

Reviewed: · Content owner: Interview Preparator

What you will learn

  • Build a versioned set of expected sources and abstentions.
  • Test provider serialization without claiming live model quality.
  • Use denied cases as release gates, not average-score offsets.

Before you start

The flow

  1. 1.Version cases
  2. 2.Run layer tests
  3. 3.Compare sources
  4. 4.Inspect failures
  5. 5.Decide release

Four different kinds of evidence

Unit tests replace VectorStore or ChatModel to isolate service decisions. A provider contract test uses the real Ollama adapter against a local HTTP fixture. Integration tests use a real pgvector database with deterministic model doubles. Live evaluation uses actual chosen models and labeled real questions.

Do not merge these into one claimed quality score. Passing an HTTP fixture proves serialization and parsing, not that Ollama is installed or a downloaded model answers correctly.

A small, inspectable dataset

The JSON fixture defines five cases: two tenants asking about records, a supported virtual-thread question, an unknown topic and a topic inaccessible to the second tenant. The integration test asserts exact status and source IDs from actual retrieval.

RagService tests separately reject malformed output, absent/fabricated citations, invalid input and provider failure. The tool/memory tests verify denied resources and cross-user context isolation. A regression in a denied case should block release even if common-case averages improve.

Grow toward production evaluation

Add real multilingual questions, source/claim annotations, stale-document cases and repeated runs. Record model, prompt, dataset and index versions with latency and usage. Calibrate any LLM judge against human labels; five deterministic cases are regression fixtures, not a statistical benchmark.

Try it

rag-cases.json
json
1
2
3
4
5
6
7
[
  {"id":"alpha-record","tenant":"tenant-a","question":"record","status":"ANSWER","sources":["00000000-0000-0000-0000-000000000001"]},
  {"id":"beta-record","tenant":"tenant-b","question":"record","status":"ANSWER","sources":["00000000-0000-0000-0000-000000000003"]},
  {"id":"alpha-thread","tenant":"tenant-a","question":"virtual thread","status":"ANSWER","sources":["00000000-0000-0000-0000-000000000002"]},
  {"id":"unknown","tenant":"tenant-a","question":"spaceship","status":"ABSTAIN","sources":[]},
  {"id":"inaccessible","tenant":"tenant-b","question":"virtual thread","status":"ABSTAIN","sources":[]}
]

examples/spring-ai-lab/src/test/resources/rag-cases.json · mvn -f examples/spring-ai-lab/pom.xml -Pintegration verify. Java 21+, Spring Boot 4.0.5, Spring AI 2.0.2 where applicable. The demo uses synthetic models; live inference quality is a separate check.

Interview questions

What does the golden fixture test measure?

The expected status and retrieved source set under deterministic conditions; not general semantic quality.

Can provider fixtures replace live evaluation?

No. Adapter contracts and actual inference behavior are separate risks.

How should an access leak affect the score?

Treat it as a hard failure requiring investigation, not a small deduction in an aggregate average.

Practice

Add a Turkish query, annotate its expected source and record the real embedding model separately from fixture results.

Check your solution

  • Fixture version is retained in source control.
  • Denied and unknown cases are present.
  • The report states which model mode ran.

Official references

Continue learning

Category sequence

Spring AI

Lesson 7 / 7

Was this lesson useful?

Change notes: Phase 2: bilingual lesson with versioned executable reference.