Sections in this lesson

Retrieval Failure Analysis

Separate a knowledge gap from a ranking miss.

Reviewed: · Content owner: Interview Preparator

What you will learn

  • Ask whether evidence exists
  • Diagnose before widening context
  • Preserve the authorization boundary

Before you start

The flow

  1. 1.Identify scope
  2. 2.Inspect evidence
  3. 3.Apply host control
  4. 4.Run negative cases
  5. 5.Compare benign behavior
  6. 6.Review remaining risks

Ask whether evidence exists

Separate a knowledge gap from a ranking miss. If the authorized corpus lacks the answer, changing top-K cannot manufacture evidence. If the evidence exists but was not retrieved, inspect ingestion, metadata, chunk boundaries and query compatibility.

Use relevant source labels to calculate recall at K. An empty relevance set has no recall denominator and returns null in the example. Do not silently count unanswerable questions as perfect retrieval.

Diagnose before widening context

Larger K can increase latency and distract the generator. Query rewriting and reranking may help a measured failure but can also introduce drift. Change one versioned component and compare the same held-out cases.

The fixture distinguishes empty retrieval, a wrong document, expired evidence and a foreign source. They may all produce ABSTAIN, but the responsible repair is not identical.

Preserve the authorization boundary

A selected-document filter narrows the retrieved candidate set; it must not become an arbitrary ID fetch. The existing Java source selector operates after tenant-scoped top-three retrieval, so an allowed selected source outside those candidates may be missed.

Document that recall limitation instead of calling the selector complete corpus search. The real pgvector tests now also reject unapproved or missing-approval documents; this is an ingestion gate, not a semantic correctness proof.

Try it

reliability.mjs
javascript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
// Closed-world fact fixtures only. Equality of structured facts is NOT an NLI judge.
const object = value => value !== null && typeof value === 'object' && !Array.isArray(value);
export function authorizedSources(sources, scope, now) {
  if (!object(scope) || typeof scope.tenant !== 'string' || typeof scope.user !== 'string' || !Number.isFinite(now)) throw new Error('Invalid trusted scope/time');
  return sources.filter(source => source.tenant === scope.tenant && source.approved === true && !source.deleted
    && source.expiresAt > now && source.readers.includes(scope.user));
}
export function verifyClaims(claims, sources, scope, now) {
  if (!Array.isArray(claims) || claims.length > 20) throw new Error('Invalid claims');
  const evidence = authorizedSources(sources, scope, now);
  return claims.map(claim => {
    if (!object(claim) || Object.keys(claim).sort().join(',') !== 'key,sourceId,value'
      || ![claim.key,claim.sourceId,claim.value].every(value => typeof value === 'string' && value.length > 0 && value.length <= 200)) throw new Error('Invalid structured claim');
    const source = evidence.find(item => item.id === claim.sourceId);
    const value = source?.facts[claim.key];
    return { ...claim, label: value === undefined ? 'INSUFFICIENT_EVIDENCE' : value === claim.value ? 'SUPPORTED' : 'CONTRADICTED' };
  });
}
export function answerPolicy(claims, sources, scope, now) {
  const labels = verifyClaims(claims,sources,scope,now);
  return { status:labels.length > 0 && labels.every(item => item.label === 'SUPPORTED') ? 'ANSWER' : 'ABSTAIN', claims:labels };
}
export function retrievalRecall(relevantIds, retrievedIds) {
  if (!relevantIds.length) return null;
  return [...new Set(relevantIds)].filter(id => retrievedIds.includes(id)).length / new Set(relevantIds).size;
}
export function judgeAgreement(human, judge) {
  if (human.length !== judge.length) throw new Error('Unpaired annotations');
  return human.length ? human.filter((label,index) => label === judge[index]).length / human.length : null;
}

Run npm run examples:security and npm run eval:security. This source is a deterministic teaching fixture; it does not prove live model safety, NLI, production auth or OS/network isolation.

Interview questions

Can a larger K repair a missing corpus?

Use relevant source labels to calculate recall at K. An empty relevance set has no recall denominator and returns null in the example. Do not silently count unanswerable questions as perfect retrieval.

Why separate stale from wrong retrieval?

The fixture distinguishes empty retrieval, a wrong document, expired evidence and a foreign source. They may all produce ABSTAIN, but the responsible repair is not identical.

Does source selection search the full corpus?

Document that recall limitation instead of calling the selector complete corpus search. The real pgvector tests now also reject unapproved or missing-approval documents; this is an ingestion gate, not a semantic correctness proof.

Practice

Extend one negative scenario and its authorized positive counterexample for Retrieval Failure Analysis.

Check your solution

  • State the trusted input and effect boundary.
  • Assert the expected status and observed effect.
  • Report coverage and limitations without claiming production readiness.

Official references

Continue learning

Category sequence

AI Reliability & Hallucinations

Lesson 4 / 10

Was this lesson useful?

Change notes: 2026-10-31: security/reliability curriculum and source-bound fixture.