What you will learn

Separate the failed layer
Use counterexamples
Preserve uncertainty in labels

Before you start

RAG Fundamentals

The flow

1.Identify scope
2.Inspect evidence
3.Apply host control
4.Run negative cases
5.Compare benign behavior
6.Review remaining risks

Separate the failed layer

A wrong answer may arise from missing knowledge, a retrieval miss, malformed output, a failed tool or an unsupported generated claim. Calling all of them hallucinations hides the repair location. Classify the pipeline stage using observed signals and independently adjudicated claim labels.

Factuality asks whether a claim is true against a reliable reference. Faithfulness asks whether it follows the supplied evidence. A model can faithfully repeat an outdated or incorrect document; a relevant-looking citation can accompany a false statement.

Use counterexamples

The lab classifier checks authorization before parsing and tools, then corpus availability and retrieval, then a trusted claim label. It does not infer truth by looking for confident wording. A fluent sentence and a low temperature are not correctness evidence.

Compare an unknown source ID, an existing source with a contradicted field and a question whose evidence never entered retrieval. Their answers may look equally plausible, but their failure classes and responsible components differ.

Preserve uncertainty in labels

Do not force every disagreement into true/false. SUPPORTED, CONTRADICTED and INSUFFICIENT_EVIDENCE describe the closed-world fixture. Natural language requires evidence spans, annotation rules and adjudication; this module is not a semantic judge.

The dataset pairs English/Turkish prompts with the same fixture operation. It checks language-field coverage, not bilingual model comprehension. Keep this limitation next to the reported metrics rather than hidden in a footnote.

Try it

taxonomy.mjs

javascript

1

2

3

4

5

6

7

8

9

10

11

12

// Trusted diagnostic signals/labels, not a classifier that infers truth from prose.
export function classifyFailure({ authorized, parsed, toolSucceeded, relevantInCorpus, relevantRetrieved, claimLabel }) {
  if (!authorized) return 'AUTHORIZATION';
  if (!parsed) return 'OUTPUT_CONTRACT';
  if (!toolSucceeded) return 'TOOL_FAILURE';
  if (!relevantInCorpus) return 'KNOWLEDGE_GAP';
  if (!relevantRetrieved) return 'RETRIEVAL_MISS';
  if (claimLabel === 'CONTRADICTED') return 'CONTRADICTION';
  if (claimLabel === 'INSUFFICIENT_EVIDENCE') return 'UNSUPPORTED_CLAIM';
  if (claimLabel === 'SUPPORTED') return 'SUPPORTED';
  throw new Error('Missing adjudicated claim label');
}

Run npm run examples:security and npm run eval:security. This source is a deterministic teaching fixture; it does not prove live model safety, NLI, production auth or OS/network isolation.

Interview questions

How do you distinguish retrieval failure from an unsupported claim?

Factuality asks whether a claim is true against a reliable reference. Faithfulness asks whether it follows the supplied evidence. A model can faithfully repeat an outdated or incorrect document; a relevant-looking citation can accompany a false statement.

Can a faithful answer still be false?

Compare an unknown source ID, an existing source with a contradicted field and a question whose evidence never entered retrieval. Their answers may look equally plausible, but their failure classes and responsible components differ.

Why is the classifier not a semantic judge?

The dataset pairs English/Turkish prompts with the same fixture operation. It checks language-field coverage, not bilingual model comprehension. Keep this limitation next to the reported metrics rather than hidden in a footnote.

Practice

Extend one negative scenario and its authorized positive counterexample for Hallucinations & Failure Taxonomy.

Check your solution

State the trusted input and effect boundary.
Assert the expected status and observed effect.
Report coverage and limitations without claiming production readiness.

Hallucinations & Failure Taxonomy