What you will learn

An ID is not support
Apply access before grounding
Calibrate a semantic extension separately

Before you start

Reliability Evaluation & Metrics

The flow

1.Identify scope
2.Inspect evidence
3.Apply host control
4.Run negative cases
5.Compare benign behavior
6.Review remaining risks

Does the evidence actually support the claim?

Closed-world experiment: fixed field/value comparison. No natural-language entailment or live model call is performed.

Scenario

What is the approved retention in days for synthetic plan 1?

Candidate claim

[
  {
    "key": "retentionDays",
    "value": "7",
    "sourceId": "DOC-1"
  }
]

Retrieved source

[
  {
    "id": "DOC-1",
    "tenant": "tenant-a",
    "readers": [
      "alice"
    ],
    "approved": true,
    "deleted": false,
    "expiresAt": 2000,
    "facts": {
      "retentionDays": "7"
    }
  }
]

An ID is not support

Split a response into checkable claims and bind each to evidence. A source identifier may be valid while the stated value is absent or contradicted. Citation membership, semantic support and source truth are separate checks.

The lab uses approved structured facts such as retentionDays. Exact field/value equality labels a fixture claim; it does not handle paraphrases, inference, negation or evidence quality as a general natural-language entailment system would need to.

Apply access before grounding

authorizedSources filters tenant, reader, approval, deletion and expiry before claims are compared. A fact from another tenant cannot make an answer acceptable even if it happens to be correct. Current authorization must also be rechecked when reading a cached answer.

Missing keys produce INSUFFICIENT_EVIDENCE; differing known values produce CONTRADICTED. Only a non-empty set of entirely supported claims yields ANSWER. This conservative policy is inspectable but can abstain too often.

Calibrate a semantic extension separately

For prose, annotate atomic claims and source spans with an adjudication guide. Compare a judge with independent labels and include claims whose source is real but irrelevant. Keep supported-but-stale and supported-by-unreliable-source cases distinct.

The Java service intentionally demonstrates its citation-ID-only limitation in a test: an unsupported statement with an allowed ID can pass. The new Node verifier is a separate closed-world teaching control, not a hidden claim that Java now verifies all semantics.

Try it

reliability.mjs

javascript

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

// Closed-world fact fixtures only. Equality of structured facts is NOT an NLI judge.
const object = value => value !== null && typeof value === 'object' && !Array.isArray(value);
export function authorizedSources(sources, scope, now) {
  if (!object(scope) || typeof scope.tenant !== 'string' || typeof scope.user !== 'string' || !Number.isFinite(now)) throw new Error('Invalid trusted scope/time');
  return sources.filter(source => source.tenant === scope.tenant && source.approved === true && !source.deleted
    && source.expiresAt > now && source.readers.includes(scope.user));
}
export function verifyClaims(claims, sources, scope, now) {
  if (!Array.isArray(claims) || claims.length > 20) throw new Error('Invalid claims');
  const evidence = authorizedSources(sources, scope, now);
  return claims.map(claim => {
    if (!object(claim) || Object.keys(claim).sort().join(',') !== 'key,sourceId,value'
      || ![claim.key,claim.sourceId,claim.value].every(value => typeof value === 'string' && value.length > 0 && value.length <= 200)) throw new Error('Invalid structured claim');
    const source = evidence.find(item => item.id === claim.sourceId);
    const value = source?.facts[claim.key];
    return { ...claim, label: value === undefined ? 'INSUFFICIENT_EVIDENCE' : value === claim.value ? 'SUPPORTED' : 'CONTRADICTED' };
  });
}
export function answerPolicy(claims, sources, scope, now) {
  const labels = verifyClaims(claims,sources,scope,now);
  return { status:labels.length > 0 && labels.every(item => item.label === 'SUPPORTED') ? 'ANSWER' : 'ABSTAIN', claims:labels };
}
export function retrievalRecall(relevantIds, retrievedIds) {
  if (!relevantIds.length) return null;
  return [...new Set(relevantIds)].filter(id => retrievedIds.includes(id)).length / new Set(relevantIds).size;
}
export function judgeAgreement(human, judge) {
  if (human.length !== judge.length) throw new Error('Unpaired annotations');
  return human.length ? human.filter((label,index) => label === judge[index]).length / human.length : null;
}

Run npm run examples:security and npm run eval:security. This source is a deterministic teaching fixture; it does not prove live model safety, NLI, production auth or OS/network isolation.

Interview questions

Does a valid citation ID prove support?

The lab uses approved structured facts such as retentionDays. Exact field/value equality labels a fixture claim; it does not handle paraphrases, inference, negation or evidence quality as a general natural-language entailment system would need to.

Why filter authorization before checking facts?

Missing keys produce INSUFFICIENT_EVIDENCE; differing known values produce CONTRADICTED. Only a non-empty set of entirely supported claims yields ANSWER. This conservative policy is inspectable but can abstain too often.

What must a natural-language verifier add?

The Java service intentionally demonstrates its citation-ID-only limitation in a test: an unsupported statement with an allowed ID can pass. The new Node verifier is a separate closed-world teaching control, not a hidden claim that Java now verifies all semantics.

Practice

Extend one negative scenario and its authorized positive counterexample for Grounding & Claim-Level Verification.

Check your solution

State the trusted input and effect boundary.
Assert the expected status and observed effect.
Report coverage and limitations without claiming production readiness.

Grounding & Claim-Level Verification