Design a Multi-Tenant RAG Platform

The flow

1. 1.Upload & authorize
2. 2.Parse and chunk
3. 3.Version embeddings
4. 4.Retrieve under ACL
5. 5.Generate with sources
6. 6.Evaluate & invalidate

Architecture and boundaries

Design assumptions

These values are example sizing inputs, not measurements or capacity guarantees.

Clarify the product contract

Assume employees ask questions over their organization's documents. Each answer must either cite authorized, current evidence or explicitly abstain. Define supported file types, maximum document size, update freshness, deletion requirements and latency/error budgets before selecting a vector database.

Keep source objects authoritative. Chunk IDs include source/version lineage; embedding model and index versions are tracked separately. A successful vector query over a stale policy can still produce the wrong business answer.

Size the expensive paths

At 100,000 documents and ten chunks each, the corpus has one million vectors. With 768 float32 dimensions, raw vectors occupy 3.072 decimal GB. This excludes text, metadata, indexes, replicas and backups; it is not a database disk estimate.

At 20 queries/second with 1,800 input and 600 output tokens per request, demand is 36,000 input and 12,000 output tokens/second. This is workload demand, not guaranteed model throughput. Measure the provider/hardware and apply admission control, queue bounds and per-tenant quotas.

Enforce access before context assembly

Derive tenant and ACL filters from authenticated identity before ranking and generation. Cache keys must account for the permission scope and corpus/index version; a shared semantic cache without those boundaries can leak evidence. Retrieved text remains untrusted even after access checks.

On updates, index a new source version and switch the active view without mixing embedding spaces. On deletion, remove active chunks and invalidate dependent caches, then handle source/backups under the retention policy. An immediate active-index deletion is not proof that every retained copy vanished.

Verify the boundaries independently

Measure retrieval recall against expected sources before grading answer grounding. Track empty retrieval, stale results, foreign-tenant denial, citation support and cost/latency slices. The Phase 2 lab implements real pgvector filtering and deletion over a small fixture corpus; a production ingestion queue, migration system and fleet are design extensions here.

Try it

// Sizing assumptions, not measurements or provider capacity guarantees. Decimal GB.
function positive(value) {
  if (!Number.isFinite(value) || value <= 0) throw new Error('Positive finite input required');
  return value;
}
export function ragCapacity({ documents, chunksPerDocument, dimensions, qps, inputTokens, outputTokens }) {
  [documents, chunksPerDocument, dimensions, qps, inputTokens, outputTokens].forEach(positive);
  const chunks = documents * chunksPerDocument;
  const rawVectorGB = chunks * dimensions * 4 / 1_000_000_000;
  return { chunks, rawVectorGB, inputTokensPerSecond: qps * inputTokens, outputTokensPerSecond: qps * outputTokens };
}
export function agentCapacity({ jobsPerSecond, meanSeconds, slotsPerWorker, toolCallsPerJob, headroom }) {
  [jobsPerSecond, meanSeconds, slotsPerWorker, toolCallsPerJob, headroom].forEach(positive);
  if (headroom < 1 || !Number.isInteger(slotsPerWorker)) throw new Error('Invalid worker configuration');
  const meanInFlight = jobsPerSecond * meanSeconds;
  return { meanInFlight, workers: Math.ceil(meanInFlight * headroom / slotsPerWorker), toolCallsPerSecond: jobsPerSecond * toolCallsPerJob };
}

examples/agentic-ai/capacity-planning.mjs · npm run examples:agents. Java examples target 21. Models and usage are fixtures; no live inference or billing guarantee is implied.

Interview questions

Official references

• Pgvector — official project
• PostgreSQL — row security

Continue learning

• Design an Agent Execution Platform →
• RAG with PostgreSQL & pgvector →
• Testing & Evaluation for Spring AI →