Sections in this lesson
Design a Multi-Provider LLM Gateway
Route model requests with tenant quotas, bounded retries, streaming and auditable usage.
Reviewed: · Content owner: Interview Preparator
What you will learn
- Define trust boundaries and component responsibilities.
- Estimate capacity and cost from explicit assumptions.
- Explain retry, failure and recovery behavior.
Before you start
The flow
- 1.Authenticate
- 2.Reserve quota
- 3.Route model
- 4.Stream bounded response
- 5.Reconcile usage
Architecture and boundaries
Admission API
Identity, policy, schema checks and atomic budget reservation.
Provider adapters
Capability-aware routing, deadlines, bounded retry and stream cancellation.
Usage ledger
Reservation, actual usage and unknown-outcome reconciliation with redacted traces.
Design assumptions
These values are example sizing inputs, not measurements or capacity guarantees.
| Variable | Assumption |
|---|---|
| Request rate | 40/s |
| Mean residence | 6 s |
| Input / output tokens | 1,500 / 500 |
Admission and model routing
Map trusted identity to tenant, permitted providers and data-residency policy. Normalize requests but preserve provider-specific capabilities: tool schemas, context limits and structured-output support are not interchangeable.
Reserve an upper bound for prompt plus output tokens before sending. A distributed quota ledger needs atomic updates; per-process counters do not enforce a global tenant budget. Reject or queue bounded work rather than allowing unbounded waiting.
Retries, fallback and streaming
Use a total request deadline with smaller per-attempt timeouts. Cap attempts and concurrent retries; overload should reduce admitted work, not multiply upstream calls. A timeout can leave provider usage uncertain, so retrying may incur another charge.
Allow fallback only to explicitly approved models with compatible contracts. After the first stream fragment reaches the client, terminate with an explicit partial-result status on failure; do not splice a second model response into the stream. Propagate cancellation and record any unresolved provider work.
Capacity and usage reconciliation
Assume 40 requests/second and 6 seconds mean residence time: mean in-flight work is 240 requests. At 1,500 input and 500 output tokens per request, steady demand is 60,000 input and 20,000 output tokens/second before retries. These are planning assumptions, not provider quotas.
Record request ID, selected model, policy version, attempts, time to first token and terminal status. Release unused reservations only after a known outcome; reconcile unknown usage separately. Redact prompts and restrict traces. Test provider outage, slow streams, quota races and cancellation under load.
Try it
{ "totalDeadlineMs": 12000, "maxAttempts": 2, "maxOutputTokens": 500, "fallbackBeforeFirstTokenOnly": true, "allowedProviders": [ "primary", "approved-secondary" ], "logPromptBodies": false}Illustrative design contract, not a deployable service configuration. Validate limits with load and failure tests.
Interview questions
Why not retry every timeout?
The original call may still run and consume tokens; retry budgets and outcome reconciliation limit amplification.
Can fallback change the output contract?
It must not silently do so. Validate model capabilities, privacy policy and schema compatibility before routing.
Does a local rate limiter enforce tenant-wide limits?
No. Multiple replicas require shared accounting or explicitly allocated sub-budgets.
Practice
Trace a dependency outage and a duplicate request; identify admission, durable state and recovery steps.
Check your solution
- Limits and capacity assumptions are explicit.
- Retries do not duplicate effects or results.
- Observability and recovery policy are defined.
Official references
Continue learning
Category sequence
System Design
Lesson 19 / 21
Was this lesson useful?
Change notes: Added AI platform design interview case.