Design an LLM inference platform

01Define the generation service and its limits

Build a service that accepts an authenticated text prompt, generates output from a selected model version and streams that output to the caller. Include tenant quotas, cancellation and model rollouts. Exclude model training and execution of generated tool calls. A proposed tool call is output data; another application decides whether it may run.

Ask whether output must stream, which prompt/output sizes matter, and whether a stream must continue seamlessly after worker failure. Here output streams, the worked request has 4,000 input tokens and at most 600 output tokens, and worker loss may interrupt it.

Follow generation G81: a 4,000-token prompt and an upper limit of 600 output tokens. Tokens are units from the model's tokenizer, often word fragments. The service must budget this request differently from a twenty-token question, even though both count as one HTTP request.

Choose an explicit recovery contract. Request identity and status are durable; live model execution state is not. Worker failure interrupts active generations. A client can start a new generation, but the service does not silently append a different regenerated answer to the old partial stream.

02Functional requirements

Agree on what the service must do before choosing its components.

1. Generate versioned output. Accept an authenticated prompt and selected immutable model version, then stream numbered output events with a visible terminal status.
2. Inspect and retry a request. Return a generation identity and status. Repeating the same tenant/request identity and effective settings retrieves that logical generation; conflicting settings fail.
3. Cancel generation. Accept an idempotent cancellation request and show whether execution has stopped, completed or been interrupted. Recording a cancel request does not prove the worker has already stopped.
4. Report usage and roll out models. Expose recorded token usage, apply tenant quotas and deploy tested model versions while draining old requests. Training and execution of generated tool calls are outside this service.

03Non-functional requirements

Use these hypothetical requirements for the worked interview. Confirm the assumptions with the interviewer; the numerical targets require testing and are not measured results. For latency, p95 and p99 mean that 95% and 99% of measured delays, respectively, are no greater than the reported value.

1. Workload. Size the illustrative load at 100 requests/s with 4,000 input and 600 output tokens per request: 400,000 input and 60,000 output tokens/s. Validate the actual mix of shorter/longer requests separately rather than assuming every request costs the same.
2. Interactive latency. Target p95 time to first token below one second and p95 gaps between subsequent tokens below 50 ms for the tested input/output distribution. Queue waiting consumes the first-token budget; throughput alone does not validate either target.
3. Durable control, interruptible execution. Request identity, status and recorded usage survive the supported single control-database-node failure through durable majority commits. Live worker state is temporary: worker loss interrupts active generations, and lost database majority stops new admissions. Do not claim transparent continuation or instantaneous cancellation through a partition.
4. Bounded, fair work. Limit tenant token work, concurrency, queue time, output length and absolute deadlines. Admit only work fitting the worker’s actual memory budget; slow clients receive bounded buffers and cannot consume capacity forever.
5. Private, compatible execution. Authorize generation status/stream ownership and scope private input reuse to trusted tenants. Pin compatible weights, tokenizer, prompt template and runtime; protect artifacts and define private-input retention. Generated output is data, not permission to execute server code.
6. Consistent accounting. Persist cumulative usage monotonically and finalize a terminal total. This internal service charges only durably recorded work, so worker loss can leave an unbilled tail; repeated or late reports cannot silently create a second charge.

04Serve one bounded request before scaling

Start with an API gateway, a replicated control database and one inference worker. The worker loads an immutable model version that fits its accelerator memory. Durable artifact storage holds the model files and configuration needed to reload it. An accelerator such as a graphics processing unit, or GPU, performs the model calculations.

The gateway authenticates the caller, applies the model's exact prompt template, tokenizes the resulting input and validates context and output limits. It records G81 under a unique tenant/request ID. A short bounded queue feeds the worker; the first implementation handles one generation at a time. Excess work receives a retryable overload response rather than waiting indefinitely.

The worker first performs prefill: it processes the prompt and builds cached attention keys and values. During decode, it repeatedly computes the next output token using the existing prefix and newly generated tokens. The key/value, or KV, cache avoids rebuilding the whole prefix's attention state for every next token. The worker sends numbered output events through the gateway until an end condition, output limit, deadline or cancellation stops generation.

The gateway returns final status and recorded usage. If G81's worker crashes, the status becomes interrupted and its partial output remains visibly incomplete.

05Calculate token work, active streams and memory

Assume 100 requests per second with G81's 4,000 input and 600 output tokens: 400,000 input tokens and 60,000 output tokens per second. If an illustrative worker separately sustains 15,000 input tokens per second or 1,500 output tokens per second, the isolated lower bounds are 27 prefill workers and 40 decode workers. Forty mixed workers are not thereby sufficient; both phases compete for resources. Benchmark the combined workload with headroom.

At fifty output tokens per second per stream, 600 output tokens take about twelve seconds. Stable traffic therefore has roughly 100 × 12 = 1,200 active decoders, before accounting for queued or prefilling requests. Longer outputs occupy memory and execution capacity longer even at unchanged request rate.

KV-memory calculation for the sample architecture

Bytes per token = 2 × 32 × 8 × 128 × 2
                = 131,072 bytes
                = 128 KiB

The first factor counts keys and values. Use the stored KV-head count, which may differ from the query-head count. A fully grown 4,600-token sequence needs about 575 MiB, excluding model weights and other execution memory.

A 20 GiB pool reserved for KV can hold only 35 such maximum-length sequences before additional overhead. Define latency targets for a tested input/output distribution, not arbitrary prompt lengths.

06Separate durable identity from temporary execution

Start a generation

POST /generations

A repeated tenant/request ID with the same effective settings returns the existing generation; different content conflicts.

Inspect G81

GET /generations/G81

Return generation status after checking ownership. All stream access checks ownership too.

Request cancellation

DELETE /generations/G81

Record an idempotent cancellation request. Invalid limits, tenant quota exhaustion and unavailable worker capacity are distinct errors.

Durable control record

Use three database replicas across failure domains and acknowledge updates after a durable majority commit. Loss of a majority stops new admissions and authoritative state changes. Model input requires an explicit private-retention policy; routine logs contain identifiers and lengths rather than raw prompts.

Temporary worker state and stream identity

Expired replay buffers or worker loss mean interruption.

Persist assignment to a specific worker process incarnation: an identifier created on each process start. That incarnation deduplicates repeated dispatch; a restarted process rejects assignments naming the old incarnation. Accept output and status callbacks only from the assigned incarnation while the generation is nonterminal. Never transparently reassign a running generation; a new attempt has a new identity.

07Use admission and batching to protect latency

Time to first token, or TTFT, measures arrival to the first output token. Inter-token latency, or ITL, measures gaps between later output tokens. Suppose the interactive targets are one-second p95 TTFT and 50-millisecond p95 ITL for the tested traffic mix. Queue delay consumes TTFT; a long prefill can also interrupt output for already active requests.

First, bound tenant input/output tokens, concurrency and queue waiting time. The router can use worker capacity reports, but the worker makes the final guarded reservation against its actual memory budget. Two gateways seeing one free slot must not both consume it. Initially reserve enough KV capacity for each admitted request's maximum allowed length; this is conservative but straightforward.

Continuous batching lets the scheduler insert new sequences and remove finished ones between execution iterations. A short request need not wait for the longest member of an original fixed batch to finish. However, simply enlarging the batch can improve aggregate throughput while worsening each stream's latency.

Chunked prefill divides a long prompt's prefill computation across bounded scheduling intervals. This gives active decoders opportunities to produce output between those intervals. It does not split the user's prompt into independent questions or remove attention to the prior context. Tune the token budget against both TTFT and ITL; prioritizing ongoing decode too strongly can make new requests wait.

Use a mature serving runtime for these mechanisms. Explain the bottleneck each change addresses and measure the result. Keep long-running batch jobs in a separate queue with a different latency budget.

08Reuse computation without losing memory ownership

A paged KV allocator stores attention state in fixed-size blocks and records which blocks hold each sequence’s tokens. A growing sequence can acquire another block instead of needing one large continuous free region. This reduces wasted space, but each distinct live token still needs storage. Initially reserve for the maximum sequence length; relax that policy only after measuring memory pressure and deciding how to pause or restart work when memory runs short.

Prefix caching reuses prefill state for identical leading tokens, such as repeated instructions and the same source document. It saves input computation, not the work of generating G81's new 600-token answer. Cache identity must include compatible model weights, tokenizer, template, adapter settings and exact effective tokens. A server-controlled tenant scope prevents private prefix reuse across unrelated tenants.

Retaining a 2,000-token prefix at 128 KiB per token occupies 250 MiB. Decide whether repeated prefill savings justify that space instead of another active sequence. Eviction stops the cache from keeping a block, but running computations may still need it. Wait for them before reusing its memory. A shared prefix stays read-only: to append tokens, copy a partial block into private storage or share only completed blocks.

Cancellation follows memory ownership

1. Stop future scheduling. Mark G81 unschedulable.
2. Drain active use. Let any already running kernel finish using its blocks.
3. Release memory. Reclaim the blocks only after that use has ended.

Freeing immediately when the HTTP connection closes can let a new request overwrite memory still being read by the old computation.

09Add ready replicas before specialized execution

When one worker reaches its tested capacity, add compatible worker replicas and route independent generations among them. Readiness means the exact model and tokenizer are loaded and a representative request succeeds, not merely that the process has started. Each worker owns its scheduler and memory pool. Keep spare ready capacity for the stated worker-loss target; a cold artifact download is not immediate failover capacity.

For each model version, route using queue delay and token/memory pressure rather than open connection count alone. A worker can reject a reservation when its local capacity has changed. An explicit rejection before execution allows reassignment. A lost admission response is uncertain, so report interruption rather than risk a second execution. Running generations are never transparently reassigned.

If the model itself does not fit one device, tensor parallelism divides layer computations across devices and introduces communication between them. A coordinated group then acts as one worker, and losing a required device can interrupt that group's requests. If the model already fits, replication is generally the simpler way to increase independent-request throughput.

Quantization reduces numerical storage precision and may save weight or KV memory, but needs compatible kernels and quality evaluation. Separating prefill and decode onto different pools is a later alternative: transferring G81's 4,000-token prefix at the example KV size moves about 500 MiB per request.

10Make interruption, backpressure and accounting explicit

If a worker incarnation disappears, mark all its nonterminal generations interrupted and close their streams. A restarted worker registers a new incarnation and cannot reclaim those assignments. Do not copy only a saved prompt to another worker and claim the old continuation survived. New requests can use ready replicas; a replacement attempt is visibly separate. A durable control row preserves identity, not lost GPU tensors.

A slow client gets a bounded event buffer. If it cannot catch up within the declared grace period, cancel its generation rather than accumulating unlimited output or continuing expensive work forever. A cancel response initially means the request was recorded; the terminal status becomes canceled after the worker acknowledges stopping future scheduling. Worker loss may instead produce interrupted status. Every generation also has an absolute deadline, limiting abandoned execution when control communication fails.

For usage, periodically save cumulative input/output counts and advance the durable total monotonically. Reports of 100 and then 120 tokens mean 120, not 220. On ordinary completion, persist final usage before sending the final usage event. This internal service charges only durably recorded work: a crash at token 120 with only 100 saved leaves an unbilled twenty-token tail. Finalize the interrupted total so a late report cannot silently add a charge; record late physical work separately for capacity analysis.

During a control-database outage, stop new admissions. Existing generations can run within their already granted bounds and deadlines, but status, cancellation and final accounting may be delayed. Report that limitation instead of claiming instantaneous cancellation through a partition.

11Measure useful throughput and roll out complete versions

Measure queue time, TTFT, ITL, completion latency, input/output tokens per second, active sequences, KV occupancy, prefix reuse and cancellation delay. Break results down by prompt length, output length, model version and tenant. High GPU utilization alone can hide unacceptable token gaps or repeated wasted work.

Pin the weights, tokenizer, template, adapters and compatible runtime as one tested deployment configuration. Load a new version on separate workers, run quality and mixed-load tests, route a small canary, then expand traffic. Drain old workers until their active requests finish or reach the documented deadline. Replacing weights beneath a live KV cache invalidates the state used for continuation.

Exercise worker crashes after partial output, cancellation during a kernel, simultaneous final-slot admissions, expired reconnect buffers and repeated usage reports. A basic fairness test places a short request behind a burst of long prompts and measures both its first-token delay and existing streams' token gaps.

Protect artifact integrity and private prompts. Cache tenancy comes from trusted identity, not a client-supplied value. Generated output must not execute as server code. Compare optimizations using successful workload throughput within latency and quality targets, including memory and warm-spare costs; a larger benchmark token count is not automatically a better user experience.

12Check the design against its requirements

Before closing, check the final design against the agreed requirements. FR means functional requirement and NFR means non-functional requirement; the numbers refer to the lists above. These are proposed validation checks, not test results.

13Rapid revision

Rehearse the numbered functional requirements and non-functional targets first. Use this table to recall the mechanisms, then close with the requirements check above.

Remember: A capacity report is a hint; the worker reserves memory.

14Close with a measured capacity and failure contract

Trace G81 from token counting through admission, prefill, decode, numbered output and cleanup. Explain the pinned model version, worker-owned reservation and safe cancellation.

Next, mix short prompts with G81-sized requests while injecting worker failure and cancellation. Measure useful token throughput within both latency targets. Exact continuation after worker loss remains an Advanced requirement.

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.