LLM Inference at Scale: Batching, KV Caches, and Serving

Serving a language model is not ordinary request-response hosting with a more expensive handler. Requests have different prompt lengths, produce unknown numbers of output tokens, and remain active while tokens are generated. Their memory grows with retained context. A long prompt can delay many interactive users, while an unconstrained generation can occupy a scarce batch slot long after its useful answer should have ended.

The serving problem is therefore a scheduling and capacity problem. Model quality may determine whether users want the output, but the inference system determines when the first token arrives, how smoothly later tokens appear, how many requests complete within service objectives, and how much accelerator memory sits useful rather than stranded.

The central design rule is to manage prefill work, decode work, and KV-cache memory as explicit resources. Continuous batching, quantization, routing, and autoscaling are tools for allocating those resources. They should be judged by goodput under a production-shaped workload, not by a peak tokens-per-second number obtained from one convenient sequence length.

Separate Prefill from Decode in the Service Model #

An inference request has two operationally distinct phases. During prefill, the server processes the input tokens and creates the request’s initial cached state. Prefill can expose substantial parallel work, but its cost rises with prompt length. During decode, the server generates tokens incrementally while reusing that cached state. Each active sequence contributes a small unit of work per decoding step and remains resident until it finishes, is cancelled, or reaches a limit.

This distinction creates separate user-facing metrics:

• Time to first token (TTFT): queue time plus prefill and first-token work.
• Inter-token latency (ITL): delay between streamed output tokens.
• Time per output token (TPOT): aggregate decode time divided by generated tokens, with a precisely documented convention.
• End-to-end latency: admission through final token or terminal error.
• Goodput: requests or tokens completed while satisfying the defined latency and correctness constraints.

Throughput without service constraints can be misleading. Increasing a batch may produce more tokens per second while making interactive TTFT unacceptable. A server can also report high token throughput by completing many short requests while a few long ones wait indefinitely. Publish distributions by prompt-length bucket, output-length bucket, priority class, model route, and finish reason.

Workload demand should likewise be split. If request class $i$ arrives at rate $\lambda_i$, with expected input length $I_i$ and output length $O_i$, first-order token demands are

$P$ and $D$ are not interchangeable capacity units: prefill and decode exercise hardware differently. They are useful for describing the mix and selecting load fixtures, not for predicting one universal token rate. Measure both phases on the actual model, quantization, accelerator, runtime, and sequence-length distribution.

Budget Weights, KV Cache, and Temporary Memory #

Accelerator memory holds more than model weights. The serving runtime also needs KV-cache blocks for active sequences, temporary operator workspaces, communication buffers, graph captures, allocator metadata, and runtime state. A deployment that fits weights with only a narrow margin may load successfully yet admit almost no useful concurrency.

For a conventional cache layout, an approximate per-token KV footprint is

where $L$ is the number of layers, the factor 2 represents key and value storage, $H_{kv}$ is the number of KV heads, $d_h$ is head dimension, and $b$ is bytes per stored element. Architectures and runtimes vary, so verify the actual allocation rather than treating the formula as an invoice.

Total cache use follows the sum of cached tokens across active requests, not merely the configured maximum context:

This is why a handful of long conversations can evict or block many short ones. Cache-block allocators reduce fragmentation by assigning fixed-size pages and can reclaim completed sequences without moving every neighbor. Prefix caching can let requests sharing an identical stable prefix reuse prefill state, but cache keys must include every input that affects computation: model and adapter version, tokenized prefix, relevant position configuration, and any runtime-specific parameters. Never share a prefix merely because two prompts look similar as text.

Set explicit maximum input, output, and total-token limits per request class. Reserve headroom for temporary memory and runtime variance. Admission control should estimate whether a new sequence can fit through its allowed lifetime, then queue or reject it before an out-of-memory failure destabilizes the replica. Cancellation must release scheduled work and cache blocks promptly; disconnecting the HTTP stream without cancelling generation wastes capacity invisibly.

Use Continuous Batching with an Explicit Scheduler Policy #

Static batching waits for a group of requests, pads or groups them, runs them together, and releases the batch when all finish. It works well for homogeneous offline jobs, but generation lengths are variable. Short sequences leave unused slots while the server waits for the longest member.

Continuous batching rebuilds the active batch at scheduling boundaries. Finished sequences leave, newly prefetched sequences enter, and cache blocks remain associated with each request. This improves utilization under mixed arrivals, but it introduces policy decisions: how much prefill to admit, which decode sequences run next, whether priorities can preempt, and how to prevent starvation.

Large prefills can monopolize compute long enough to disrupt decode smoothness. Chunked prefill splits a long prompt into bounded pieces so decode iterations can run between them. Smaller chunks improve responsiveness but add scheduling overhead and may reduce peak prefill efficiency. The correct size follows the TTFT and ITL objectives of the actual workload.

A scheduler should make these controls visible:

Priority should represent product policy, not whichever client claims the highest integer. Authenticate class assignment and cap each tenant’s queued and active work. Weighted fair scheduling or separate pools may be easier to reason about than one global priority queue. Strict physical isolation is appropriate when a batch workload’s context lengths or adapters make its interference unacceptable.

Backpressure starts at admission. Bound queues by count and estimated token work, return a retryable overload response before the caller’s deadline expires, and expose retry guidance. Allowing an unbounded gateway queue converts a capacity shortage into high latency, memory growth, and a retry storm.

Work an Illustrative Capacity and Routing Plan #

Consider a serving replica for an illustrative model with 48 layers, 8 KV heads, head dimension 128, and a two-byte KV representation. The approximation above gives

or 192 KiB per cached token before allocator and runtime overhead. If profiling shows that 32 GiB can safely be assigned to KV blocks after weights and required headroom, the arithmetic upper bound is about 174,000 cached tokens. That is not an admission target: block rounding, temporary memory, output growth, and workload variance require a safety margin established by stress tests.

The product has two request classes:

Suppose forty active interactive requests each retain 4,000 tokens near the end of generation. They alone account for 160,000 cached tokens, already close to the arithmetic bound. Admitting them because their initial prompts were only 3,000 tokens would ignore expected output growth. A safer controller reserves each request’s allowed growth, uses observed length distributions for planning, and stops admission before the physical limit.

Routing the nightly 12,000-token prompts into the same pool would create both memory pressure and prefill interference. The plan assigns them a separate deployment configured for throughput, while the interactive deployment keeps headroom for bursts. If demand is too small to justify permanent physical pools, the router can use bounded class queues and scheduled capacity windows, but the service objective must state the resulting contention.

The capacity experiment sweeps interactive arrival rate and document concurrency across their expected distributions. At each point it records TTFT, ITL, completed requests, queue expiry, cached tokens, cache-block occupancy, prefill/decode token rates, and allocation failures. The safe operating point is the highest offered load that preserves all objectives with headroom, not the point immediately before a crash.

Apply Quantization and Parallelism for a Measured Constraint #

Quantization stores weights, and sometimes activations or caches, at lower precision. Weight quantization can reduce memory traffic and allow a model to fit on fewer devices. KV-cache quantization can increase concurrent token capacity. Speedup depends on hardware kernels, shapes, conversion overhead, and runtime support; a smaller representation is not automatically a faster request.

Quality must be evaluated by task and slice. Quantization error can affect rare tokens, structured output, long-context use, numeric reasoning, or particular languages differently. Compare the quantized candidate against the serving baseline with production-shaped prompts, repeated generations where stochasticity matters, and the same output policy. Include TTFT, ITL, energy or cost where available, and maximum stable concurrency. Reject a quantization that improves peak throughput but violates the product’s quality floor.

Parallelism addresses different constraints. Tensor parallelism divides work for one model operation across devices, requiring fast communication. Pipeline parallelism places groups of layers on different stages and can introduce bubbles. Data parallel replicas avoid cross-replica communication during inference and scale independent requests, but each replica needs its own weights. Select the smallest topology that fits the model and service objective; spreading a model across slow links can make it fit while making it unusable.

Replicated serving also changes batching. One large replica may form efficient batches but has a larger failure domain. Several smaller replicas offer isolation and flexible scaling but split arrivals into thinner batches. Use request routing that considers queue token work and cache availability, not round robin alone. Session affinity can improve prefix-cache reuse, yet hard affinity creates hotspots; use it as a preference with a load-aware escape path.

Route and Autoscale on Leading Capacity Signals #

The front-door router should validate model and adapter access, enforce token and concurrency quotas, classify workload, and choose a compatible deployment. Routing keys may include model version, quantization, context limit, tenant isolation, adapter, latency class, region, and data residency. Reject incompatible requests explicitly rather than silently truncating context or switching to a model with different behavior.

Autoscaling on accelerator utilization alone is reactive and ambiguous. High utilization can mean healthy batching; low utilization can coexist with a queue blocked on memory admission. Better leading signals include queued prefill tokens, queued requests by deadline, active cached tokens, available cache blocks, and arrival rate by class. Scale decisions should also account for model-load time and the minimum warm capacity needed during that delay.

A useful control loop forecasts demand over the model-loading horizon, starts replicas early, and drains them deliberately. New replicas become ready only after weights load, runtime graphs initialize, and a representative warm-up succeeds. Draining stops new admission but allows active streams to finish within a deadline; forced termination reports a clear finish reason and releases resources.

Scale-in can destroy prefix-cache value and strand sessions, so include cache warmth and active sequence age in victim selection. Keep hysteresis and cooldown around noisy signals, but do not use cooldown to ignore a fast overload. Admission control remains necessary even with autoscaling because no scaler can create an accelerator before an immediate burst arrives.

Plan failure isolation. A health check that generates one short token proves little about long-context allocation. Readiness should include model identity, memory pool initialization, and scheduler acceptance. Stop routing when allocation errors, corrupted output, or deadline misses cross a threshold, while preserving enough telemetry to diagnose the replica.

Observe the Scheduler, Not Only the API #

Request traces should separate gateway queue, serving queue, prefill, decode, streaming, and cancellation. Record input and output token counts, batch participation, cache blocks, finish reason, model configuration, adapter, and tenant class without logging sensitive prompt text by default.

At the replica level, measure:

• TTFT, ITL, and end-to-end distributions by length and class.
• Prefill and decode tokens per second, reported separately.
• Active sequences, batched tokens per iteration, and scheduler delay.
• KV-cache occupancy, allocation failures, fragmentation indicators, and prefix-cache hit rate.
• Queue depth in estimated tokens, deadline expiry, rejection, and cancellation latency.
• Accelerator memory, compute utilization, communication time, power, and error state.
• Goodput and cost per successful request under the product’s objectives.

Common failure modes become recognizable. Rising TTFT with stable ITL suggests admission or prefill pressure. ITL spikes when long prompts arrive point to prefill interference. High cache occupancy with moderate compute indicates context-heavy requests or leaked cancellation. High utilization plus falling goodput means the scheduler is doing work that no longer meets deadlines. One tenant dominating cached tokens calls for quotas or isolation, not another global replica alone.

Never infer quality from serving health. A fast quantized deployment can produce unusable answers, while a quality regression can leave every infrastructure graph green. Join serving configuration with offline evaluation version and sampled online task outcomes. Operational and semantic gates are separate and both are required.

Verify with Load Shapes, Faults, and Controlled Rollout #

Benchmark the complete serving path using a matrix of input lengths, output limits, arrival patterns, cancellation rates, adapters, and priority classes. Include steady load, bursts, long-tail prompts, and mixed interactive/offline traffic. An open-loop load generator is useful for preserving intended arrivals when the service slows; otherwise coordinated omission can make overload look healthier than it is.

For each experiment, freeze model artifacts, runtime version, quantization, hardware, scheduler settings, topology, and prompt/output distributions. Warm replicas through the same readiness path used in production. Report raw distributions and sample counts, not only averages or one maximum throughput figure. Verify generated outputs against the task evaluation suite when changing precision, runtime kernels, or decoding behavior.

Inject failures: cancel clients during prefill and decode, kill a worker after cache allocation, exhaust the admission budget, make one parallel device unavailable, delay model loading, and duplicate gateway retries. Assert that memory is reclaimed, idempotent request policy is honored, streams end clearly, unhealthy replicas leave rotation, and queues remain bounded.

Roll out with shadow load where cost permits, then canary by a representative traffic slice. Compare goodput, TTFT, ITL, errors, output quality, and cost against the current deployment. Keep configuration-level rollback for scheduler policy, quantization, runtime, and model route. A benchmark win is not complete until the canary survives real length distributions and cancellation behavior.

Efficient inference is not one batching switch. It is a resource contract joining request limits, cache memory, phase-aware scheduling, workload routing, and measured service objectives. Separate prefill from decode, reserve memory for sequence growth, reject work before overload, scale from leading token and cache signals, and test quality whenever representation or runtime changes. That is how peak accelerator throughput becomes dependable user-facing capacity.