LLM Inference Optimization: Speed, Memory and Serving

What inference optimisation changes

Training adjusts a model’s parameters using data and a learning objective. Inference uses a trained model to process a request and produce an output. Optimising inference usually changes how that workload is executed, although techniques such as quantisation can also change the numerical representation used for the model. [1] [2]

Serving efficiency sits at several levels. The application decides which context to send and how much output to request. The engine schedules requests and manages memory. Attention and matrix-operation kernels execute on the hardware. Improving one level does not automatically fix a bottleneck at another.

Quality belongs in the definition of success. Removing essential prompt context can reduce work while making the answer less useful. A lower-precision model can fit into less memory yet fail a task that the original version handled. “Faster” is only an improvement when the service still satisfies its intended requirements.

How prefill and decoding use the hardware differently

Before generation, a tokenizer converts the input into the model’s token sequence. During prefill, the model processes that known prompt and computes the states needed to begin output generation. Since the input is already available, many prompt positions can be processed in parallel. [1]

During autoregressive decoding, each new output token depends on the accepted preceding sequence. Small-batch decoding often has less arithmetic work available per step and can be limited by memory access or communication rather than peak compute capability. Long prompts, larger batches and different architectures can shift the bottleneck, so this is not a rule that all inference is memory-bound. [1]

Time to first token includes more than a fast attention kernel: request handling, queueing and prompt processing can all matter. After generation starts, inter-token latency describes the pace of the response. These stages need separate measurements before deciding which optimisation is likely to help.

The main techniques solve different problems

Quantisation, cache management, batching, attention kernels and speculative decoding are complementary ideas, not five interchangeable switches. Some save memory, some reduce repeated work and some increase useful concurrency. Their compatibility and combined effects depend on the model and runtime. [1] [2] [6]

Model quantisation: smaller representations, not free speed

Quantisation represents selected model values at lower precision. INT8 and INT4 describe integer representations using eight and four bits; FP8 and FP4 are different low-precision floating-point formats. A bit count alone does not identify the complete quantisation method or its numerical behaviour. [2] [6]

Weight-only quantisation can reduce the memory used to store weights, but does not automatically quantise the KV cache or every activation. Some modules stay at higher precision, and scales or other metadata consume storage. Hugging Face’s configuration documentation explicitly includes choices for weight and activation formats and modules left unconverted. [2]

Lower precision can introduce error. Evaluate the model on representative tasks and check the actual execution path: suitable low-precision kernels can help, while conversion overhead or a fallback implementation can reduce the benefit. A model that occupies less memory is not guaranteed to generate every response faster.

KV cache: avoid repeated computation, then manage its memory

A key-value cache retains attention states for tokens already processed. During decoding, the model can reuse those stored keys and values instead of recomputing them from the entire preceding sequence at every step. New token states are added as generation advances. [1]

The cache is not the model’s weight file or a library of completed answers. In a conventional dense attention design, its size depends on cached token count, layer count, KV-head count, element precision and active sequences. Grouped-query attention can have fewer KV heads than query heads, so using the query-head count indiscriminately overestimates the cache. [1]

PagedAttention manages KV data in blocks rather than requiring one large contiguous reservation per sequence. The original paper explains how this reduces fragmentation and redundant duplication and permits flexible sharing. It improves allocation efficiency; it does not create unlimited memory, erase all model states or make every context length supported. [4]

Prefix caching: reuse compatible prompt work

Across several requests, an identical prefix can provide another reuse opportunity. For example, a stable system instruction or repeated reference passage may already have compatible KV states in memory. A cache hit can avoid repeating the corresponding prefill work. It is not a guarantee that the whole response is cached. [5]

SGLang’s original RadixAttention design indexes token sequences and their KV states in a radix tree, with eviction when capacity is needed. Semantically similar text is not automatically an exact prefix match. Reuse must also be compatible with the model and other state that affects the computation. [5]

Cache retention, eviction and tenant isolation need to fit the application. Prefix reuse is an execution optimisation, not permission to mix private material between users. Where prompts have little shared content, the cache-hit rate may be low and the benefit small.

Continuous batching: keep useful work entering the engine

A static batch groups requests together even when their response lengths differ. In a simple static arrangement, short requests can leave capacity unused while longer requests continue. Continuous or in-flight batching allows a scheduler to admit new work as requests finish rather than waiting for the original group to end. [1] [6]

This can improve aggregate throughput, but the scheduler still has to balance prompt processing, decoding and memory capacity. A larger batch may use the accelerator more efficiently while increasing individual response time or queueing. High throughput and low tail latency are separate objectives.

Chunked prefill is another scheduling option: a long prompt can be processed in smaller portions so it fits better alongside other work. TensorRT-LLM lists this capability with its serving features. The outcome depends on scheduling policy and workload; chunking is not an automatic first-token latency improvement for every request. [6]

FlashAttention: fewer memory transfers for attention

FlashAttention is an IO-aware attention algorithm. Its original exact form uses tiling to reduce reads and writes between GPU high-bandwidth memory and on-chip SRAM. It reorganises how attention is computed rather than approximating the attention calculation simply to save work. [3]

That differs from lowering model-weight precision. A compatible attention kernel can reduce intermediate-memory traffic without a weight-quantisation change. However, exact mathematical attention does not promise byte-identical floating-point results across every implementation.

The technique is useful where the attention workload and supported kernel make memory traffic significant. It does not eliminate model weights, the autoregressive dependency between output steps or all long-context compute costs. Do not convert the original paper’s training benchmark gains into guaranteed inference gains for an unrelated model. [3]

Speculative decoding: propose cheaply, verify correctly

Speculative decoding proposes several candidate tokens using a cheaper mechanism and verifies them with the target model. A suitable exact sampler can preserve the target’s sampling distribution while reducing some sequential target-model steps. It does not guarantee identical sampled text or factually correct answers. [8]

The performance trade is saved target work against drafting, verification and scheduling overhead. Acceptance quality, concurrency and the method’s resource use all matter; an ineffective draft can make serving slower. Some systems offer approximate variants with different guarantees, so verify the selected sampler rather than assuming every speculative method is exact.

This is one technique within the broader serving system. It does not replace prefill optimisation, weight-memory planning or request scheduling. A long-prompt request can still spend much of its time before output decoding begins.

A memory budget must include more than the weights

The advertised parameter count supplies only one part of memory planning. Add the active KV cache, temporary activations, kernel workspaces, runtime allocations and any extra draft components. The exact allocation also depends on implementation details, sharding and quantisation format. [1] [2]

Consider a hypothetical dense model with exactly seven billion parameters, all stored at one precision. At 16 bits per parameter, its raw weights occupy 14 GB in decimal units. At eight bits, the ideal raw weight storage is 7 GB; at four bits it is 3.5 GB. These are arithmetic estimates, not actual files or measured GPU allocations.

Those three figures exclude scales, metadata, unconverted modules and all non-weight memory. A real quantised implementation may therefore use more than the ideal four-bit figure. Decimal GB and binary GiB are different units; do not silently substitute one for the other when checking a device’s capacity.

A worked KV-cache example with explicit assumptions

Now consider a separate hypothetical Transformer with 32 layers, eight KV heads per layer, head dimension 128 and two-byte KV elements. A simple dense full-attention cache stores keys and values, so its per-token size is 2 × 32 × 8 × 128 × 2 = 131,072 bytes. The first factor of two represents keys plus values.

At 8,192 cached tokens, one sequence uses 1 GiB of KV storage under these assumptions. Four independent sequences at that length use 4 GiB. These figures describe cache storage only—not a complete model, an advertised GPU requirement or measured serving capacity.

Prefix sharing, cache quantisation, sliding attention windows, compression and offloading can change this estimate. Architecture-specific cache layouts can change it too. The purpose is to show why longer contexts and more active sequences can consume meaningful memory even when the weights already fit. [1] [4]

What to know about the four named inference frameworks

The framework names describe software ecosystems with changing model and hardware support. A fair selection checks the installed release, supported model architecture, precision, serving features and operational needs. No current source establishes one engine as universally fastest for every combination.

vLLM

vLLM grew from the PagedAttention work on memory-efficient LLM serving. The original design addresses dynamic KV allocation and batched execution, while its current documented features include multiple speculative-decoding methods. It is a serving engine rather than a new language-model architecture. [4] [8]

Treat its model, parallelism and precision support as version-specific. The headline capability “supports speculative decoding” does not prove that any target/draft pair or every combination of settings is available in the installed build. [8]

NVIDIA TensorRT-LLM

NVIDIA’s current overview describes TensorRT-LLM as a PyTorch-native open-source library for inference on NVIDIA GPUs. It lists in-flight batching, paged KV management, quantisation, chunked prefill, speculative decoding and distributed inference. Describing the current stack only as a compulsory ahead-of-time TensorRT engine build would miss that documented architecture. [6]

Its precision and kernel features still depend on supported hardware and models. Use those compatibility requirements when comparing it with another engine; no specific GPU purchase or checkpoint configuration is validated by this guide.

SGLang

SGLang’s current project covers LLM and multimodal inference and large-scale serving. Its original RadixAttention work focuses on reusing compatible KV prefixes across model calls. That can be relevant to repeated prompts and multi-call applications, but it is not proof that every workload has a high cache-hit rate. [5] [9]

The current repository lists several hardware-platform paths, each with its own compatibility documentation. Support for a platform does not establish identical feature coverage, numerical behaviour or performance across all devices. [9]

Hugging Face Text Generation Inference: maintenance mode

As retrieved on 5 November 2026, the official TGI repository states that the project is in maintenance mode, accepting minor fixes, documentation improvements and lightweight maintenance work. It recommends downstream inference engines such as vLLM and SGLang going forward. This status matters when the outline is used for a new implementation decision. [7]

TGI still documents serving features including continuous batching, token streaming, quantisation and tracing. Maintenance mode does not mean an existing installation has stopped working, nor is it a reason to replace a healthy deployment without compatibility and migration testing. [7]

How hardware requirements change with the workload

A useful hardware plan specifies the model and precision, expected prompt/output lengths, active concurrency and latency target. Memory capacity determines what states can fit; memory bandwidth, compute and supported kernels help determine how quickly they can be processed. Parameter count alone cannot answer both questions. [1] [6]

Multi-GPU strategies can split model operations or layers across devices and enable larger working sets, but communication and synchronisation add costs. Twice as many devices do not automatically halve response time. Independent serving replicas address a different objective from splitting one model across devices.

The photo in this guide shows ordinary desktop RAM, not GPU HBM. Host memory and device memory are not interchangeable performance resources. Offloading can reduce device-memory pressure, but transfers and the slower memory path may affect latency; measure the actual implementation rather than assuming a free capacity extension.

The application decides which optimisation is useful

An interactive chat service may care most about first-token wait and a steady output pace. A document-processing queue may prefer aggregate throughput within an acceptable completion window. A repeated-reference workflow may benefit from prefix reuse. These are workload examples, not measured customer results.

For AI search or coding assistance, reducing redundant context can help, but essential evidence and instructions must remain available. Multilingual input also needs measurement with the model’s actual tokenizer: equal character counts do not necessarily give equal token counts or processing cost.

For a service used in India, network delay and application work can remain noticeable even after the GPU becomes faster. Measure the end-to-end experience, not just a token-generation trace inside the serving engine. No provider’s local rollout or performance is asserted here.

Measure latency, throughput, memory and quality together

Time to first token captures the initial wait. Inter-token latency captures output pacing. End-to-end request time includes the whole response and its surrounding work. Aggregate throughput measures completed work across concurrent requests. These metrics answer different questions.

Keep a comparable baseline: same target model and version, equivalent prompts and generation limits, precision recorded, warmup handled and representative request rates. Track memory pressure and failures as well as average performance. Longer responses can make a raw tokens-per-second result look different without improving the task being completed.

Cost should be assessed per useful completed workload under the service requirements, not inferred from peak tokens per second. Extra devices, draft work, idle capacity or a quality regression can offset an apparent kernel improvement. This article contains no paid-service quote, hands-on comparison or guaranteed cost reduction.

Where ongoing inference work is heading

The current software work is broader than one faster decoding trick. TensorRT-LLM’s overview includes chunked prefill, distributed execution and beta disaggregated serving, which separates prompt and generation work across hardware. These are documented directions with implementation trade-offs, not a promise that every service should adopt them. [6]

Other decisions remain workload-specific: better memory reuse, suitable low-precision formats and compatible proposal methods. On-device execution introduces its own capacity, latency and energy limits. A sensible future-facing question is which bottleneck the next change removes, and which new constraint it creates.

Common questions about LLM inference optimisation

Which inference framework is fastest?

There is no universal answer supported by these sources. Performance depends on model architecture, hardware, precision, context lengths, request rate and the metric being measured. Choose supported candidates and compare the actual workload, not unrelated headline benchmarks.

Does quantisation affect LLM accuracy?

It can. Lower-precision representations introduce a possible numerical trade-off, while algorithms and higher-precision exceptions try to manage the error. Evaluate the model’s intended tasks rather than assuming fewer bits always preserve the same behaviour. [2]

If the weights fit, can I use any context length?

No. Active KV states and other runtime memory still need space, and the model itself has supported context and attention limits. Efficient allocation does not remove those requirements. [1] [4]

Is higher throughput the same as lower latency?

No. Throughput describes aggregate work served over time; latency describes a request’s wait or completion time. Batching can improve the first without automatically improving the second. Measure both at the intended traffic level.