Aura Digital
Sofia · 42.69°N 23.32°E
Get a quote
All articles
Guides · November 16, 2026 · 6 min read

SGLang vs vLLM: where each fits in production inference

SGLang and vLLM compared on prefix caching, routing, prefill/decode split, hardware and published benchmarks, using the November 2026 releases of both.

SGLang and vLLM are open-source inference servers for running large models on GPU fleets, and both ship a new release about every two weeks. This comparison uses vLLM 0.33.0 and SGLang 0.5.23, both released on November 2, 2026.

Which one for whom

The trade-off: no neutral head-to-head result could be found, and both engines change every few weeks. Your own prompts and prefix mix, run on both, will settle more than any table.

SGLang vs vLLM at a glance

vLLM 0.33.0 SGLang 0.5.23
Prefix cache Hash chain over full KV blocks, LRU Radix tree over token sequences, LRU
Cache beyond GPU memory KV connectors, including CPU offload and LMCache HiCache: host memory, then storage
Cache-aware routing vLLM Router, a separate project Model Gateway, cache_aware by default
Prefill/decode transfer NIXL, Mooncake, MoRI-IO (ROCm), LMCache and others Mooncake by default, NIXL, MoRI and others
Grammar backends xgrammar, guidance, chosen by auto XGrammar by default, Outlines, Llguidance
CUDA builds 13.0 only 13 only
Image and video Separate project Built in

The last rows are close to parity. The first four are where the two projects made different design choices.

Prefix caching: a radix tree in SGLang, hashed blocks in vLLM

SGLang's RadixAttention keeps KV tensors in a radix tree that maps token sequences to their cached values. It finds the longest matching prefix, splits a node when two requests diverge, and evicts least recently used leaves. It is on unless you pass --disable-radix-cache. The scheduling policy defaults to fcfs, with lpm among the alternatives.

vLLM hashes each KV block from three things: the parent block's hash, the block's tokens, and extras such as LoRA IDs, multimodal hashes and cache salts. Only full blocks are cached, so the tail of a shared prefix that doesn't fill a block gets recomputed. A free queue gives LRU eviction. A per-request cache_salt limits reuse to callers who share that salt, which the docs describe as protection against timing attacks.

A hit-rate comparison between the two structures did not turn up in either project's docs.

Routing: vLLM's Router began as a fork of SGLang's gateway

The SGLang Model Gateway is written in Rust, and its default policy combines cache locality with load. It also handles prefill/decode pools, several models through one endpoint (--enable-igw), retries, circuit breakers, Kubernetes discovery and Prometheus metrics.

The SGLang team's December 2024 test shows the size of the effect on its own engine: 8x A100 80 GB, Llama 3.1 8B, generated shared-prefix traffic. Moving from v0.3 to the v0.4 balancer raised throughput from 82,665 to 158,596 tokens per second, and cache hit rate from 20% to 75%. That workload was built to reward prefix reuse.

vLLM Router appeared in December 2025, also in Rust, and its announcement says it derives from a fork of the SGLang gateway. Its repository lists cache-aware, consistent-hash, power-of-two, random and round-robin policies.

Routing for vLLM also comes from outside the project. Kinference, announced by Buoyant on November 4, 2026, balances on queue sizes, past performance and KV cache contents. Buoyant reports a 95% lower median time to first token on one workload compared with standard Kubernetes load balancing. That figure is Buoyant's.

Prefill/decode split: both run it, only vLLM's docs call it experimental

SGLang's docs give two reasons for separating the phases: prefill batches interrupt decode batches, and data-parallel workers fall out of step when one prefills while another decodes. Mooncake can run over RDMA or TCP.

vLLM's page gives similar reasons: tuning time to first token and inter-token latency separately, and controlling tail latency. It also states that the feature does not improve throughput. The experimental label doesn't mean it stays small: the vLLM team's own results below use it on a full rack.

Published benchmarks: each team reports its own

  • SGLang team, October 24, 2026. DeepSeek-V4.1-Flash against DeepSeek's DeepInfer engine: up to 1.04x prefill throughput on 4x GB300 and on 4x B300, and decode steps 1.1% to 7.4% shorter from 16 to 256 requests per GPU on 64x GB300. The post lists method and configuration differences, and DeepInfer's decode figures are read from charts.
  • vLLM team and Inferact, October 23, 2026. The same model on one GB300 NVL72 rack, with prefill/decode split and Wide-EP: 954K tokens per second per GPU at a P90 of 196 tokens per second per user, on SemiAnalysis AgentX. That throughput counts cached input, uncached input and output tokens.
  • SemiAnalysis, June 9, 2026. It benchmarks both engines but declined to plot them on one graph for the same hardware, citing the feud between the two communities. It said both worked out of the box on CUDA for DeepSeek V4 Pro, and that native vLLM on ROCm was progressing more slowly than native SGLang.
  • AMD and Embedded LLM, October 16, 2026. Eight MI355X GPUs running Kimi K3 on vLLM: 24.4% more tokens per second per GPU than NVIDIA HGX B300 at a P90 of 50 tokens per second per user, on AgentX. By AMD's own numbers, vLLM on AMD had moved on in the four months after the SemiAnalysis note.

The first two use different metrics and setups, so they can't be compared with each other. An older LMSYS claim of up to 5x over vLLM v0.2.5, on A10G GPUs from January 2024, describes engines that no longer exist in that form.

Hardware support overlaps more than it differs

Both now target Linux and CUDA 13 on NVIDIA. vLLM has no native Windows build, and SGLang's install guide targets Linux with NVIDIA GPUs.

On AMD, vLLM's v0.33.0 notes add MonoKernel decode on MI300X, MI325X and MI355X, while SGLang 0.5.23 adds an opt-in monokernel plugin for MI350X. On Google TPUs, there are vLLM TorchTPU and SGLang-TorchTPU. On Macs, there are vLLM Metal and SGLang's Metal/MLX path.

Beyond that, vLLM lists plugins for Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions, MetaX and Iluvatar CoreX. SGLang lists Ascend, Moore Threads and Intel GPUs and CPUs, with several integrations in progress.

Frequently asked questions

How hard is it to move a client from vLLM to SGLang? Client code changes little, because vLLM and SGLang both offer OpenAI-compatible and Anthropic-compatible APIs and load Hugging Face models. Server launch commands need a rewrite. Parallelism, cache and disaggregation flags differ, so vllm serve settings don't carry over. Tool-call and reasoning parsers are also configured per engine.

Do SGLang and vLLM work on a single GPU? Yes. vLLM's sizing guide says no distributed setup is needed when a model fits on one GPU, and SGLang's docs describe serving from one GPU up to large clusters. RDMA, routers and disaggregation are optional on both.

Is a prefill/decode split needed for a small deployment? Not to gain throughput, according to vLLM's docs. Both engines offer chunked prefill as a lighter way to limit how much long prompts delay tokens in flight: on by default where possible in vLLM, and set with --chunked-prefill-size in SGLang.

How much host memory does SGLang HiCache use? HiCache is off until you pass --enable-hierarchical-cache. The host pool then defaults to twice the device pool through --hicache-ratio 2.0, and --hicache-size overrides the ratio. Storage backends such as file, Mooncake and 3FS sit behind host memory.

#sglang#vllm#llm inference#prefix caching#gpu server

We design custom AI servers

We configure each server for the models you will run, and our build partner assembles and tests it. 3-year warranty, or 5 years at extra cost.

Get in touch
Call Us