Aura Digital
Sofia · 42.69°N 23.32°E
Get a quote
All articles
Guides · November 16, 2026 · 7 min read

vLLM vs Ollama vs llama.cpp: concurrency decides the choice

How vLLM, Ollama and llama.cpp differ on concurrent load, memory, model files and hardware, with named benchmarks and versions as of November 2026.

vLLM, Ollama and llama.cpp can all put an open-weight model behind an OpenAI-style API, but they treat load, memory and hardware differently. This comparison uses their own documentation and public benchmarks that name their hardware, as of November 2026.

Which one for whom

The trade-off: vLLM gives up desktop convenience and serves one model per server. Ollama and llama.cpp give up throughput under heavy parallel load.

vLLM vs Ollama vs llama.cpp at a glance

vLLM Ollama llama.cpp (llama-server)
Parallel requests by default Continuous batching 1 per model, then a queue of 512 Automatic slots, continuous batching on
Models on one server 1 3 per GPU, if they fit 1, or 4 in router mode
GPU memory Default share of 0.92, or 1.0 with extensible KV cache Loaded on demand, unloaded after 5 idle minutes Set by model and context size
Model files Hugging Face weights; GGUF experimental Library, GGUF, Safetensors GGUF
Operating systems Linux; Windows through WSL macOS, Windows, Linux, Docker Windows, macOS, Linux
Release checked v0.33.0rc3, Nov 2, 2026 v0.40.15, Nov 3, 2026 v0.9.0, Nov 4, 2026
License Apache-2.0 MIT MIT

The first two rows matter on a shared server. The release row will age quickly, so check it before relying on any version detail below.

vLLM vs Ollama under concurrent load

vLLM is built around continuous batching and PagedAttention, which allocates KV-cache memory in blocks. Ollama's FAQ describes a per-model setting instead. OLLAMA_NUM_PARALLEL defaults to 1, OLLAMA_MAX_QUEUE holds 512 waiting requests, and anything beyond that returns a 503. Raising the first setting costs memory, because required RAM scales with OLLAMA_NUM_PARALLEL times OLLAMA_CONTEXT_LENGTH.

A public test with named hardware comes from Harshith Umesh at Red Hat, published in August 2025. It ran Llama 3.1 8B on one NVIDIA A100-PCIE-40GB with vLLM 0.9.1 and Ollama 0.9.2, using GuideLLM from 1 to 256 virtual users. Red Hat reports a peak of 793 tokens per second for vLLM against 41 for Ollama, and P99 latency of 80 ms against 673 ms. With OLLAMA_NUM_PARALLEL=32, the highest value stable on that GPU, Ollama's throughput plateaued and its inter-token latency turned erratic.

Three limits apply to those numbers:

  • They are Red Hat's, and Red Hat's inference stack is vLLM-powered.
  • The versions are from mid-2025. The test describes an Ollama default of four parallel requests, while the docs now say 1, and v0.40.15 added concurrent serving for recurrent and vision GGUF models.
  • At default settings, the same test saw vLLM's inter-token latency rise above 16 concurrent users while Ollama's stayed low and stable.

llama.cpp vs vLLM: level at one user, far apart at 64

Red Hat's September 2025 test used one NVIDIA H200-PCIe-141GB, Llama 3.1 8B at full precision, vLLM 0.10.0 and llama.cpp build b6100, with 1 to 64 users. Output tokens per second were comparable at one user. At peak load, vLLM delivered more than 35 times the requests per second and more than 44 times the output tokens per second. A June 2026 Red Hat article adds that llama.cpp took more than three minutes to first token at 64 users. Above four users, though, llama.cpp's inter-token latency was the lower and steadier one.

The published llama.cpp flags cover GPU layers and threads, with no slot count listed. Current llama-server defaults to automatic slots with continuous batching on, so a rerun on a recent build could differ. Red Hat also says the result reflects a high-concurrency data-center scenario, not a general ranking.

Memory: vLLM reserves, Ollama and llama.cpp flex

--gpu-memory-utilization in vLLM defaults to 0.92, or 1.0 with the extensible KV cache. The startup log reports GPU KV cache size and Maximum concurrency, which are the figures to size against. The vLLM FAQ says one server does not host several models, and suggests one instance per model with a routing layer in front.

Ollama keeps a model in memory for 5 minutes by default, unloads idle models to make room, and queues requests when a new model does not fit. llama-server offers something similar through router mode, with --models-dir and a default cap of 4 loaded models.

A model too large for VRAM also takes a different path in each. llama.cpp runs hybrid CPU and GPU inference, and ollama ps can report a split such as 48%/52% CPU/GPU. vLLM's memory guide lists tensor parallelism across GPUs, quantization, and lower context length and batch size.

Model files: GGUF belongs to Ollama and llama.cpp

llama.cpp supports integer quantization from 1.5-bit to 8-bit, and its quick-start examples load GGUF files. Ollama imports GGUF and Safetensors, but it does not quantize GGUF during import and points to llama-quantize for that step.

vLLM states support for 200+ architectures on Hugging Face, with FP8, INT8, INT4, GPTQ and AWQ among its quantization formats. GGUF now lives in a separate vllm-gguf-plugin, and the docs call that support "highly experimental and under-optimized". They also advise using the base model's tokenizer instead of the one in the GGUF file.

Hardware and operating systems: vLLM is Linux-first

vLLM's install guide says Windows is not supported natively and points to WSL. NVIDIA GPUs need compute capability 7.5 or higher, AMD needs ROCm 6.3 or above, and Intel XPU starts with basic inference and serving. On Apple silicon, GPU acceleration comes from vLLM-Metal, a community plugin built on MLX.

Ollama accepts NVIDIA compute capability 5.0 or higher, AMD with ROCm v7, Metal on Apple devices, and Vulkan on Windows and Linux. Its MLX engine for Apple silicon entered preview in March 2026. llama.cpp lists Metal, CUDA, HIP, Vulkan, SYCL, OpenCL, WebGPU and CANN, plus AVX, AVX2, AVX512 and AMX on x86 CPUs.

Across several GPUs, Ollama stays on one GPU when the model fits and spreads across all of them when it does not. vLLM uses --tensor-parallel-size within a node, and Ray or the native --nnodes flags across nodes. llama.cpp lists an RPC backend.

How Ollama and llama.cpp relate

Ollama's README lists llama.cpp as a supported backend, and its June 2026 post on version 0.30 credits llama.cpp for better GGUF performance and compatibility. The MLX path on Apple silicon is separate. On top of the engine, Ollama adds a model library and automatic loading, plus optional cloud models.

Moving between them: the APIs are close, not identical

All three offer OpenAI-compatible routes, so client code often needs only a new base URL. All three also speak the Anthropic Messages API: vLLM, llama-server and, since January 2026, Ollama.

Ollama's compatibility layer lists tool_choice, logprobs, logit_bias and n as unsupported on chat completions. Its local endpoint requires an API key value but ignores it, and it binds to 127.0.0.1 by default. Opening it to a network therefore calls for your own authentication in front.

FAQ

Can vLLM run without a GPU?

Yes, with limits. vLLM's CPU guide covers basic inference and serving on x86 under Linux and on ARM with NEON. Pre-built x86 wheels exist since version 0.17.0, and avx512f is recommended. CPU support on Apple silicon is experimental and needs a build from source.

Does Ollama send prompts to a cloud service?

Local models run on your machine. Only cloud models run on Ollama's servers, where prompts and responses are processed to answer requests, and Ollama says it does not use them for training. Cloud features can be turned off if only local models are wanted.

How do you run two models on one GPU with vLLM?

Start two vLLM instances, one per model, and put a routing layer in front. --gpu-memory-utilization is a per-instance limit, so set each instance to a share that leaves room for the other.

Does llama-server support more than one user?

Yes. Its feature list includes parallel decoding with multi-user support and continuous batching. The -np flag sets the slot count, and its default of -1 means automatic.

#vllm#ollama#llama.cpp#llm inference#gpu server

We design custom AI servers

We configure each server for the models you will run, and our build partner assembles and tests it. 3-year warranty, or 5 years at extra cost.

Get in touch
Call Us