Most people size an inference server by looking at TFLOPS. Then the model does not fit, or it fits and runs at a third of the speed they expected. Both failures come from the same place: inference is bound by memory, not by compute.
Here is the arithmetic, and the part of it that estimates usually leave out.
Step 1: the weights
A model's weights need parameters × bytes per parameter:
| precision | bytes/param | 7B | 13B | 34B | 70B |
|---|---|---|---|---|---|
| FP16 / BF16 | 2 | 14 GB | 26 GB | 68 GB | 140 GB |
| INT8 | 1 | 7 GB | 13 GB | 34 GB | 70 GB |
| INT4 | 0.5 | 3.5 GB | 6.5 GB | 17 GB | 35 GB |
This table is where most planning stops. It should not.
Step 2: the KV cache — the part everyone forgets
Every token the model has already seen is kept as key/value tensors so it does not have to be recomputed. That cache grows with context length and with how many requests you serve at once:
KV_bytes = 2 × layers × kv_heads × head_dim × context × batch × bytes_per_element
The 2 is for K and V. For a 70B-class model with 80 layers, 8 KV heads (grouped-query attention) and head_dim 128, at FP16:
- 8k context, batch 1 → ~2.6 GB
- 32k context, batch 1 → ~10 GB
- 32k context, batch 16 → ~168 GB
That last line is the one that ruins plans. The weights were 140 GB and looked like the whole problem; serving sixteen concurrent users at long context more than doubles the requirement.
Practical consequence: decide your context length and your concurrency before you choose a card. A server sized for batch 1 is a demo, not a service.
Step 3: add the overhead
Framework, CUDA context, activations and fragmentation take real space. Budget 10–15% on top of weights + KV. If the result lands within a few gigabytes of a card's capacity, treat it as not fitting — you will spend the difference the first time someone sends a long prompt.
Why bandwidth matters more than TFLOPS
Generating one token requires reading the entire active weight set from memory. At batch 1 that is the dominant cost, and the ceiling is:
tokens/second ≈ memory_bandwidth / active_bytes_per_token
A 70B model at INT8 reads roughly 70 GB per token. On a card with ~1400 GB/s of bandwidth that caps out around 20 tokens/second, no matter how many TFLOPS the card advertises. The compute units are idle waiting for memory.
This is why a card's bandwidth number deserves more attention than its FLOPS number when the workload is inference rather than training. On our own build we measured 1,396.4 GB/s — the figure is in the Vast.ai verification write-up, along with what else the platform actually checks.
Batching changes the picture: with many concurrent requests the same weight read serves multiple tokens, and compute starts to matter again. That is precisely why concurrency belongs in the sizing decision and not in a footnote.
Choosing the card
VRAM per card beats total VRAM. Two 48 GB cards do not equal one 96 GB card. Splitting a model across GPUs means every token crosses the interconnect, and the tensor-parallel overhead is real. Fit the model in one card if you can.
- RTX 5090, 32 GB — comfortable for 7B–13B at FP16, or ~30B at INT8, at modest context. Excellent tokens per euro when the model is small enough to fit.
- RTX PRO 6000 Blackwell, 96 GB — holds a 70B at INT8 with room for a real KV cache, on one card. This is the size where single-card 70B serving stops being a compromise.
- Multi-card — necessary above that, or when you need many concurrent long-context streams. Plan for the interconnect, not just the card count.
The host is not a detail
Inference makes the GPU the bottleneck only if the host lets it. Budget:
- System RAM ≥ VRAM. Models are loaded through host memory; being short here turns every start-up into a swap storm.
- NVMe, not SATA. A 140 GB model off a slow disk is minutes of load time on every restart.
- CPU threads for preprocessing. Tokenisation, image decode and request handling are CPU work. A starved host makes an expensive GPU wait.
- Network if you are serving others: 10 GbE is the sane floor.
Worked example
Serving a 70B model at INT8, 16k context, up to 8 concurrent requests:
weights 70 GB
KV cache (16k × 8) ~42 GB
overhead (~12%) ~13 GB
───────────────────────────
total ~125 GB
That does not fit one 96 GB card. Options: drop to INT4 (~35 GB weights, total ~90 GB — fits), cut concurrency to 4 (~104 GB — still tight), or go to two cards. The sizing decision and the quantisation decision are the same decision — which is why they should not be made by different people at different times.
Run your own numbers
Put your model, context and concurrency through the arithmetic above before you shortlist hardware. If it comes out in favour of buying, configure a server and we come back within 24 hours with a specification and a price. If you are still deciding whether to own the hardware at all, the rent-versus-buy break-even is the piece to read first, and the GPU ROI calculator does that arithmetic with live market rates.
A closing caveat, because sizing guides tend to overpromise: these are planning figures. Real throughput depends on your framework, your quantisation quality, your batch scheduler and your prompt shape. Use them to reject configurations that clearly cannot work, and benchmark the one that survives.