Sizing GPU memory for inference workloads

GPU memory sizing starts with a simple question: what must remain resident while the model serves a request? The answer is more than the model file. Weights, key-value cache, temporary activations, runtime overhead and any parallel requests all occupy memory. A card that appears large enough for the weights alone can fail as soon as the production context length or batch size is applied.

Start with the weights.

A useful first estimate is parameter count multiplied by bytes per parameter. A model stored at 16-bit precision needs roughly two bytes per parameter before runtime overhead. An 8-bit representation needs about one byte, while a 4-bit representation needs about half a byte plus scaling metadata. Quantisation therefore changes the starting point substantially, but it should be tested against the required output quality, supported operators and target runtime. Do not assume that every layer will run at the advertised precision.

Leave headroom after loading the weights. Runtime libraries reserve memory, kernels need workspaces, and memory fragmentation can make the final free capacity less useful than the headline figure suggests. A sizing test should record the peak allocated and reserved memory after warm-up, not the idle reading immediately after the model loads.

Account for context and concurrency.

For transformer inference, the key-value cache often becomes the variable that changes the result. It grows with layer count, hidden dimensions, active sequences and tokens retained in context. Doubling the supported context is not a free configuration change. Serving several simultaneous conversations also multiplies cache demand, even when the model weights remain shared.

Batching improves device utilisation, but larger batches need more working memory and may increase response latency. Define whether the service optimises interactive latency, total tokens per second or a mixture of both. Then test realistic prompt and output lengths at the expected concurrency. A synthetic batch of short prompts can hide the pressure created by long production conversations.

One 48 GB card or two 24 GB cards?

One 48 GB card is usually the cleaner choice when the model and its working set fit on one device. It avoids splitting layers or tensors across a link, keeps deployment simpler and leaves both software and failure handling with fewer moving parts. This matters for latency-sensitive inference, runtimes with limited multi-device support, and servers where peer-to-peer paths are constrained by PCIe topology.

Two 24 GB cards can make sense when the runtime supports model parallelism well, the host exposes enough PCIe lanes, and aggregate throughput matters more than the latency of one request. They may also serve two independent model replicas, which improves request-level concurrency but does not help a single model that needs more than 24 GB. Memory is not automatically pooled: the deployment method must explicitly partition the model or workload.

Build the RFQ from measured constraints.

• Model family, parameter count and tested quantisation
• Maximum input context and expected output length
• Concurrent sequences and batching policy
• Required latency and throughput targets
• Runtime, framework and host-server PCIe layout
• Measured peak memory after warm-up, plus operating headroom

Treat the card capacity as one part of a node specification. Power connectors, server airflow, slot width, CPU-to-GPU lane allocation and the number of cards per chassis can rule out an otherwise suitable memory size. A short production-shaped benchmark on the intended runtime is the final check before quantity and delivery dates are committed.