GPU Systems
GPU systems are useful when a workload has enough tensor arithmetic to amortize data movement. For ML, the hard limits are usually not “has a GPU” but HBM capacity, HBM bandwidth, PCIe or NVLink movement, kernel launch overhead, and whether the model can use lower precision safely through mixed precision. A deployment choice in managed compute therefore has to name the accelerator type, memory size, interconnect, driver stack, and serving batch shape.
The roofline model
The operating model is a roofline constraint:
Arithmetic intensity is FLOPs per byte moved from memory. A transformer prefill matmul can be compute-heavy; token-by-token decoding often becomes bandwidth- and KV-cache-bound, which connects directly to storage and decoding bottlenecks at the input side and distributed model training at the synchronization side. PyTorch also reserves memory through its CUDA caching allocator, so nvidia-smi can show reserved memory that is not currently occupied by tensors; inspect memory_allocated() and memory_reserved() when debugging PyTorch jobs.
Worked capacity check
For a 7B-parameter model, the weight footprint depends directly on bytes per parameter:
| footprint | estimate |
|---|---|
| fp32 weights | 26.08 GiB |
| fp16/bf16 weights | 13.04 GiB |
| int8 weights | 6.52 GiB |
| Adam training state | 104.31 GiB |
| 8-way FSDP state per rank | 13.04 GiB |
| 32-layer, 2048-token fp16 KV cache for one request | 1.00 GiB |
For an NVIDIA A100 80GB SXM, NVIDIA’s published 312 TFLOP/s FP16 Tensor Core peak and 2,039 GB/s memory bandwidth imply a roofline threshold of about FLOP/byte. A kernel with arithmetic intensity 32 is memory-bound under that roofline, while one at 256 can be compute-bound. A 7B model’s fp16 weights fit on one 40GB GPU, but a naive Adam training state does not fit even on 80GB without sharding, offload, or recomputation. During inference, KV cache can dominate capacity: this 32-layer, hidden-size-4096, 2048-token, fp16 example uses about 1 GiB per active request before batching overhead.
Caveats
GPU utilization can be high while user latency is bad if batching hides queueing delay. It can also be low for the wrong reason: CPU tokenization, object-store reads, or decompression may starve kernels before compute saturates. In cluster scheduling, a request for “one GPU” is underspecified; A10G, L4, A100 40GB, A100 80GB, and H100 instances expose very different memory and interconnect behavior, which changes both cost management and failure handling.
References
- NVIDIA A100 Tensor Core GPU specifications
- Amazon EC2 P4 instances
- PyTorch CUDA semantics: memory management
Nav