How Does One GPU Serve Hundreds of Users at the Same Time?

TL;DR The model weights are loaded into GPU memory once and shared by every request. Only the per-request state is duplicated. A request runs in two phases with opposite hardware behaviour: prefill is compute-bound, decode is memory-bandwidth-bound. During decode the GPU reads the entire model from memory to produce one token. Batching amortises that read across many requests, which is why it works so well. Continuous batching schedules at the token level instead of the request level, so finished requests leave and new ones join mid-flight. Concurrency is usually capped by KV cache memory, not by compute. Latency needs two numbers, not one: TTFT and TPOT. Raising batch size improves throughput and worsens both.