The idea, explained
An interactive assistant and an overnight document-scoring job can use the same model but need different serving designs. The assistant cares about time to first token, or TTFT: how long the user waits before output begins. It also cares about end-to-end latency, the time until the full answer arrives. The overnight job usually cares more about throughput because finishing the entire collection efficiently matters more than any one item starting immediately.
Batching can raise throughput because a GPU performs large parallel operations efficiently. Instead of launching nearly identical work separately for every request, the server combines compatible requests. The trade-off is that a request may wait briefly while a batch forms. Microsoft’s guidance says batching is most useful for GPU-bound, batch-friendly models under concurrent traffic; if preprocessing on the CPU is the real bottleneck, adding a batching stage can merely add delay.
Modern continuous batching is more flexible than a fixed batch. A scheduler can admit a new request when another finishes, reducing idle hardware and padding waste. But more concurrency is not automatically better. As arrival rate approaches the server’s maximum processing rate, a queue forms and tail latency can rise sharply. Tail latency means the slow end of the distribution, commonly measured at the 95th or 99th percentile rather than by an average.
A responsible benchmark therefore holds the workload shape constant: model version, numerical precision, typical prompt and output lengths, expected concurrency, and serving engine. Measure TTFT, end-to-end latency, tokens per second, requests per second, error rate and queueing delay together. Select a configuration that meets a stated service-level objective, or SLO, rather than declaring one system fastest from a single number.
Follow a worked example.
Suppose a support assistant receives 24 requests during a 6-second peak, an arrival rate of 4 requests per second. With batch size 1, one GPU completes 3 requests per second. Demand exceeds capacity by 1 request per second, so after 6 seconds roughly 6 requests are waiting. Buying time with a queue does not solve the capacity gap; latency keeps growing while the burst continues.
A test with batches of 4 raises measured capacity to 6 requests per second. Each request waits an average of 0.15 seconds for a batch, model execution takes 0.55 seconds, and network plus application work takes 0.10 seconds. Estimated average end-to-end latency is 0.15 + 0.55 + 0.10 = 0.80 seconds. If the product SLO is “95% of responses begin within 1 second,” the team must still measure p95 TTFT under realistic prompt lengths; the 0.80-second average alone cannot prove compliance.
The decision is not “batching is faster.” It is: batching supplies enough throughput for this traffic shape, and its added waiting time remains acceptable when verified at the required percentile.
Where people get stuck
A common misconception is that maximum GPU utilization proves a good user experience. A server can be fully utilized while requests spend too long in a queue. Another mistake is comparing tests with different prompt lengths or generated-output lengths; longer requests consume more memory and compute, so the comparison is not controlled. Separate interactive traffic from large offline jobs when possible, define latency and throughput targets first, and test realistic bursts rather than only a smooth average load.
Now try it yourself.
1. Check the core idea
An endpoint handles 8 requests per second at acceptable latency, but traffic rises to 10 requests per second for 30 seconds. If no requests are rejected or scaled elsewhere, approximately how many requests accumulate in the queue, and what should the team test next?
Reveal the answer & reasoning
The excess arrival rate is 10 − 8 = 2 requests per second. Over 30 seconds, approximately 2 × 30 = 60 requests accumulate, assuming the rates remain steady. The team should not rely only on this estimate. It should load-test the real workload at 10 requests per second, measure queueing delay and p95 or p99 TTFT, and compare options such as higher safe concurrency, batching, another replica or controlled load shedding. The correct choice is the least costly configuration that meets the stated SLO without causing unacceptable errors or tail latency.
Why this was added
The library covers training, quantization and general reliability, but not the operational trade-off among AI inference latency, throughput, concurrency, queueing and batching.