Tuning GPUs for token generation: 2× the output, measured

Most GPU servers running inference today are running engine defaults. Defaults are written to work on any card with any model, which means they are tuned for nothing in particular. We measured what that costs: on one 8-GPU A100 node, the same hardware with the same model weights served 1.94× more tokens per second after tuning, at the same latency, for under a point of accuracy.

Why stock settings leave throughput on the table

Token generation is a memory problem before it is a compute problem. Every decode step streams the model's weights through the GPU, and every concurrent user holds a KV cache that competes with those weights for the same VRAM (the mechanics, the arithmetic). So the settings that decide throughput are mostly memory decisions: how many bytes each weight takes, how many bytes each cached token takes, how the model is split across GPUs, and how much work the scheduler is allowed to batch at once.

A serving engine's defaults answer all four conservatively: 16-bit everything, one big tensor-parallel group spanning the node, modest batch budgets. That configuration is safe. It is also, on most real deployments, half of what the hardware can do.

The test

The measurements below come from one 8× A100 SXM4 40GB HGX node, run October 14-20, 2026, on the validation and tuning bench at Core 4's facility in Minnesota. Every serving number is output tokens per second for the whole node, counted only inside latency targets: 50 ms per token and 2 s to first token for chat, 40 ms and 6 s for agentic work. "Stock" means engine defaults. "Tuned" means the measured recipe below. Same node, same weights, both ways.

Before any of it, the node passed a 30-minute all-GPU burn-in: 2,060 TFLOPS aggregate, 1.1% worst-case slowdown, 64 °C peak, zero errors, DCGM level 3 pass. Tuning a node that throttles or throws memory errors just produces fast wrong answers, so health comes first.

The headline: Qwen3.8 27B, chat

StockTunedChange
Output tokens/s per node2,1824,2351.94×
Tokens per joule (whole node, at the wall)0.531.042.0×
Concurrent users under 50 ms/token1042502.4×
GSM8K accuracy96.2%95.6%-0.6 pt

One 8× A100 SXM4 40GB HGX node, October 2026. The accuracy change sits inside the benchmark's 95% confidence interval. A single user gets 98 tokens/s on the tuned configuration.

Read the third row twice. Throughput per node is the number benchmarks report, but capacity is what a service sells: under the same 50 ms/token ceiling, the tuned node holds 250 concurrent users instead of 104.

What we changed: the five settings that moved the number

1. INT8 weights, because Ampere has INT8 tensor cores

Halving the bytes per weight nearly halves the bytes streamed per token. The format matters as much as the bit width: A100s have no FP8 compute, so FP8 checkpoints run weight-only and dequantize on the fly. INT8 runs on the tensor cores directly, and in our runs it beat FP8 checkpoints by 50% on the same card (why tensor core generations decide this).

2. An FP8 KV cache

Storing the cache at 8 bits instead of 16 doubles the number of tokens of context the node can hold, and cached context is the ceiling on concurrent users.

3. Smaller tensor-parallel groups, more model copies

Splitting a 27B model across all eight GPUs spends a large share of every step on GPU-to-GPU communication. Once 8-bit weights make the model small enough, running it on smaller groups and serving several copies side by side keeps each GPU busy computing instead of waiting on the interconnect.

4. Bigger batch and prefill budgets

With memory freed up, the scheduler can batch more sequences per step and admit longer prompts without stalling decode. Continuous batching only pays when it is allowed to batch.

5. The right engine for the model

The headline run used SGLang. Engine choice and its flags interact with everything above, so the recipe is per model, not one config for the fleet.

Every model we served

ModelWorkloadStock tok/sTuned tok/sGainNote
Qwen3.8 27Bchat2,1824,2351.94×SGLang, INT8 weights, FP8 KV cache
Qwen3.8 27Bagentic, 32k context1,0061,7181.71×
Qwen3.8-Flash-Nextchat1,5892,3671.49×two model copies per node
GLM-5.3-Flashchat7218991.25×
DeepSeek-V4-Flashchat9231,0861.18×
Nemotron 3.5 Super VL 120Bchat1,6971,8831.11×agentic: 1,318 tok/s
MiniMax-M3 428Bchat9381,0241.09×memory-bound on 40GB cards
MiMo-V2.6-Flashchat2,4652,4621.00×already 8-bit; 151 tok/s single user

Same node, chat latency targets unless noted. Full runs, configurations, and raw results are published on Charg's A100 benchmark page.

The spread is the useful part. Dense and hybrid models in the 27B class nearly double. Large mixture-of-experts models gain 9-25%, because on 40GB cards they are memory-bound before tuning has much room to work. A model that already ships in 8-bit gains nothing, since the biggest lever was pulled before we got there. Tuning is not a multiplier you apply to a fleet; it is a measurement you take per model.

The energy side: tokens per joule

The node drew 2.4 kW idle, 5.1 kW under full GPU load, and 3.5-4.2 kW while serving, before and after tuning alike. Output doubled inside the same power envelope, so tokens per joule at the wall went from 0.53 to 1.04. For anyone paying for power, which is everyone running their own hardware, that is the same as halving the energy bill per token.

One finding surprised us: power caps did nothing useful. While serving, each GPU drew only 210-250 W. Roughly 2 kW of the node's draw is chassis fans, which no GPU setting touches. The lever for energy per token is output, not wattage.

What did not work

Where tuning stops, and what the card size still decides

Tuning does not change what fits. On the same configuration, the 40GB node delivered 0.89-0.97 of an 80GB node across seven models, so most of the difference between the two cards is memory headroom rather than speed. For very large models it is more than that: an 80GB node fits two copies of a 150-250 GB model where a 40GB node fits one, and best layout against best layout the 40GB node lands at 0.61-0.83 of the 80GB on those models. If your model is that big, the larger card is the right tool; the VRAM calculator will tell you quickly, and the 40GB vs 80GB guide covers the trade in depth.

What this does to cost per token

Cost per token is cost per hour divided by tokens per hour. Tuning leaves the numerator alone and doubles the denominator, so on the headline model the cost per token of the same node halves, with no hardware change. For anyone already running GPUs, it is worth measuring before buying more of them. The inference cost estimator shows the effect at your own utilization, and the cost-per-token playbook ranks it against the other levers.

It also changes how long hardware stays useful. NVIDIA's Jensen Huang said at GTC Berlin on October 21, 2026 that GPUs now remain economically useful "eight years or longer," against the three-to-five-year schedules most fleets are depreciated on. Our numbers are one concrete version of that argument: a node that serves twice the output it did on engine defaults is not the same asset it was on day one.

Disclosure and next steps. Charg and Core 4 are run by the team behind Virtualized. Charg operates validated, enterprise-grade A100 HGX nodes as dedicated, single-tenant capacity, each shipped with its burn-in report, a before/after sheet, and the tuned stack installed. Every number in this article links back to its run on the public benchmark page. If you already own A100 racks, Core 4 runs the same audit, test, and tuning process on hardware you send to the Minnesota bench, and returns it with a per-rack before/after sheet. Reserve capacity →

Related