Qwen3.8-27B · NVFP4 · single RTX 5090 (32 GB) · hot-attach, no engine fork

The Context Dividend

Attaching compiled weight structures to a stock serving engine returns the memory it saves as KV cache — roughly tripling usable context on one consumer GPU, with decode speed held at depth.

Same checkpoint, same engine binary, same flags plus three environment variables. Everything below was measured on one machine, by one person, with stock baselines re-run in the same session as the attached arms.

What this is — and isn't. vLLM and SGLang each ship many launch configurations, and the right one depends on your workload; the stock arms here are honest defaults, not the best any expert could tune. So this page is not a better-or-worse comparison. It's an exploration of hot-plugging: attaching verified structures to an unmodified engine at load time to cover boundary needs the default configuration doesn't reach — in this case, context length on a single consumer GPU.

Why context grows, in one sentence: the attached structures adopt the checkpoint's quantized weight packs in place and release the engine's duplicated host-format buffers, and every byte released flows straight back into the KV cache pool — which is exactly where context lives.

Try it on your own boundary. The structures layer is built to be extended by coding agents: point one at AGENTS.md — the complete operating procedure for producing new structures — profile your own serving hot path, and assemble the hot-plug your workload actually needs. What's demonstrated here for context, the same recipe can do for whatever your profile says is in the way.

Where the stock engine stops — and where attach stops

Maximum servable context per configuration. Stock and attached arms use identical engine flags; the attached arm's extra context is memory the structures released.

vLLM · MTP speculative (K=6)native model length 262,144
stock
~102K ✕ out of memory
attach
262,144 — full native length2.56×
vLLM · DFlash2 block draftexperimental PR branch, concurrency 1
stock
36,256 ✕
attach
107,1202.95×
SGLang · DSpark speculativeKV pool capacity, tokens
stock
34,659 ✕
attach
87,1282.51×
065,536131,072196,608262,144 tokens

Watch the race

Live recordings of the same experiment: context fills turn by turn until one side runs out of memory. Left pane is the stock server, right pane is the attached one; the curve below tracks decode tok/s against accumulated context.

With MTP speculation — stock dies at 96K accumulated context; the attached server keeps answering to 224K, up against the model's native 262,144 boundary.
No speculation, prefix caching on — the same wall, without the speculative arm in play.

Decode speed in territory the stock engine can't enter

Real long requests served by the attached engine at depths where the stock configuration refuses or runs out of memory. Decode tok/s, greedy, real code prompts.

60K tokensvLLM · MTP K=6
274.9 tok/s
80K tokensSGLang · DSpark
210.8 tok/s
100K tokensvLLM · DFlash2
192.7 tok/s
200K tokensvLLM · MTP K=6
178.3 tok/s

stock arms at these depths: request refused or server out of memory — there is no grey bar to draw

Decode, stock vs attach — no speculative decoding

Decode-only tok/s, real code prompts, greedy. The cleanest apples-to-apples cut: no acceptance-length variance in play.

vLLM · 2K
stock
71.8
attach
84.3+17%
vLLM · 32K
stock
68.1
attach
79.4+17%
SGLang · 2K
stock
75.2
attach
83.4+11%
SGLang · 32K
stock
71.5
attach
78.9+10%
0255075100 tok/s

Decode, stock vs attach — with speculative decoding

Same protocol, speculative arms on. Per-token deltas here carry acceptance-length content variance (see †); the full set is shown, wins and losses alike.

vLLM MTP · 32K code
stock
129.9
attach
≈198+52%
vLLM MTP · 32K text
stock
202.8
attach
≈183 (−5…−15%†)
SGLang DSpark · 2K code / text
stock
143.8 / 116.1
attach
156.7 / 132.5+9% / +14%
SGLang DSpark · 32K code / text
stock
205.4 / 118.2
attach
200.6 / 152.0 (−2% / +29%)
052.5105157.5210 tok/s

bars scale to the code figure of each pair; the clean kernel metric across all speculative arms is verify-step rate: +9…+13% attached

Time to first token — reduction from attach

Prefill wins are uniform: every configuration starts its answer sooner when attached. Bars show TTFT reduction (longer bar = bigger win).

vLLM · no spec2K
−21%
vLLM · no spec32K
−11%
vLLM · MTP serve141 → 123 ms
−13%
vLLM · DFlash232K
−13%
SGLang · no spec2K
−14%
SGLang · DSpark32K
−8%
0−25%

The full matrix — with and without speculative decoding

Decode-only tok/s, real code and text prompts, greedy. Stock baselines re-measured in the same window (32K vLLM figures are 3-boot medians).

configurationstockattachΔ decodeΔ TTFT
vLLM — no spec−11…−21%
2K code / text71.8 / 66.184.3 / 84.2+17% / +27%
32K code / text68.1 / 70.379.4 / 79.4+17% / +13%
vLLM — MTP K=6−11%
2K (paired by prompt)——+2…+6%
32K code129.9≈198+52%
32K text202.8≈183−5…−15%†
SGLang — no spec−8…−14%
2K / 32K code75.2 / 71.583.4 / 78.9+11% / +10%
SGLang — DSpark−8%
2K code / text143.8 / 116.1156.7 / 132.5+9% / +14%
32K code / text205.4 / 118.2200.6 / 152.0−2% / +29%†
vLLM — DFlash2 (PR branch, conc 1)−13%
2K code169.6224.8step rate +9%†
32K code250.5223.2step rate +4.5%†

† Speculative-decode tok/s carries acceptance-length variance: at temperature 0 the two arms' generations diverge after a few hundred tokens, and the acceptance length re-rolls with the content. The clean kernel-level metric is verify-step rate, which is uniformly +9…+13% for the attached arms (vLLM MTP; +1.6…+9% on the DFlash2 branch). Per-token decode deltas above are reported as honest bands, not point claims.

How to read these numbers — a fairness explanation, not a benchmark claim

This is a hot-plug experiment: the question tested is "what happens when compiled structures attach to an unmodified serving engine," not "which engine is fastest." Stock configurations follow each engine's documented defaults for this model plus the flags shown in the reproduction docs; they are not exhaustively tuned, and both engines move fast — treat their absolute numbers as a snapshot.

All comparisons are paired: same checkpoint, same prompts, same flags, stock baseline re-measured in the same session. The DFlash2 rows come from an unmerged experimental vLLM pull request, measured at concurrency 1, with a one-line local relaxation to allow a quantized LM head in candidate selection (acceptance lengths stayed in the normal band).

Everything here was measured by one engineer on one GPU. Reproduction recipes, serve scripts, and the raw receipts ship with the repository documentation.