Qwen3.8-27B · NVFP4 · single RTX 5090 (32 GB) · hot-attach, no engine fork
Attaching compiled weight structures to a stock serving engine returns the memory it saves as KV cache — roughly tripling usable context on one consumer GPU, with decode speed held at depth.
Same checkpoint, same engine binary, same flags plus three environment variables. Everything below was measured on one machine, by one person, with stock baselines re-run in the same session as the attached arms.
What this is — and isn't. vLLM and SGLang each ship many launch configurations, and the right one depends on your workload; the stock arms here are honest defaults, not the best any expert could tune. So this page is not a better-or-worse comparison. It's an exploration of hot-plugging: attaching verified structures to an unmodified engine at load time to cover boundary needs the default configuration doesn't reach — in this case, context length on a single consumer GPU.
Why context grows, in one sentence: the attached structures adopt the checkpoint's quantized weight packs in place and release the engine's duplicated host-format buffers, and every byte released flows straight back into the KV cache pool — which is exactly where context lives.
Try it on your own boundary. The structures layer is built to be extended by coding agents: point one at AGENTS.md — the complete operating procedure for producing new structures — profile your own serving hot path, and assemble the hot-plug your workload actually needs. What's demonstrated here for context, the same recipe can do for whatever your profile says is in the way.
Maximum servable context per configuration. Stock and attached arms use identical engine flags; the attached arm's extra context is memory the structures released.
Live recordings of the same experiment: context fills turn by turn until one side runs out of memory. Left pane is the stock server, right pane is the attached one; the curve below tracks decode tok/s against accumulated context.
Real long requests served by the attached engine at depths where the stock configuration refuses or runs out of memory. Decode tok/s, greedy, real code prompts.
Decode-only tok/s, real code prompts, greedy. The cleanest apples-to-apples cut: no acceptance-length variance in play.
Same protocol, speculative arms on. Per-token deltas here carry acceptance-length content variance (see †); the full set is shown, wins and losses alike.
Prefill wins are uniform: every configuration starts its answer sooner when attached. Bars show TTFT reduction (longer bar = bigger win).
Decode-only tok/s, real code and text prompts, greedy. Stock baselines re-measured in the same window (32K vLLM figures are 3-boot medians).
| configuration | stock | attach | Δ decode | Δ TTFT |
|---|---|---|---|---|
| vLLM — no spec | −11…−21% | |||
| 2K code / text | 71.8 / 66.1 | 84.3 / 84.2 | +17% / +27% | |
| 32K code / text | 68.1 / 70.3 | 79.4 / 79.4 | +17% / +13% | |
| vLLM — MTP K=6 | −11% | |||
| 2K (paired by prompt) | — | — | +2…+6% | |
| 32K code | 129.9 | ≈198 | +52% | |
| 32K text | 202.8 | ≈183 | −5…−15%† | |
| SGLang — no spec | −8…−14% | |||
| 2K / 32K code | 75.2 / 71.5 | 83.4 / 78.9 | +11% / +10% | |
| SGLang — DSpark | −8% | |||
| 2K code / text | 143.8 / 116.1 | 156.7 / 132.5 | +9% / +14% | |
| 32K code / text | 205.4 / 118.2 | 200.6 / 152.0 | −2% / +29%† | |
| vLLM — DFlash2 (PR branch, conc 1) | −13% | |||
| 2K code | 169.6 | 224.8 | step rate +9%† | |
| 32K code | 250.5 | 223.2 | step rate +4.5%† |
† Speculative-decode tok/s carries acceptance-length variance: at temperature 0 the two arms' generations diverge after a few hundred tokens, and the acceptance length re-rolls with the content. The clean kernel-level metric is verify-step rate, which is uniformly +9…+13% for the attached arms (vLLM MTP; +1.6…+9% on the DFlash2 branch). Per-token decode deltas above are reported as honest bands, not point claims.
This is a hot-plug experiment: the question tested is "what happens when compiled structures attach to an unmodified serving engine," not "which engine is fastest." Stock configurations follow each engine's documented defaults for this model plus the flags shown in the reproduction docs; they are not exhaustively tuned, and both engines move fast — treat their absolute numbers as a snapshot.
All comparisons are paired: same checkpoint, same prompts, same flags, stock baseline re-measured in the same session. The DFlash2 rows come from an unmerged experimental vLLM pull request, measured at concurrency 1, with a one-line local relaxation to allow a quantized LM head in candidate selection (acceptance lengths stayed in the normal band).
Everything here was measured by one engineer on one GPU. Reproduction recipes, serve scripts, and the raw receipts ship with the repository documentation.