Skip to content
← All posts

Blog2 min read

A Quality Tier for the Fast Kernels

Most profitable diffusion fusions change the output at the level of bf16 rounding order, enough to fail a bitwise CI and not enough for a person to see. So we gave the request a quality tier instead of arguing about the diff.

A large share of the profitable kernel fusions in a diffusion serving stack change the output only at the level of bf16 rounding order. Folding the FFN tanh-GELU into the up-projection GEMM as a cuBLASLt epilogue is the canonical example: mathematically the same computation, a different order of operations, and therefore a different last bit.

SGLang's diffusion CI compares generated images bitwise against the reference. So none of these fusions could ever land on the default path. This is not hypothetical. An ungated FLUX GELU fusion was merged as #28166, with pixels a mean 2.56 out of 255 from main, and reverted four days later in #28708 because it "can move generated images away from the original model behavior".

The usual next move is to argue about the tolerance in CI. That trade is bad in both directions: loosen it and the guarantee everyone relies on is gone; keep it and a whole class of real speedups is unshippable.

The tier ​

#33453 gives every request a quality of lossless or high, and nothing else:

One request, two tiers
requestlossless · defaultreference path · bitwise CIhigh · opt-infast paths · PSNR > 25 dB

Swipe the diagram sideways to see all of it →

1/0 Every request says which quality it wants.

  • lossless, the default, is the exact reference path. Its output stays byte-identical, the ground-truth CI stays green, and nobody's guarantee changed.
  • high opts into validated fast paths whose deviation is bounded and visually indistinguishable.

Within two days, six fast paths had shipped behind it, including the GELU fusion that had been reverted in June.

Diffusion fast paths in SGLang · 2026

Reverted on the default path, shipped behind the tier

Swipe the plot sideways to see all of it →

A claim with a gate ​

"Visually indistinguishable" is a claim, so it has a gate. #33453 sets the bar at a same-seed PSNR above 25 dB against the lossless output, and leaves enforcing it to follow-up work; every fast path that shipped under the tier reports it, with side-by-side samples. The six kernel fast paths of the first two days measured 34.7 to 59.1 dB. The one figure near the bar belongs to the tier's first consumer, which is not a kernel at all: a caching policy for MiniMax-H3, at 28.2 dB.We first wrote that 34–58 dB was measured across every pull request shipped under the tier. The kernel range is 34.7–59.1 dB, and the caching policy sits lower, at 28.16 dB.

PSNR against the lossless output · lowest and highest sample
#33451FLUX.2 VAE decode
56.70 dB
59.11 dB
#33818AutoencoderKL decode
55.17 dB
57.90 dB
#33822Ideogram 4 norm chains
37.91 dB
58.07 dB
#33536FFN GELU epilogue
37.08 dB
55.35 dB
#33546Wan VAE RMSNorm + SiLU
37.67 dB
39.73 dB
#33819FLUX.1 gates + GELU
34.68 dB
40.08 dB
#33453MiniMax-H3 Cache-DiT policy
28.16 dB
28.16 dB
Each pull request's own quality table: several prompts, and for #33546 the mean over 81 video frames per prompt.

Set that against what each one buys, and the two kinds of deviation separate cleanly. Kernel fusions save a few percent and stay far above the bar. Skipping work by caching saves far more, and spends most of the margin.

What quality=high buys, and what it costs
  • #33536 Qwen-Image: End-to-end time saved 2.2%, PSNR vs. lossless 37.1 dB
  • #33822 Ideogram 4 FAST_20: End-to-end time saved 2.9%, PSNR vs. lossless 37.9 dB
  • #33819 FLUX.1-dev: End-to-end time saved 3.3%, PSNR vs. lossless 34.7 dB
  • #33822 Ideogram 4 INSTANT_8: End-to-end time saved 3.4%, PSNR vs. lossless 41.1 dB
  • #33818 Z-Image-Turbo: End-to-end time saved 4.9%, PSNR vs. lossless 57.5 dB
  • #33546 FastWan2.2: End-to-end time saved 5.1%, PSNR vs. lossless 37.7 dB
  • #33818 FLUX.1-schnell: End-to-end time saved 7.2%, PSNR vs. lossless 55.2 dB
  • #33451 FLUX.2-klein 2048²: End-to-end time saved 10.2%, PSNR vs. lossless 57.9 dB
  • #33451 FLUX.2-klein 1024²: End-to-end time saved 13.5%, PSNR vs. lossless 56.7 dB
  • #33453 MiniMax-H3 Cache-DiT: End-to-end time saved 28.6%, PSNR vs. lossless 28.2 dB

Time saved is each pull request's own end-to-end measurement (for #33453, its 1.40× speedup). Every point is above the 25 dB bar.

The general lesson is not about diffusion. When correctness is a spectrum and the CI has to be a boolean, the fix is usually to make the request say which end of the spectrum it wants, rather than to move the boolean.

When correctness is a spectrum and the CI is a boolean, let the request choose.

KDA · the PR · Lossless Is a Choice

Read next

News6 KDA PRs merged into SGLangSince August, SGLang has merged six pull requests carrying kernels our agents wrote, the best adding 8.7% end-to-end throughput. The MLSys contest kernels are public with a benchmark anyone can rerun. One merge elsewhere, Flash-KMeans, was reverted the next day.Read it BlogKDA²: Kernel Design Agents (KDA) Optimize Kimi Delta Attention (KDA)Our agents wrote Kimi Delta Attention kernels that run up to 2.96× faster than FlashKDA on B300 with a tenth of its state error. Here is how, and how the agents tried to cheat along the way.Read it News6.5× faster MSA prefill indexerOn B300, KDA found the hardware the MSA prefill and decode indexers were leaving idle: a 6.5× geometric mean on prefill, up to 14× on chunk tails, up to 3.3× on decode, and bitwise-identical output throughout. Humanfia-reported.Read it