Skip to content
← All news

News1 min read

40+ agent-tuned operators in SGLang

More than forty agent-optimised operators are upstream in SGLang. Six of them, with their own pull requests' numbers: +71% throughput, −41% time to first token, 1.52× KDA prefill, up to 2.78× denoising and a 1.41× VAE decode.

More than 40 operators optimised by agents have been merged into SGLang, with another batch of twenty running for GLM5.2.The count, and which operators it covers, is Humanfia-reported. The numbers for the six below are each pull request's own. Six of them have numbers worth showing.

Six merged pull requests · sgl-project/sglang · 2026

When each one landed

Swipe the plot sideways to see all of it →

  • Merged

Prefill: Kimi Delta Attention on Blackwell ​

#27488 fixed the CuteDSL prefill kernel for Kimi-Linear, whose real gates overflowed exp() in fp32 and turned the output into NaN, and then made it faster than the Triton path it replaces, by reusing its scratch memory between calls. On B200, with real Kimi-Linear gates:

KDA prefill · CuteDSL vs. Triton · B200 · H = 32
20.525 → 0.346 ms
1.52×
40.768 → 0.628 ms
1.22×
81.277 → 1.182 ms
1.08×
  • CuteDSL prefill
Single sequence, K = V = 128, bf16. GSM8K on Kimi-Linear-48B: 0.915 with Triton, 0.920 with this prefill.

Serving: throughput and time to first token ​

#22664 turns on FlashInfer's allreduce fusion for Qwen3-Next, where unfused cross-device reduces dominated prefill.

Request throughput · #22664

+71.4%

Qwen3-Coder-Next on 4 × H100: 5.49 → 9.41 requests a second.

Mean time to first token

−63.3%

456.24 → 167.54 ms. Time per output token fell 49.4%.

#27401 unblocks the TensorRT-LLM NVFP4 fused-MoE kernels for Command-A-Plus on B300. Switch the baseline to compare against vLLM.

Command-A-Plus NVFP4 · 1 × B300 · requests a second
Chat1000 in / 1000 out
2.80
3.40
3.54
Summarisation8000 in / 1000 out
1.68
1.90
2.03

#28744 has the router tokenise a prompt once and hand the token ids to the engine, rather than both tokenising it.

Router tokenises once · DeepSeek-V4-Flash · 7 engines
Idle60k-token prompt
29%
Idle125k-token prompt
41%
Under load, least60k · 10–30 QPS
34%
Under load, most60k · 10–30 QPS
49%
  • Lower TTFT

Diffusion: denoising and decode ​

#27524 denoises the early, low-frequency steps at half resolution. The tolerance δ sets how many steps run coarse, so speed rises with it. Drag across the plot to read each model.

Spectral progressive diffusion · RTX A6000 · denoising loop only

Speedup against the tolerance δ

Swipe the plot sideways to see all of it →

δ = 0 is the full-resolution run. The pull request's summary gives Qwen-Image 1.6× at δ = 0.05; its own table measures 1.29× there and 1.69× at δ = 0.20, which is what is drawn.

#27431 runs the LTX-2 video VAE decode in the channels-last-3d layout, where Conv3d is several times faster on Hopper.

VAE decode stage · #27431

1.41×

5.41 → 3.84 s per video; denoising unchanged.

Peak reserved memory

−9.7 GiB

71.81 → 62.12 GiB, from fewer transposed copies.

Why an upstream pull request is the test we like most ​

A contest track is scored by a harness we can read. A production serving framework is scored by maintainers who did not ask for the change, do not care that an agent wrote it, and will reject it if it is slower on a shape we did not think about, harder to maintain, or wrong. Every number above survived that.

Two gates between a candidate and main
our harnessmaintainersmain■ refused■ merged

Swipe the diagram sideways to see all of it →

1/0 The agent produces candidate operators.

KDA · KDA-Pilot

Read next

News6 KDA PRs merged into SGLangSince August, SGLang has merged six pull requests carrying kernels our agents wrote, the best adding 8.7% end-to-end throughput. The MLSys contest kernels are public with a benchmark anyone can rerun. One merge elsewhere, Flash-KMeans, was reverted the next day.Read it BlogKDA²: Kernel Design Agents (KDA) Optimize Kimi Delta Attention (KDA)Our agents wrote Kimi Delta Attention kernels that run up to 2.96× faster than FlashKDA on B300 with a tenth of its state error. Here is how, and how the agents tried to cheat along the way.Read it News6.5× faster MSA prefill indexerOn B300, KDA found the hardware the MSA prefill and decode indexers were leaving idle: a 6.5× geometric mean on prefill, up to 14× on chunk tails, up to 3.3× on decode, and bitwise-identical output throughout. Humanfia-reported.Read it