Skip to content
← All news

News1 min read

6.5× faster MSA prefill indexer

On B300, KDA found the hardware the MSA prefill and decode indexers were leaving idle: a 6.5× geometric mean on prefill, up to 14× on chunk tails, up to 3.3× on decode, and bitwise-identical output throughout. Humanfia-reported.

This one is not a contest. It is a kernel in a serving path that somebody is paying for by the hour, which is a harder test in one specific way: there is no track to win, only a regression to avoid.The kernels are in a production serving stack that is not public, so every number here is Humanfia-reported.

MSA indexers · B300 · speedup over the previous kernel
Prefillgeometric mean, all workloads
6.5×
Prefillend-of-prefill chunk tails, best
14.0×
DecodeTP4 promotion shape
2.9×
Decodelonger contexts, best
3.3×
  • KDA
Humanfia-reported. No workload in the set regressed, and the output is bitwise identical in every one.

Prefill ​

The MSA prefill indexer was badly underusing the hardware during long-context serving on B300. KDA found it and rewrote around it: 6.5× as a geometric mean across the workloads tested, up to 14× on the end-of-prefill chunk tails, where the underuse was worst and the shape least convenient, and no regression anywhere in the set.

The tails are the interesting part. A profile averaged over a run hides them, and a human optimising by eye tends to spend the week on the shape in the middle of the histogram. An agent reading ncu output as evidence has no such preference.

Decode ​

The MSA decode indexer, used in speculative verification, went 2.9× faster at the TP4 promotion shape, and up to 3.3× on longer contexts. The first number is past the 2.78× that a bandwidth-bound kernel could reach at that shape, so the win is not bandwidth alone.

Decode at the TP4 promotion shape
  1. 12.78× the theoretical bandwidth band
  2. 22.9× KDA at the TP4 promotion shape
  3. 33.3× KDA on longer contexts
A speedup of this size isbeyond the bandwidth band: more than bandwidth alone

The gate ​

Accuracy was bitwise identical across every tested workload. That is the gate, not a footnote: a kernel that is faster because it has quietly become incorrect will report a speedup just as happily as one that is faster because it is better.

Fast, and bitwise identical, or not at all
no regressionbitwise identicalship■ faster, one bit off■ faster, identical

Swipe the diagram sideways to see all of it →

1/0 Two gates: no workload slower, and every output bit the same.

KDA · NVlabs/kda

Read next

News6 KDA PRs merged into SGLangSince August, SGLang has merged six pull requests carrying kernels our agents wrote, the best adding 8.7% end-to-end throughput. The MLSys contest kernels are public with a benchmark anyone can rerun. One merge elsewhere, Flash-KMeans, was reverted the next day.Read it BlogKDA²: Kernel Design Agents (KDA) Optimize Kimi Delta Attention (KDA)Our agents wrote Kimi Delta Attention kernels that run up to 2.96× faster than FlashKDA on B300 with a tenth of its state error. Here is how, and how the agents tried to cheat along the way.Read it BlogA Quality Tier for the Fast KernelsMost profitable diffusion fusions change the output at the level of bf16 rounding order, enough to fail a bitwise CI and not enough for a person to see. So we gave the request a quality tier instead of arguing about the diff.Read it