Kernel geomean over 16 production shapes; Qwen3.5-4B output throughput +8.73% at concurrency 1.
ProjectOpen research prototype · built with MIT HAN Lab
An agent workflow that writes and tunes GPU kernels on real hardware, and checks that each speedup keeps the output correct. Built with MIT HAN Lab.
KDA² · Kimi Delta Attention · B300
2.96×
geomean speedup over FlashKDA, Moonshot AI's kernel, with a tenth of its final-state error.
time as a fraction of FlashKDA's · geomean of six 8,192-token workloads
01What it has done
Each tile links to its write-up.
KDA² · v0.6 · B300
2.96×
Kimi Delta Attention over FlashKDA at 8,192 tokens, up from 1.61× in eight weeks.
The write-upSOL-ExecBench · board v1.1 · 5 October
71/ 235
Per-kernel first places on 5 October, then the most of any entrant (53 in June, Humanfia-reported). Newer entries have taken most of them since.
The write-upMLSys 2026 · FlashInfer
#1 #2 #3
On MoE, DSA and GDN in the Full-Agent tracks. Public KDA 0.5 kernels later passed the best human entry on all three.
The write-upSGLang · upstream
40+
Agent-optimised operators merged by June; six more pull requests since August, the best adding 8.73% end to end.
The write-upMSA indexer · B300 · in production
6.5×
Geomean on prefill; up to 3.3× on long-context decode. Bitwise-identical output (Humanfia-reported).
The write-upSOL-ExecBench L1 · board v1.0
0.7639
First on 94 single-operation kernels, by 0.00006. Third on the newer v1.1 board on 5 October.
The write-upSGLang-Diffusion · quality tier
34.7dB
Lowest PSNR of the first six fast paths shipped under --quality high. The bar is 25 dB.
The write-upAMD MI350X · ROCm/FlyDSL
2PRs
Merged in AMD's FlyDSL, which has few published kernels to learn from.
The write-up02The problem
Speed is easy to measure; correctness is easy to break without noticing. KDA's own agents produced kernels like this on the way to KDA². Six gates now stop them before release.
KDA² candidates · speedup over FlashKDA · B300
Replaced the L2 normalisation of Q and K with a constant tuned to the test inputs. With the shortcut removed, 2.48×.
Dropped state older than 32 tokens. Measured on one H64 sequence.
Decay past 126 bits underflowed to zero. The tests never went past about 52.
Passes the real Kimi-Linear acceptance suite.
On the synthetic tests: Input-distribution overfitting, 3.74×; Illegal history truncation, 5.16×; Overflow under extreme gate ranges, 3.57×; KDA + TIRx, released, 2.96×.
Six gates every candidate must pass
03How the loop works
The plan starts from ncu reports, a kernel wiki of known techniques, and the repository.
The objective, constraints, validation command and promotion bar are fixed before any code is written.
Incorrect candidates stop at validation. Only a candidate that clears the bar is promoted.
Benchmarks, profiles and decisions are logged so another engineer can check the run.
04Speedups
In May, two of KDA's five contest kernels were still slower than the baseline. With CuteDSL, a profiling skill and a better flow, the same model passed the best human entries on all three tracks.
B200, speedup over the FlashInfer baseline; each track has its own scale. From the public release's table: geomean over the official workloads, timed as CUPTI kernel spans, not as the contest scored. Margins are geomeans of per-workload ratios over the best human entry.
05KDA²
Eight weeks of results on B300. Scroll to play the chart, or point at any date.
Geomean over six 8,192-token workloads. Candidates at 3.28×, 3.57× and 3.74× overfitted the tests and were refused; see the KDA² post.
06Upstream
Each pull request reports the kernel speedup and the end-to-end effect on the model.
Kernel geomean over 16 production shapes; Qwen3.5-4B output throughput +8.73% at concurrency 1.
flash-kmeans#23 · reverted A KDA backend for Flash-KMeans (7.31× on LongCat-Video) was merged in August and reverted the next day. It is not counted here.
07Results, as they came in
08Why it is here
09Try it
git clone --recurse-submodules https://github.com/NVlabs/kda.gitEach task brings its own evaluator, data and profiling tools. The repository has the agent flow, prompts and skills. Kernels people want next are in the KDA wishlist: request one as a pull request to its wishlist branch.