Skip to content

ProjectOpen research prototype · built with MIT HAN Lab

KDAKernel Design Agents

An agent workflow that writes and tunes GPU kernels on real hardware, and checks that each speedup keeps the output correct. Built with MIT HAN Lab.

KDA² · Kimi Delta Attention · B300

2.96×

geomean speedup over FlashKDA, Moonshot AI's kernel, with a tenth of its final-state error.

Measured on
B200
B300
H200
RTX PRO 6000
GB10
MI350X

01What it has done

Every result here was measured by an outside contest, leaderboard or maintainer.#

Each tile links to its write-up.

KDA² · v0.6 · B300

2.96×

Kimi Delta Attention over FlashKDA at 8,192 tokens, up from 1.61× in eight weeks.

The write-up

SOL-ExecBench · board v1.1 · 5 October

71/ 235

Per-kernel first places on 5 October, then the most of any entrant (53 in June, Humanfia-reported). Newer entries have taken most of them since.

The write-up

MLSys 2026 · FlashInfer

#1 #2 #3

On MoE, DSA and GDN in the Full-Agent tracks. Public KDA 0.5 kernels later passed the best human entry on all three.

The write-up

SGLang · upstream

40+

Agent-optimised operators merged by June; six more pull requests since August, the best adding 8.73% end to end.

The write-up

MSA indexer · B300 · in production

6.5×

Geomean on prefill; up to 3.3× on long-context decode. Bitwise-identical output (Humanfia-reported).

The write-up

SOL-ExecBench L1 · board v1.0

0.7639

First on 94 single-operation kernels, by 0.00006. Third on the newer v1.1 board on 5 October.

The write-up

SGLang-Diffusion · quality tier

34.7dB

Lowest PSNR of the first six fast paths shipped under --quality high. The bar is 25 dB.

The write-up

AMD MI350X · ROCm/FlyDSL

2PRs

Merged in AMD's FlyDSL, which has few published kernels to learn from.

The write-up

02The problem

An incorrect kernel can still report a speedup.#

Speed is easy to measure; correctness is easy to break without noticing. KDA's own agents produced kernels like this on the way to KDA². Six gates now stop them before release.

KDA² candidates · speedup over FlashKDA · B300

AInput-distribution overfitting
3.74×

Replaced the L2 normalisation of Q and K with a constant tuned to the test inputs. With the shortcut removed, 2.48×.

CIllegal history truncation
5.16×

Dropped state older than 32 tokens. Measured on one H64 sequence.

DOverflow under extreme gate ranges
3.57×

Decay past 126 bits underflowed to zero. The tests never went past about 52.

✓KDA + TIRx, released
2.96×

Passes the real Kimi-Linear acceptance suite.

On the synthetic tests: Input-distribution overfitting, 3.74×; Illegal history truncation, 5.16×; Overflow under extreme gate ranges, 3.57×; KDA + TIRx, released, 2.96×.

Six gates every candidate must pass

  1. 01Fresh seeds and hidden distribution holdouts
  2. 02A strict specification of every legal input
  3. 03CUDA Graph replay with new inputs
  4. 04Stress probes for underflow and overflow
  5. 05Element-wise error bounds, no percentages
  6. 06Real end-to-end Kimi-Linear traces
How each gate works →

03How the loop works

Research, a contract, then small verified steps.#

  1. 01 / 04

    Read the code, the profiler and prior art first.

    The plan starts from ncu reports, a kernel wiki of known techniques, and the repository.

  2. 02 / 04

    Write down the objective and the bar to clear.

    The objective, constraints, validation command and promotion bar are fixed before any code is written.

  3. 03 / 04

    Validate and benchmark each iteration.

    Incorrect candidates stop at validation. Only a candidate that clears the bar is promoted.

  4. 04 / 04

    Log every candidate, result and decision.

    Benchmarks, profiles and decisions are logged so another engineer can check the run.

04Speedups

Ahead of the best human entries on all three MLSys 2026 tracks.#

In May, two of KDA's five contest kernels were still slower than the baseline. With CuteDSL, a profiling skill and a better flow, the same model passed the best human entries on all three tracks.

B200, speedup over the FlashInfer baseline; each track has its own scale. From the public release's table: geomean over the official workloads, timed as CUPTI kernel spans, not as the contest scored. Margins are geomeans of per-workload ratios over the best human entry.

05KDA²

Kimi Delta Attention: from 1.61× to 2.96× over FlashKDA.#

Eight weeks of results on B300. Scroll to play the chart, or point at any date.

1.5×2.0×2.5×3.0×3.5×6/167/217/308/108/148/258/309/19/29/39/49/69/123.28×3.57×3.74×refusedHumanizeTIRxTIRxCAKETIRx
KDA bestCAKEINT21 referencerefused candidates

Geomean over six 8,192-token workloads. Candidates at 3.28×, 3.57× and 3.74× overfitted the tests and were refused; see the KDA² post.

06Upstream

Kernels merged into SGLang by its maintainers.#

Each pull request reports the kernel speedup and the end-to-end effect on the model.

40+operators by June 19sgl-project/sglang · mainflash-kmeans

Merged Sep 2 · SM120

#36865

NVFP4 GEMM for Qwen3.x

The pull request ↗
The kernel alone1.319×
Output throughput1.087×

Kernel geomean over 16 production shapes; Qwen3.5-4B output throughput +8.73% at concurrency 1.

flash-kmeans#23 · reverted A KDA backend for Flash-KMeans (7.31× on LongCat-Video) was merged in August and reverted the next day. It is not counted here.

08Why it is here

KDA tests our flows on benchmarks we do not run.

KDA runs on our flows: its first runs used RLCR Flow, now the humanize1 flow, and KDA² runs on Humanize flows. Results come from outside benchmarks and from pull requests that upstream maintainers merged.

09Try it

Try the early prototype and send us feedback.

git clone --recurse-submodules https://github.com/NVlabs/kda.git

Each task brings its own evaluator, data and profiling tools. The repository has the agent flow, prompts and skills. Kernels people want next are in the KDA wishlist: request one as a pull request to its wishlist branch.