Can a Profiler Judge AI-Generated CUDA Kernels Without Cheating?
We pointed Fournex at SakanaAI's archive of about 30,000 AI-generated CUDA kernels to ask a harder question: can our analyzer produce useful, evidence-backed judgments on messy generated code without ever seeing a human label?
The wrong way to use this dataset
The AI-CUDA-Engineer-Archive is a tempting thing for a GPU tooling company. It contains about 30,000 kernels across KernelBench tasks, each with CUDA source, a PyTorch reference, correctness flags, runtimes, speedups, and Nsight Compute profiles. The obvious move is to bolt on "Sakana dataset support" and call it a feature.
We think that is the weak version. The dataset was generated by another AI system, it is centered on PyTorch-module-to-CUDA tasks, and the published discussion around it raised real correctness and evaluation concerns. Treating speedup as success would be a mistake.
So we used it as an independent test of our own diagnostic quality. The command runs fully offline, needs no GPU, and for every kernel does what a human reviewer would: read the profile, read the source, reconcile them, and produce an evidence-backed verdict. Then we score that verdict against the kernel's real outcome.
frx eval sakana --sample 100 --out reports/sakana_eval.jsonWe are testing Fournex, not Sakana.
What the profiles actually contain
Before trusting any result, we inspected real cells from the live dataset instead of relying on the dataset card. Three findings shaped the design.
Fixed, partial NCU slice
Every NCU_Profile is a Speed-of-Light plus Occupancy section set: occupancy, block limits, SM busy, memory busy, cache hit rates, and a small number of divergence stats.
The byte/second trap
Memory Throughput is bytes per second, not percent of peak. The adapter refuses to put values into percentage fields unless the unit is literally percent.
Correctness is mostly invisible
Broken kernels can compile, run, profile cleanly, and produce numerically wrong output. A wrong kernel can profile exactly like a right one.
Consistently missing evidence
- No warp-stall breakdown, so memory stalls versus barrier stalls cannot be confirmed.
- No sectors-per-request, so coalescing cannot be measured from the profile.
- No tensor-core utilization, so tensor-core bottlenecks cannot be confirmed.
How the harness scores itself honestly
The dataset has no ground-truth bottleneck labels. We refuse to invent an accuracy number out of thin air, so the harness separates objective measurement from truth labels and circular heuristics.
Objective metrics are measured against the dataset's real Correct, Error, and speedup fields.
A 52-row hand-labeled gold set is the only source of accuracy versus truth.
The NCU-only weak label is reported strictly for self-consistency and marked as circular.
Every leaderboard line carries a basis tag: objective, heuristic, or vs-truth.
Results
On a curated, offline 102-kernel subset, 78 correct kernels had profiles and 24 were broken. The results we care about most are not just the high-coverage numbers. They are the places where the tool avoids pretending to know more than the evidence allows.
OBJECTIVE
concrete diagnosis coverage : 53.9%
confidence ceiling respected : 100.0% (cap = medium-high; no warp-stall data to justify more)
max confidence emitted : medium
slow kernels explained : 52.4% (n=82)
fast kernels not over-claimed : 100.0% (n=20)
CORRECTNESS (never inferred from the profile)
warned on build/runtime errors: 100.0% (n=11)
silent numerical mismatches : 0.0% (n=13) <- documented blind spot
warnings on correct kernels : 6.4% (n=78)
GOLD (52/52 hand-labeled rows present)
primary bottleneck accuracy : 100.0% (n=30)
confidence ceiling respected : 100.0% (n=30)
correctness-warning accuracy : 100.0% (n=36)Confidence ceiling respected
100%
Max confidence emitted
medium
Silent numerical mismatches
0% documented blind spot
It explains the slow kernels
The archive is full of naive matmuls running at 0.02-0.25x of cuBLAS. Even without stall sampling, reconciliation plus roofline evidence can flag memory-bandwidth saturation at medium confidence.
It does not over-claim on the good kernels
Of 20 kernels that beat PyTorch, zero were tagged with a high-severity, high-confidence bottleneck. A fast elementwise ReLU can still be memory-bound by nature, and Fournex says so carefully.
It catches the failures it can catch
Build and runtime failures get correctness warnings. Silent numerical mismatches do not, and the harness reports that zero percent as a known limitation instead of hiding it.
Why the ceiling matters more than coverage
Confidence is not a vibe. It is defined by how many independent analysis layers agree. Because this dataset cannot supply stall, coalescing, or tensor evidence, the ceiling is structurally pinned at medium-high, and in practice Fournex tops out at medium.
| Confidence | Meaning |
|---|---|
| medium | 1 layer confirms, and it is the only layer available |
| medium-high | 2 layers confirm, a 3rd is still available |
| high | 2 of exactly 2 available layers confirm |
| confirmed | 3+ layers confirm |
That is not a weakness in the tool. It is the tool correctly refusing to manufacture certainty the data does not support.
The honest scorecard
- 8/10 as independent validation that our diagnostics hold up on real, messy, generated kernels.
- Not a moat. The dataset is generated by another system and narrow in scope; we will not overfit to it.
- Correctness stays first-class. Speedup is never treated as success, and undetectable failures are labeled as such.
The harness ships with the kernels and gold labels packaged in the wheel, a dependency-free refresh script, and a verified clean-install path. Point it at the data and it will tell you, in its own evidence, exactly how much it actually knows and where it does not.
frx eval sakana --out reports/sakana_eval.jsonNo GPU required. No human labels required. No fake certainty included.