Contract-grade verifier with 12 adversarial gates finds 39.5% of LLM-generated GPU kernels broken
A new paper describes a contract-grade verifier with twelve adversarial gates for testing LLM-generated GPU kernels, addressing weaknesses in current loose correctness checks. The verifier audited 2,638 machine-generated kernels that a public system's own harness had already accepted as correct, finding 39.5% broken beyond any tolerance argument and 62.1% carrying at least one failure.
Coverage timeline
Hacker NewsJimmc414
View PDFHTML (experimental) > Abstract:Systems that generate GPU kernels with language models report high correctness rates. Those rates come from a single loose test: run the kernel on a few random inputs at one fixed shape and accept it if the output is close to a reference. A kernel can pass that test and still be silently wrong. It can return an ordinary number where the true answer is a NaN or an infinity, differ from run to run, break when the shape changes, or accumulate in fp16 where the reference keeps an fp32 total. We build the instrument that checks correctness properly: a contract-grade verifier of twelve adversarial gates, each a property a correct kernel must satisfy, several of them tolerance-free, so no choice of threshold can explain a failure away. Aimed outward, the verifier audits 2,638 machine-generated kernels that a public system's own harness had already accepted as correct. It finds 39.5% broken beyond any tolerance argument and 62.1% carrying at least one vio