Retire the Abstractions

TL;DR: We think CUDA DSLs are heading toward retirement

Last year, we set out to write megakernels. But implementing megakernels can be quite miserable. It requires complex data structures, extensive inter-thread/SM/GPU synchronization, deeply nested control flow, etc. We couldn't hold it in our heads, so we did what computer science has done for 70 years: we built a layer of abstraction. Even with it, we had to fight through race conditions and deadlocks for a couple months before the Llamas ran blazingly fast.

This year, we built an MoE megakernel, but we deleted the abstraction. With agents, we could work through the complexity directly and build target-optimized code from scratch. No intermediate layers of C++ abstraction were needed.

So what could this mean?

Offloading the offloader

We've noticed an interesting pattern over the past quarter. The tasks we could already do without abstractions (e.g., writing an optimized GEMM kernel) are now nearly automated given the right prompt. For now, we still have to tell the agent which PTX instructions to use, the warp-specialization design, and so on, but it gets to the state-of-the-art pretty darn quickly.

The tasks we needed abstractions for (e.g., writing a megakernel) haven't been automated, unfortunately; we certainly can't one-shot a megakernel today. But agents let us put the abstraction in the prompt, in an incomplete, messy form, rather than in carefully designed C++ templates. Complexity that was unmanageable last year suddenly became manageable, because we now have a “compiler” that can take vague instructions and produce code. The ideas that went into the megakernel still mattered just as much, but it was much easier to express.

The job of abstraction as a cognitive offloader is starting to retire. Agents are taking that job.

What next

By inductive reasoning, CUDA DSLs are next on the retirement list, including our beloved ThunderKittens.¹ Probably next year, or maybe sooner. We don't know how far down this will go. Somewhere below us is a floor where an abstraction is purely the contract, rather than an offloader, and it stops. But we can't see it from here.

Which leads to the provocation:

A codebase's pitch is precision. It's the one artifact that isn't ambiguous, the thing a particular machine will execute the same way twice. But precision has a cost, and the cost is brittleness. A codebase is tied to a language, a framework, a hardware target, and a set of conventions only the team that wrote it really understands. It appears portable, but barely.

A prompt is the opposite. It's fuzzy, but it travels. Hand the same intent to a different worker, human or machine, and if that worker is smart enough to fill in the gaps correctly, you get a correct result without ever having standardized the gaps in advance.

That's the interesting part. With agents, there's now a competition between an intelligent executor reading underspecified instructions and doing the right thing, versus specifying every gap so a dumb executor doesn't need judgment. DSLs and frameworks are that specification, expressed as a codebase. If the executor stops being dumb, the DSL loses its ground.

Codebase is an imperfect medium. We love it, but it's huge, baroque, and full of bugs. Not because the engineers who wrote it are careless, but because:

• precise enough for a machine,
• readable enough for a human, and
• flexible enough for the future

is a nearly impossible set of constraints for a string of tokens to satisfy.

And we've accepted opaque transformation before. Few write the assembly that actually runs. The compiler reorders, inlines, vectorizes, and changes what executes in ways no one on the team could point to. We took that deal because the compiler was right more often than we were. Intelligence already sits between intent and machine. The question is whether letting it sit higher is a difference in degree or in kind, and we don't think it's in kind.

Eligibility

Admittedly, we've been a little glib about an abstraction's job. There's more to consider before letting one go.

First, an abstraction isn't just a way to manage cognitive load; it's a shared surface where application, reuse, and review attach. Without it, you get a Cambrian explosion of verification challenges. Ten teams on ThunderKittens test the same tile semantics, and that compounds over time. Ten teams generating bespoke megakernels have ten disjoint sets of problems, and they don't stratify.

Second, there is no shared oracle. For a megakernel, the tests are the contract and the work. Who writes them, and what checks those? When we deleted the framework, we kept the oracle: reference implementations, numerics tolerances, our intuition of what the profiled Gantt chart should look like, etc. That is what the retiring abstraction left behind. We can only retire a layer once we have an oracle that outlives it. In domains where nobody knows what the oracle is, none of this applies, and we'd keep the scaffolding.

Lastly, we're a single, biased sample. We deleted abstractions in a domain we knew deeply. Abstractions also exist to transmit knowledge to people who don't have it yet, and we can't say whether an agent that fills the gaps for us fills them for someone hearing about warp specialization for the first time. Maybe agents make onboarding layers obsolete. Or maybe humans still need them, and we've just described a comfortable position for people who are already trained.

The pension

Regardless, the transformation seems inevitable, and we may have to let our beloved frameworks go.

Then what do we keep? The intent, the invariants, the tests, and the hard-earned domain knowledge that currently lives inside ThunderKittens's abstractions and keeps them correct on hardware. The library may go. But that knowledge doesn't evaporate just because the CUDA or HIP or whatever underneath gets regenerated instead of hand-tuned.

And what changes? Trust moves up a level. We scrutinize the spec and the oracle, not the diff. The implementation becomes disposable: a cache of one particular compilation, not the source of truth. Codebases won't disappear. They just stop being what we mean when we say "the project."

Abstractions retire. Ideas stay.