Transformers lose to ResNets at small scale, but the gap closes fast as compute grows
The dominant architecture in AI does not dominate everywhere. At small data budgets and in constrained domains like Go, older network designs hold their ground. The more unsettling finding is what that implies about scale as the real variable.
The transformer’s grip on AI research looks less absolute once the data gets thin. Eric Jang, a robotics researcher, reports directly from his own work: in small data regimes, ResNets still outperform transformers and deliver more performance per dollar at lower budgets. That finding cuts against the default assumption that the architecture powering large language models is simply better, full stop.
Jang’s experience with Go sharpens the point. He has, by his own account, been unable to find a way to make transformers beat ResNets for the game. That matters because Go is a domain where the training signal is dense and the rules are exact. It is not a case of messy real-world data overwhelming a newer model. The constraint is structural: game-tree search and local pattern recognition may simply favor convolutional and residual designs in ways that scale alone does not erase. Recent work accepted at the International Joint Conference on Artificial Intelligence lends some public context here. Researchers developing a hybrid architecture called ResTNet, which interleaves residual and transformer blocks, found meaningful win-rate gains in both nine-by-nine and nineteen-by-nineteen Go over either design used alone, suggesting the two approaches address different things and that neither cleanly dominates the other in this setting.
Ramin Hasani, whose research has focused on liquid neural networks, reaches a similar conclusion from a different starting point. At smaller scale, he argues, transformers cannot beat other formats. Scale is the condition under which transformers win, not some intrinsic property of the architecture itself.
For small data regimes, my experience is that ResNets still outperform transformers and give you more bang for the buck at lower budgets. Eric Jang
What makes Jang’s full account worth sitting with is that he holds both positions without contradiction. Architecture choices, he says, do not matter that much at current graphics processing unit speeds, where model sizes are large enough that the difference between transformer and ResNet becomes secondary. The transformer’s apparent supremacy at the frontier may therefore reflect the conditions of the frontier more than the architecture’s inherent qualities. Push resources down and the hierarchy reshuffles.
Ryan Greenblatt supplies a figure that reinforces this reading from a different angle. Training a model on the compute budget that produced GPT-3, using today’s methods and hardware, would yield something a moderate amount better than GPT-4. That is a striking compression: hardware efficiency and training improvements have advanced far enough that yesterday’s budget now buys what was, not long ago, a frontier result. If that trajectory holds, the practical question for most builders shifts away from which architecture to use and toward where on the compute curve a given project actually sits.
Neil Patil raises a wrinkle the mainstream discussion tends to skip. The compute market, he notes, has gotten what he calls the large language model pill. Infrastructure, memory hierarchies, and chip optimization have been built around the demands of large language models. Domains with meaningfully different compute and memory profiles, including structural biology, operate on hardware that was not designed with their workloads in mind. The architecture debate is partly a proxy for a deeper infrastructure question: which scientific and engineering domains get hardware that fits their problems, and which inherit someone else’s stack.
The picture that emerges is more conditional than the headline narrative allows. Transformers at scale, trained on abundant data, on hardware optimized for them, are genuinely formidable. Strip away any one of those conditions and the advantage narrows or inverts. That is not an argument against transformers. It is an argument for being precise about what the evidence actually shows, which is that architecture is one variable among several, and at the low end of the compute and data spectrum, it may not be the most important one.