Inference clusters are about to grow by orders of magnitude, and the interconnect race has barely started

The standard mental model of an AI inference cluster is a small, tightly coupled group of chips. Gavin Uberti thinks that model is already obsolete. In his telling, the inference side of AI compute is on the verge of moving from configurations of around 8 chips, or the NVL72 scaleup domain that has become a common reference point, to thousands or tens of thousands of chips, and it will happen quickly.

That is not a routine capacity expansion. Moving from single-digit chip configurations to clusters measured in thousands changes almost everything about how inference infrastructure is designed, procured, and operated. The networking substrate, the cooling architecture, the software that schedules work across the fabric, the physical form of the rack itself: all of it gets rethought at that scale.

The interconnect layer is where Uberti’s argument gets most specific. He contends that chip-to-chip latency has enormous headroom for improvement, describing physical limits that would permit reduction to a figure he characterizes as “a handful, like 2.” The synthesis of his argument places current latency at around 4,000 nanoseconds and the theoretical floor at roughly 2 nanoseconds. The directional claim is clear: the gap between current practice and what physics permits is large, and closing even part of it would meaningfully change the performance characteristics of thousand-chip deployments, making the architectural leap more attractive rather than merely more expensive.

The inference side, you know, today people usually think about it as an 8 chip cluster or maybe just, you know, NVL72 uh as the scaleup domain, but very quickly this is going to become thousands of chips and tens of thousands of chips. Gavin Uberti

The demand side of the equation is moving in parallel. Clay Bavor reports that top engineers who are leaning into coding agents are spending more than $100,000 per year on tokens. That figure comes from a small population of early adopters, not from the median engineering team, and it should be read as a leading indicator rather than a sector-wide baseline. But leading indicators at that spending level are worth taking seriously. The engineers running up those token bills are also reporting productivity gains of between 3 and 20 times in features shipped. A range that wide reflects genuine variance in how different teams use the tools, but even the lower end of that range, if it holds as adoption broadens, represents a meaningful shift in what engineering organizations can attempt with fixed headcount.

The connection between those two signals is straightforward. If productivity gains at the high end of Bavor’s range become common rather than exceptional, organizations will increase their AI usage substantially. More usage means more inference demand. More inference demand means pressure to expand cluster scale. The architectural shift Uberti describes is not merely a supply-side story about what chip vendors want to sell. It is also a demand-side story about where usage curves are pointing.

What makes the near-term picture uncertain is the gap between the direction of travel and the execution complexity of actually building and operating inference clusters at that scale. Scaling from 8 chips to tens of thousands is not a linear engineering problem. Interconnect latency, memory bandwidth, fault tolerance, and scheduling all behave differently at orders-of-magnitude larger configurations. The physical limits Uberti cites on latency describe what is theoretically achievable, not what is currently deployed. The distance between the theoretical floor and current practice is exactly where the competition for inference infrastructure will be fought over the next several years.

Whether the transition happens on the timeline Uberti implies is an open question. What the evidence does not leave much room to dispute is the direction. Demand from high-intensity users is already large and growing. The hardware community is designing toward cluster sizes that match what Uberti anticipates. And the physical argument that interconnects have substantial room to improve is grounded in limits that do not move. The 8-chip cluster is not the permanent unit of inference compute. The question is how quickly the industry builds, and operates, what comes next.