Inference cluster scale is rapidly expanding from single-digit chip configurations to thousands or tens of thousands of chips.

AI labs are already training models across multiple data centers, treating entire data centers as a single machine.

“Some of my friends at labs right now are training models across data centers. It's not even across nodes anymore. It's not even racks.”

Google has already made a deal to sell approximately 20% of its TPUs to Anthropic.

“Google already made a deal to sell like 20% of their TPUs to anthropic.”

A trillion dollars of capex has already been committed for the next year across major cloud and AI players.

“There's a trillion dollars of capex that has been committed for the next one year across all these people broadly speaking.”

vLLM is currently running on half a million GPUs at any moment.

“The open- source inference engine now running on half a million GPUs at any moment.”

Inference providers like Fireworks will become far more capex-intensive as they vertically integrate into owning data centers.

“These are going to become way more capex intensive businesses.”

Token count processed per day will increase 20x to 100x by the end of next year.

“Anywhere ranging from 20 to 100x could be possible.”

GPUs cannot effectively use tensor parallelism beyond 4-8 chips due to communication overhead from inability to overlap communication and computation.

“One of the limits of GPUs is because they don't effectively overlap communication and they have a hard time using tensor parallelism beyond four or eight.”