What is Transformer?
Transformer
The Transformer is a neural network architecture introduced in 2017 that has become foundational to modern AI, particularly for large language models. Its impact spans model design, company formation, and hardware debates.
How it developed
- Feb 2026 - Bryan Catanzaro said that in 2017, when the Transformer was invented, some companies asserted it could be made without Nvidia, with statements on official blog posts that were later redacted.
- Mar 2026 - Robert Lange said that a Rubicon moment would be when a new Transformer-like architecture or something massive is discovered by AI and widely adopted.
- May 2026 - Shiv Rao noted that his company started three months after the Transformer paper, and he categorizes AI-native companies into post-Transformer pre-LLM, post-LLM pre-agent, and post-agent variants.
- May 2026 - Eric Jang said that for small data regimes, ResNets still outperform Transformers and offer better value at lower budgets.
In the evidence
Every line below is attributed to a named speaker.
Robert Lange on the key inflection point for AI-driven science.
“I think one of the Rubicon moments is when the new Transformers architecture or something massive is discovered by AI and we're all using it.”Robert Lange · 13 Mar 2026
Shiv Rao frames AI native companies across three historical vintages: post-Transformer paper pre-LLM, post-LLM pre-agent, and post-agent, each representing a distinct strategic starting point.
“There's like three variants of like AI native company, maybe. There's like we started 3 months after the Transformer paper. There's like in my head, there's like a post-Transformer paper pre-LLM company. There's post-LLM pre-agent company. There's post-agent company.”Shiv Rao · 16 May 2026
ResNets still outperform transformers in small-data regimes, delivering more performance per dollar at lower budgets.
“For small data regimes, my experience is that ResNets still outperform transformers and give you more bang for the buck at lower budgets.”Eric Jang · 15 May 2026
In 2017, unnamed companies published official blog posts claiming transformers could be built without NVIDIA, statements that were later quietly redacted.
“There was some assertions by some other companies back in 2017 when the transformer was invented that they could only be made without Nvidia. in fact, there were statements to that effect on official blog posts which I think got redacted later on.”Bryan Catanzaro · 4 Feb 2026