What is Mamba?

Liquid AI's AFMD automated architecture search produced the double gated convolution as the optimal efficient architecture, eliminating hand-tuned gating from models like Mamba and gated delta nets.

“Turns out all of this has to go away if you want to get to the most efficient form of format of architecture and it became the double gated convolution that actually came out of this massive search space like AFMD.”

Hand-tuned gating mechanisms used in popular efficient architectures like Mamba are architecturally unnecessary; automated search eliminates them entirely in favor of a simpler double gated convolution.

“Turns out all of this has to go away if you want to get to the most efficient form of format of architecture and it became the double gated convolution that actually came out of this massive search space like AFMD.”

Ramin Hasani on the architectural lesson from their massive search: all hand-tuned gating mechanisms are unnecessary.

“Turns out all of this has to go away if you want to get to the most efficient form of format of architecture and it became the double gated convolution that actually came out of this massive search space like AFMD.”

Transformer performance is largely a function of scale, not architecture. Mamba, a state-space model, achieved comparable results to transformers simply by scaling up.

“Mamba which is a state which is a traditional state space model it's basically a common filter but like on steroids they scaled it way up and yet and now it's you know got they've you know mistral has their very nice like coding agent and it works pretty darn well, right? They got a lot of the same functionality with a completely with a you know a completely different architecture simply by virtue of scaling.”

Mamba (backed by Mistral) warrants watching as a state-space alternative to transformers that has achieved strong coding-agent performance purely through scaling, not architectural novelty.

“Mamba which is a state which is a traditional state space model it's basically a common filter but like on steroids they scaled it way up and yet and now it's you know got they've you know mistral has their very nice like coding agent and it works pretty darn well, right? They got a lot of the same functionality with a completely with a you know a completely different architecture simply by virtue of scaling.”

Nvidia Nemotron Ultra uses Mamba (hybrid) attention, positioning it architecturally differently from Chinese labs such as DeepSeek and MiMo that use sparse attention.

“In the Nvidia Ne Neumitron Ultra used the Mamba attention, whereas, you know, we see, you know, Deep Seek sparse attention and then the MiMo MSA, whatever that stands for, MiMo sparse attention.”