Nested learning (multiple MLP blocks at different update frequencies) outperforms standard transformers on sequential reasoning and multi-language in-context tasks by preventing catastrophic forgetting.

At small scale, transformers cannot beat other architectures such as liquid neural networks or RNNs.

“Transformer would be able to add scale to beat this but at a smaller scale transformers would not be able to beat any of the other formats.”

Replacing standard transformer attention with the nested learning architecture (multiple MLP blocks with different update frequencies) enables models to simultaneously learn and translate two unseen languages from in-context examples, a task standard transformers cannot handle.

“The point is the model cannot handle it context well and fully understand each of the languages separately and you know that's that's generally very hard challenge for transformer based and but the point is when we change that architecture to hope or hope attention again we have attention but we have multiple levels of in context learning multiple levels of ML blocks. And so one thing that we can see is that when we increase the number of levels, the performance of the model in both of these languages gets better and better.”

In-context learning in current AI systems amounts to a vague representation of one algorithm, essentially gradient descent in an unstructured way, akin to least squares.

“When our current AI systems do in context learning they learned a vague representation of one algorithm which is list square you know it's basically list the square you know so what they what they what they figured out is basically gradient descent in a mushy way.”