Transformer^2: Self-Adaptive LLMs

Does anyone else find their results don't match their claims? In many cases the base model or a simple LoRa beats their proposed method. The few times theirs wins, the difference is very small. I feel like some of these "wins" are more sampling error than any significant improvement.

I'm always happy to see publishing of negative results, but it seems like they are selling what are negative results as positive results.

This sounds like MoE and maybe a bit of chain-of-thought. Curious what someone with more domain expertise thinks about this

If they can test against Llama 70B and Mistral 7B, they ought to compare against Mistral 8x7b imho

Great research here. Contextual real-time weight modification is definitely one of the breakthroughs required for AGI. Why create a LoRA when you can generate one on the fly suited to the task at hand?

One weakness of this method is the storage of decomposed UV from W. My linear algebra is rusty, but it seems required if you want to scale in that U projected subspace, hence double your weight memory footprint (that has been said, U / V should be easier to quantize from information theory perspective). I also think MoE is more principled if you want to have experts activations. But I understand that Sakana's research focus mostly is about adapting existing pretrained models, not to do it from scratch.

> Transformer² represents a significant milestone in the evolution of AI systems.

Coming from a math background, it always amazes me to see how people in AI/ML brag about their papers. If someone wrote:

> My paper represents a significant milestone in the evolution of algebraic geometry/ergodic theory/combinatorics

it would be a laughing stock for the math community.

The code: https://github.com/SakanaAI/self-adaptive-llms

Obvious next step: use this kind of model in AI-Scientist, Sakana AI's AI-powered automated researcher project.

Can someone please enlighten me how this is any different from Mixture of Experts? Because I don't see any difference at all.

It is discomforting to read, in the first paragraph, that "dynamical adjustment of weights" is justified as "adaptation". Clearly it is a sought milestone to have «a future where AI models are no longer static»: but the chief reason remains, "intelligent systems reprocesses their body of knowledge and change it to improve it" - it is anterior to "adaptation to environment", it is "maintenance of the body of knowledge (of the world model)": it is the continuous practice of "thinking about things", "pondering", "reflecting", "using judgement"...

There is not just a simple «lifelong learning»: the whole past experience is still productive, requiring analysis, not "solved".

Anyway: the directions seem good.

Edit: equally interesting in another direction is the automated analysis of the internal subagents, «break[ing] down the vast, complex knowledge stored in the LLM into smaller, meaningful, and independent pieces (e.g., the different pathways or components for math, language understanding, etc)». Should not there be a general study of the dissection of systems with seemingly emergent intelligence, doing on LLMs like we do on C. Elegans?

Worth noting is that the original inventor of the transformer is part of this team

> https://sakana.ai/

I like that background animation. Seems like there's an opportunity for tiny logic gates and some punny swarm behavior.

Is this real? Or is this a hustler type paper/company.

It's all very interesting but those pictures look pretty bad. Clear visible artifacts, awful shapes.

The ideas in the paper have been implemented and tested. The authors conducted experiments on several tasks (math, coding, reasoning, and visual question answering) and showed that their approach works better than previous methods like LoRA.

Key ideas (in simple terms):

1. What’s the problem?

    - Fine-tuning LLMs for every new task is slow, expensive, and often doesn't generalize well.
    - Models trained on one task may perform poorly on others, especially unseen ones.
    - Current methods (like LoRA) can add new capabilities but aren't efficient enough.

    - Transformer² uses a new fine-tuning method called Singular Value Fine-tuning (SVF). This focuses on adjusting only certain parts of the model’s "weight matrices" rather than changing everything.
    - By tweaking specific components (called "singular values"), it trains smaller, efficient "expert" modules that specialize in particular types of tasks.

    - Training phase: Train these smaller expert modules offline using reinforcement learning (RL) to specialize in tasks like coding, math, or reasoning.
    - Inference phase: When a new input is given, the system analyzes the task (e.g., “Is this a math or coding problem?”) in the first pass. Based on this, it combines the right expert modules and adapts the model’s behavior in the second pass.

    - Prompt-based: Use a cleverly designed text prompt to figure out the task type and pick the right expert module.
    - Classifier-based: Train a separate model to classify tasks and match them to experts.
    - Few-shot adaptation: Look at a small number of examples (few-shot learning) to dynamically combine expert modules for the best results.

    - The system uses fewer parameters than traditional fine-tuning methods like LoRA.
    - Adaptation works even on small datasets without overfitting or forgetting older tasks.