Evolution of AI

published 2025-02-26

The rapid evolution of AI makes me change my mind often regarding its future evolution. However, since the beginning of the deep learning era, I hold a belief that in the long run the distinction between "training" and "inference" will fade and AI will learn online [1].

I think economics make it unlikely that the current practice of collecting data then training a model on it and doing it all over again for the next iteration will subside. I find it more likely that at some point all of the following will happen:

• AI systems will no longer be pure deep learning models but more complex systems including memory. This is already happening [2], and I expect it to continue. Parameter count will continue to increase and require sparsity at inference, similar to MoE [3].
• Large-scale systems will keep a context per user or entity they interact with. It will be possible to access several of those contexts at runtime. I am not entirely sure of the shape this will take but there are already various options to save context in LLMs, serve concurrent LoRAs [4], and more exotic approaches [5].
• New model generations will not be trained from scratch. We will develop approaches to transfer knowledge from one generation to the next, even if they have radically different architectures. This could include things like merging [6] and knowledge distillation as pre-training. There could be several teachers, i.e. a generalist model could be fine-tuned to learn from a smaller specialist teacher.
• With usage increasing, user interactions will become an essential — maybe eventually the primary — source of training data. It will be integrated into the model in "soft real-time" through some form of online RLHF [7]. This pairs nicely with the previous point, which means this online learning can be transferred.
• Much smaller task-specific models will be distilled from those large generalist models for embedded use in applications and on-device.

This vision is largely coherent with Google's Pathways [8], and I suppose other large players are seeing the same thing.

1: https://en.wikipedia.org/wiki/Online_algorithm

2: https://research.ibm.com/blog/memory-augmented-LLMs

3: https://huggingface.co/blog/moe

4: https://arxiv.org/abs/2311.03285

5: https://arxiv.org/abs/2209.07663

6: https://sakana.ai/evolutionary-model-merge/

7: https://medium.com/@bavalpreetsinghh/rlhf-ppo-vs-dpo-26b1438cf22b

8: https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/