A Brief History Of Neural Network Architectures And The Future Of Agent Thinking Structures

【【English Subs】Foreign Creator Explores Whether JEPA Can Surpass LLMs — Bilibili】 https://b23.tv/pl5sRAt

I stumbled upon this video by chance and think it’s one of the highest quality ones I’ve ever watched. In just half an hour, it carries the entire history of neural networks [thumbs up]. If you plan to reproduce all of these works one by one, your understanding of neural networks will definitely deepen significantly!

On a related note about architecture design: neural network architecture design is a lot like assuming in advance what role each network is expected to play, and then training it to hopefully perform that predetermined role. I recently realized that the “oil” of the future may very well be Agent thinking-structure architecture design (or what you could call context protocol design). This kind of thinking-structure design also assumes that each cognitive module will reason in the expected way — strikingly similar in spirit. Building more sophisticated cognitive logic on top of the React paradigm: this area may very well blossom just as richly as neural network architectures!

I feel that most people are still fixated on the macro-level cycle of “Reasoning → Action → Observation” within React, and are massively building applications on top of it. But I think that once development hits a bottleneck, people will once again turn their energy toward the architecture of Agent thinking. At that point, the ceiling for Agents will be far higher than most React-based Agents today. React has already exposed its limitations on long-horizon tasks. I believe we may need to break a mental habit: models don’t necessarily have to follow the current set of Agent protocols and thinking patterns. Perhaps by designing thinking structures more effective than React, the upper bound of Agents will be much higher.