The Core Architecture And Data Flow Of VLA

I summarized the core architecture and data flow diagram of VLA. VLA is essentially this: multimodal sensors pass through an encoder to produce an f-world, which is then fed to the agent and the policy decoder. The agent reasons and produces a condition vector; based on the f-world and the condition vector, the policy decoder precisely executes the agent’s intent under the current environment state (action parameters down to exact numbers). The agent then periodically receives incremental f-world updates from the encoder at short intervals, and receives interrupt signals from the policy decoder on an event-driven basis. In the former case, the agent may produce a new condition vector for the policy decoder to use in its next iteration, or it may temporarily produce none; in the latter case, it must produce a new condition vector so that the policy decoder can continue executing with it in the next iteration (a restart). Every f-world is tagged with a timestamp and an ID; the creation timestamp of every condition vector is also saved into the agent’s context, along with the f-world it was based on (base-on), its own newly assigned unique ID, and whether it is currently active. The agent’s context mechanism can be finely designed and managed: whether to let the agent see, how much to let the agent see, and when to let the agent see — every historical f-world and every historical condition vector together with its related information.

VLA is a highly engineering-intensive system, not just a simple algorithmic model. Engineering concepts such as asynchronous state machines, asynchronous communication, and timing management all need to be sorted out.

The policy decoder is basically done with flow matching these days.

For the encoder, 3D Gaussians are currently the best choice — high efficiency and the easiest to train — especially feed-forward 3D Gaussians, which are the current world frontier: high quality and, crucially, extremely fast. Fei-Fei Li’s team is working on exactly this. It is far faster than point clouds, meshes, and similar methods; it is their modern replacement.

All of this is knowledge I stole from Dr. Deng, hahaha.

Deng Kan is truly impressive. He and Andrew Ng were fellow students under the same advisor — Andrew Ng was his junior — and they were good classmates at CMU. Deng and Anthropic’s boss Dario were colleagues at Baidu; Deng said Dario was talkative even back then. Deng and Andrew Ng were each heading a department at Baidu at the time, and Dario was in Andrew Ng’s department. Deng’s advisor was the dean of CMU’s School of Computer Science, and Hinton and LeCun were both good friends of their lab — I believe it was Hinton, or some Turing Award winner, who once showed up at their lab in slippers for an academic exchange.