Most computer vision systems learn from fixed datasets and then process new inputs passively. Monty starts from a different premise: perception is a sensorimotor process. An agent learns by moving its sensors, associating features with locations, and building object models in explicit reference frames.
I joined the Thousand Brains Project in 2024 and work on Monty, the first implementation of a thousand-brains system. For the 2025 paper, I contributed experiments that demonstrated Monty’s capabilities and compared its learning behavior with Vision Transformer (ViT) baselines. The evaluation focused on recognizing 3D objects from the YCB dataset while also estimating their pose.
How Monty works
Monty is organized around three kinds of components:
- Sensor modules transform raw observations into messages containing a 3D pose and features.
- Learning modules build object models and use incoming sensations and movements to update hypotheses about an object’s identity and pose.
- The motor system selects actions, including movements suggested by a learning module’s internal model.
These components communicate through the Cortical Messaging Protocol (CMP). CMP gives the system a shared format for feed-forward sensory information, goal-directed movement, and votes between learning modules.
During learning, Monty moves over an object and binds sensed features to locations in the object’s reference frame. During inference, each new sensation and movement updates a set of possible object and pose hypotheses. The next action changes what the system senses, so inference unfolds as a loop between perception and movement rather than a single classification step.
What the paper examined
The experiments tested several consequences of this design. The paper reports that Monty’s structured object models support generalization to unseen rotations and noisy features. The same representations expose object symmetries, since more than one pose can be consistent with the sensations gathered from a symmetric object.
The paper also evaluates how movement policies and communication between learning modules affect inference. Model-free policies help the agent remain on and explore an object, while model-based policies use current hypotheses to move toward locations that could distinguish ambiguous objects. With multiple learning modules, voting can reduce the number of movements needed to reach a consensus. Finally, comparisons with ViT baselines examine rapid, continual, and computationally efficient learning.
Current scope
Monty is still at an early stage. These results come from a controlled 3D object recognition and pose-estimation setting, not from a general-purpose intelligent system. The paper also notes that inference cost currently grows with the number of known object models. Scaling the same principles to richer environments and tasks remains an open research problem.