The problem
You are on a live construction site and need to find a bathroom. There are three portable toilets, all identical, but the one you want is beside your crew's white delivery van. Nothing about the toilet itself distinguishes it; only where it is does. For a robot, this request requires two capabilities: a compact memory of the entire 15,000 m² site, small enough to store and search onboard, and relational retrieval: finding this toilet because it is beside that van.
One query, three kinds of constraint
Find the blue portable toilet with ‘TOI TOI’ branding closest to the white DBE-labeled delivery van.
One target among everything else mapped
There may be dozens of objects with the same category and a similar appearance. Naming the category does not single one out.
A robot may accumulate thousands of views, exceeding the practical context available to a video-language model.
A VLM small enough to run onboard — say 8B — asked to reason directly over frames or scene graphs may fail on precise spatial relationships.
Instead of asking one large model to remember and reason over everything, FARM builds a structured memory and turns language into an executable search.
Interactive demo
Turn a real scene around and see what is stored in it. Every point, box and caption here is FARM's own output, exported from the runs the paper evaluates. Click any point in the cloud, and the memory names the object it belongs to. Or pick a benchmark query — its target lights up in gold, the anchors it is described against in teal. Drag to orbit, scroll to zoom.
Loading memory…
Drag to orbit · scroll to zoom · click any point
Nothing selected yet. Click any point in the 3D view — every point belongs to an object — or pick from the list.
Video of the Scene
Results
Relational grounding against object-centric scene-graph systems (BBQ, DAAAM) and frame-based VLM grounding (RynnBrain). Two questions get asked separately below: did the method put the right object first, and did it find the right object at all.
040%
010%
025%
080%
030%
050%
RynnBrain and DAAAM+R return a single prediction, so ranked recall is undefined for them.
…and it stays small enough to run on the robot
Stress-tested on the large scenes from FARM-Scenes — long indoor and outdoor sequences, from 1,800 to 15,000 m²:
Deployment
A quadrupedal robot with onboard RGB-D sensing and onboard compute. Memory construction and relational retrieval both run live, while the robot walks.
Full deployment run
Takeaway
The robot retains structured and compact relational spatial memory.
People can refer to objects naturally, through appearance and relationships.
The same framework works from individual rooms to large outdoor environments.
FARM builds relational spatial memory online, and finds the object your words refer to — from a single room to a 15,000 m² site.
Siming He
UC Berkeley
Leo Huang
UC Berkeley
Adam Lilja
Stanford University
Fabio Hübel
Stanford University
Jonas Frey
Stanford University
Marco Pavone
Stanford University
S. Shankar Sastry
UC Berkeley
Jitendra Malik
UC Berkeley
Claire Tomlin
UC Berkeley
@misc{he2026farmusingrelationalspatial,
title={FARM: Find Anything using Relational Spatial Memory},
author={Siming He and Leo Huang and Adam Lilja and Fabio Huebel and Jonas Frey and Marco Pavone and S. Shankar Sastry and Jitendra Malik and Claire Tomlin},
year={2026},
eprint={2606.15476},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2606.15476},
}