Scene Language Model

The scene map is just text. One vision-language model reads it, looks at the world, and rewrites it with ADD, EDIT, and REMOVE — no engineered mapping backend.

Adam Lilja1,2, Fabio Hübel3, Siming He4, Junsheng Fu2, Claire Tomlin4, Lars Hammarstrand1, Jitendra Malik4, Jonas Frey4, Marco Pavone3,5

1Chalmers University of Technology 2Zenseact 3Stanford University 4University of California, Berkeley 5NVIDIA

Paper arXiv Codecoming soon Datacoming soon BibTeX
Live 3D mapping on a held-out ScanNet scene: the camera view SceneLM reads (left) and the map it maintains (right).
scene.md — live map updates
6–12×
smaller scene memory than modular pipelines
0
human 3D labels used for training
1.27M
images auto-labeled across 10 sources
On‑device
runs on a Jetson AGX Thor onboard a Spot quadruped

The idea

Learning the mapping backend

A robot exploring a building needs a persistent memory of the objects it has seen — what they are, where they are, how to refer to them. Today that memory is produced by an engineered mapping backend: per-frame object proposals are projected to 3D and merged by hand-written association, tracking and fusion heuristics, while each object drags along embeddings, image crops, and masks.

SceneLM replaces this backend with a single model. The map is a structured text list — label, one-sentence description, 3D position. Each new posed RGB-D frame is fed to the VLM together with the part of the map visible from that pose, and the model writes back a short script of ADD, EDIT, and REMOVE operations. Association, fusion, and reconciliation all happen inside one autoregressive forward pass.

Engineered backend

Modular mapping pipelines

  • Per-frame proposals → 3D projection → hand-written data association & fusion
  • Memory: visual embeddings, image crops, segmentation masks, multi-view structures
  • 23 MiB (ScanNet) / 125 MiB (HM3D) per scene
  • Every new failure mode needs a new heuristic
Learned — SceneLM

One VLM closes the loop

  • Mapt = f(Mapt−1, framet) — one forward pass per keyframe
  • Memory: plain text only — label, description, 3D point
  • 3.5 MiB (ScanNet) / 10.8 MiB (HM3D) per scene
  • No tracker, no factor graph, no feature bank
SceneLM architecture: streaming images enter the trained model, which emits add, edit and remove operations; applying the edits yields the scene.md text representation, fed back as the prior scene at the next step.
Streaming posed images go in; a script of map operations comes out; the updated text map is the only state carried to the next frame.
Three panels showing the scene map as literal text, the model reading map plus frame, and the updated map with inline edits.
The map really is text. ① The scene state as the model sees it. ② A new frame arrives and the model answers with raw operations. ③ The updated map, edits applied inline.

Three primitives

ADD, EDIT, REMOVE

Every map update is a short script of three operations. Objects the model doesn’t mention are implicitly kept, so most update scripts are sparse — often empty. Localization is a single 2D anchor point in the image, lifted to 3D with depth and pose; the model never predicts 3D coordinates directly.

ADD example: a game controller is anchored in the camera view and placed into the 3D map.

ADD a new object: the model anchors it in the image, names it, describes it, and the anchor is lifted to a 3D position — here a game controller, mapped on a held-out test scene.

EDIT example: an object previously labeled book is relabeled after a better view.

EDIT an existing entry: a better view arrives and the model corrects the label, description or position of an object already in the map — here fixing a mislabeled book.

REMOVE example: a spurious trash can entry is deleted from the map.

REMOVE a wrong or duplicate entry: the model deletes map entries that the current view contradicts — here removing a spurious trash can.

EDIT and REMOVE are the load-bearing operations. Training with ADD alone drops retrieval accuracy by 8.8 points on ScanNet — persistent mapping is not obtained by repeatedly applying a strong per-frame model. And pretrained VLMs essentially cannot do this at all: out of the box they never remove a map entry, and they loop — repeating the same object endlessly — on 20–40% of images (finetuned: 0%).

Corrections, animated — scene0000_00

Six consecutive corrections on a held-out ScanNet scene: labels refined, positions moved, and false-positive entries dropped — each shown in the camera view, the 3D map, and the record itself.

Corrections, animated — scene0002_00

The same repair behavior in a second scene: EDIT fixes what a better view contradicts, REMOVE deletes what should never have been mapped.

Training data

OVAL: supervision without human labels

Nobody has ground truth for how maps should evolve. OVAL (Open-Vocabulary Automatic Labeling) manufactures it from plain single images: open-vocabulary foundation models propose and describe objects, a VLM quality filter passes or fails each one — and the rejects are kept as structured negatives that become REMOVE supervision. Corrupting each map entry independently turns single images into map-repair problems, teaching partial correction rather than all-or-nothing rewriting.

OVAL pipeline: an object proposer (sparse or dense), an object describer, and a VLM-as-judge quality check that keeps or rejects each annotation.
The OVAL automatic labeling pipeline. Sparse (open-vocabulary boxes) and Dense (promptable segmentation) proposers, mask-conditioned description, and a reason-first VLM quality check. No human labels anywhere.
1.27M
images from 10 sources — indoor scans, driving, robot-recorded
10.7M
object annotations (8.2 per image, OVAL-Sparse)
81%
pass the VLM quality check; the retained pool is 84% human-verified correct
100%
of supervision auto-generated — zero human 3D annotation

Results

Competitive with engineered pipelines, at a fraction of the memory

Evaluation covers the maps, not just the model: language-grounded 3D retrieval on the FARM benchmark (ReferIt3D/ScanNet and IRef-VLA/HM3D) and open-vocabulary 3D localization on the TagMap Matterport3D benchmark.

3D retrieval — ScanNet

Accuracy@1, FARM benchmark (30 largest ReferIt3D scenes)
BBQ
22.6
Qwen3-VL-4b
25.6
FARM
32.0
DAAAM
37.1
RynnBrain-30b
38.9
SceneLM-2b
38.5
SceneLM-4b
39.3
The FARM bar is the FARM system itself, evaluated on its own benchmark. A 2B-parameter SceneLM matches a 30B general VLM baseline; SceneLM-4b reaches Recall@10 79.2 vs FARM’s 75.0 and MRR 52.6 vs 45.7.

3D localization — Matterport3D

Precision@0.1 m / Recall@0.5 m, TagMap benchmark (macro over 21 ObjectNav classes)
TagMap · P
28.0
OpenMask3D · P
29.0
OpenScene · P
31.0
SceneLM-4b · P
43.2
OpenMask3D · R
19.0
OpenScene · R
29.0
TagMap · R
42.0
SceneLM-4b · R
66.4
Precision beats every baseline at every threshold; gains are largest at the strictest ones — genuinely better localization, not looser matching.
3.5 MiB
per-scene memory on ScanNet — vs 23 MiB for FARM’s multi-modal memory (6.6× smaller; 10.8 vs 125 MiB on HM3D, 11.6×)
3.5 updates/s
at 2B/NVFP4 on an RTX 5090; 0.86 updates/s on a Jetson Thor — enough to keep pace with a robot at 1.8 m/s (5090) or 0.4 m/s (Thor)
2B ≈ 30B
SceneLM-2b matches RynnBrain-30b on ScanNet retrieval — the bottleneck is supervision and formulation, not capacity
The tradeoff: a full VLM pass per keyframe costs more compute than specialized modules — 965 ms/frame vs FARM’s 275 ms on ScanNet (A100). SceneLM trades per-frame speed for a radically simpler system and a much smaller memory; NVFP4 quantization lifts the on-device update rate 1.6× (0.53 → 0.86 updates/s).
A complete multi-room HM3D map built by SceneLM: cutaway point cloud, color-coded object anchors, category legend, and zoomed object crops with their stored text descriptions, next to the raw JSON scene map.
A complete HM3D house, mapped. 684 objects — each stored as nothing more than a label, a one-sentence description and a 3D point. The dark panel shows the actual scene map records.

Map building, frame by frame

Chase-cam view of the map growing as SceneLM walks an HM3D house — objects appear as they are added and refined.

Retrieval on a finished map

Language queries against a finished ScanNet map: the query is matched against the text records and the answering object lights up in 3D.

Interactive

Explore the maps yourself

Three maps built by SceneLM — a ScanNet room, an HM3D house, and a real map built onboard the Spot. Drag to orbit, scroll to zoom, shift-drag to pan. Click any marker to see the object’s record: what you see in the panel is the entire stored representation — there are no embeddings, crops or masks behind it. (For readability, drift duplicates are hidden here.)

loading …
drag: orbit · wheel: zoom · shift-drag: pan · click marker: inspect
scene.md — object record

Real robot, real world

Onboard a Spot, in unseen environments

SceneLM-2b, quantized to NVFP4, runs entirely on a Jetson AGX Thor carried by a Boston Dynamics Spot — no offboard compute — in three environments never seen in training, with real sensor noise, real odometry drift, and an outdoor route with no loop closure. Keyframes are triggered by distance traveled (0.5 m — the measured quality optimum), not by a timer.

A 1.1 km outdoor route with no loop closure — drone footage of the robot interleaved with the map being built onboard. The on-device map holds 1,290 objects.

A laboratory staged with distinctive objects: 33 of 44 manually inventoried targets end up in the on-device map (35 at workstation rates). Misses are systematic — thin, hand-sized items below effective resolution.

A working office of generic, repeated objects — the stress test for cross-view association. 17 of 19 targets at workstation rates (14 on-device); identity degrades before existence does — a leather armchair survives in the map as “a dark fabric chair”.

Map of the laboratory built on board the robot, with zoomed crops of a plush penguin, soccer ball, handcart, ladder and plastic jug and their stored descriptions.
Built on the robot. The lab map from SceneLM-2b-NVFP4 on the Jetson — penguin, soccer ball, handcart, ladder, jug, each with its stored description.
Long outdoor campus map: a thin trajectory point cloud with object anchors for palm trees, arches, sculptures and street lights.
Outdoors, unseen in training. Palm trees, arches, sculptures, street lights — open-vocabulary mapping along the campus route.

Live map building, from the robot’s seat

The full lab run replayed: the camera feed SceneLM reads, the operations log, and the 3D map it maintains — SceneLM-2b at NVFP4 on the Jetson, 258 objects by the end of the run.

Limitations

Hallucinations stay under 2% everywhere and indoor routes close within 10 cm and 10°. The main failure mode is odometry drift producing duplicate entries (up to 30% in cluttered scenes, at the same rate onboard and offboard — a cross-view association problem, not a compute problem), and thin, hand-sized objects fall below the effective resolution. These point directly at future work: stronger cross-view association to curb duplicates, and predicting object extent rather than a single 3D point.

Filmstrip showing SceneLM adding duplicate objects early in a scene and successfully removing them later.
Duplicates get added — and removed. Failure and self-repair in the same run.
A complete HM3D map where a dartboard has been classified as a wall clock.
Failure cases include mislabeling: here a dartboard is mapped as a wall clock in an otherwise complete HM3D map.

Takeaways

Three things to remember

01

The mapping backend can be learned. Association, fusion and reconciliation — the jobs of an engineered geometric pipeline — happen inside one VLM forward pass over a text map.

02

Map-maintenance supervision can be manufactured. OVAL turns 1.27M unlabeled single images into ADD, EDIT, and REMOVE training signal with zero human 3D labels.

03

Text is enough — and it’s portable. Competitive retrieval and localization from a 6–12× smaller, human-readable memory, robust enough to run on an embedded GPU on a walking robot.

Persistent scene understanding could become a fundamental capability of future vision-language models, rather than a specialized robotics module — maintaining a consistent scene representation over time may one day be a standard pretraining objective.

Citation

BibTeX

@article{lilja2026scenelm,
  title   = {A Scene Language Model for Open-Vocabulary Scene Mapping},
  author  = {Lilja, Adam and H{\"u}bel, Fabio and He, Siming and Fu, Junsheng and
             Tomlin, Claire and Hammarstrand, Lars and Malik, Jitendra and
             Frey, Jonas and Pavone, Marco},
  journal = {arXiv preprint arXiv:2609.21400},
  year    = {2026}
}