BT-Agent

Robot learning & autonomy / Preprint · 2026

BT-Agent.Deploying Onboard Robot Agents
with Behavior Trees and
Multi-Fidelity Evaluation

Fabio Hübel1Adam Lilja1Yue Yin1Siming He2Claire Tomlin2Shankar Sastry2Jonas Frey1,2Marco Pavone1,3

1 Stanford University2 UC Berkeley3 NVIDIA

Read the paper arXiv Watch the film Code on GitHub

BT-Agent research film

A 9B agent reasons, observes, and acts onboard the robot.

Read transcript

Recorded experiments; edited sequences and playback speeds are labeled in the film. The chest-opening change is enacted by an operator.

01 / The approach

Reason about the task.
Stay reactive on the robot.

Language-model decisions take time. Robot execution needs to respond as conditions change. BT-Agent connects the two through predefined behavior-tree skills.

An onboard Actor selects the next skill and its arguments. A supervisor runs it, local guards recheck execution conditions, and structured results inform the next decision. The skill implementations and guards stay fixed at runtime.

ONBOARD RUNTIMEArchitecture walkthrough · schematic
Task-level reasoning

LLM Actor

Task · history · observations

Select a skill + arguments
Orchestration

Supervisor BT

One skill at a time

Dispatch · return status
Reactive execution

Predefined skill

Local guardsSkill body
Sensors + robot control

The Actor uses the task, history, and stored results to select an available skill and its arguments. A supervisor starts one skill at a time.

View the complete architecture from the paper
The ReAct Actor and supervisor form a feedback loop. Physical skills recheck cancellation, time limits, and conditions before the skill body on every tick. Context skills check cancellation. Sensors and control connect the skills to the robot.
Figure 1. The Actor chooses skills; predefined behavior trees govern their execution. Open the image for a larger view.

02 / On the robot

Local checks.
Responsive execution.

When a guard fails, the running skill requests a stop without waiting for another model decision. The agent stays active to retry or report the constraint.

19ms

Median local acceptance
to commanded zero

Across nine interventions during motion. This measures the velocity command, not physical stopping time; the robot settles afterward.

11 of 15 real-world trials completed.
Five tasks, three trials each. Reasoning, perception, planning, and control run onboard; motion requires operator approval.

Watch: interruption, timeout, and simulated low batteryExcerpt · 2:14–2:51 of the research film · original edit retained

03 / Multi-fidelity evaluation

Develop in text.
Test with sensing and motion.

Shared task goals and skill interfaces connect a fast text environment with simulator execution. Text makes agent comparisons inexpensive; Habitat reveals how perception and motion change the outcome.

Text success is useful evidence for development. Paired simulation trials show which decisions survive contact with visual observations and navigation.

A / Text161

benchmark tasks

50 primary + 111 transfer tasks.
Controlled comparisons and injected faults.

B / Habitat39

paired tasks

Shared goals and scene priors.
VLM observations and simulated motion.

C / Onboard9B

model on the robot

Boston Dynamics Spot.
NVIDIA Jetson Thor.

The 39 paired tasks come from two configurations of one home. Isaac-derived and transfer tasks are evaluated quantitatively in text only.

Paper Figure 4 compares four 9B methods across 39 tasks, with 117 episodes per method in each backend. Of 293 pooled text successes, 168 also succeed in Habitat. Task-mean success has a Spearman correlation of 0.61.
Paired outcomes across four 9B agent configurations.
Figure 4 from the paper · open to enlarge.

What transfers?

Similar ordering.
Different outcomes.

Method ordering agrees across text and Habitat in this evaluation. But only 57.3% of text successes also succeed in Habitat: shared goals do not remove perception and motion failures.

10.9×

Higher throughput in text

950 vs. 87.3 episodes/hour at eight workers in the reported 16-task setup, excluding timeout attempts. This is a setup-specific comparison.

Text trials also show the largest BT-Agent gains over the strongest Act/ReAct baseline at 4B and 9B: 10.1 and 5.0 percentage points. Results depend on model size, task, and inference cost.

Read the evaluation and limitations
How the benchmark is constructed
Simulator scenes provide navigation graphs, object records, and camera evidence for human-reviewed tasks. The task suite contains 50 primary and 111 transfer tasks; 39 Habitat tasks are evaluated in simulation.
Figure 2. Geometry and camera records connect task construction across backends. Instance masks are construction evidence, not VLM input.

04 / Paper & code

Build on this work.

The paper is available on arXiv.
The agent code is released under Apache-2.0. The benchmark release will follow.

arXiv Code

Citation

@misc{huebel2026btagent,
  title = {BT-Agent: Deploying Onboard Robot Agents
    with Behavior Trees and Multi-Fidelity Evaluation},
  author = {H{\"u}bel, Fabio and Lilja, Adam and Yin, Yue
    and He, Siming and Tomlin, Claire and Sastry, Shankar
    and Frey, Jonas and Pavone, Marco},
  year = {2026},
  eprint = {2610.13862},
  archivePrefix = {arXiv},
  primaryClass = {cs.RO}
}