Mobile robots are increasingly expected to interpret natural-language goals and carry them out. A key deployment challenge is bringing these reasoning capabilities onboard under tight compute constraints while keeping the robot responsive to its surroundings. How should we design agents for these conditions?

Language-model agents select skills, use feedback, or generate programs over robot APIs. Prior language-model and behavior-tree systems generate, revise, or evaluate trees. Our agent instead selects predefined BT skills in a reasoning-and-acting loop.

The model chooses the next skill from the task and its previous observations. A supervisor runs that skill's predefined behavior tree, with local guards controlling execution. The result returns to the agent, closing the loop.

This real-world deployment shows the robot's surroundings alongside the onboard agent's decision trace.

The onboard nine-billion-parameter agent first inspects the chest and locates the yellow panel. When an execution check rejects pressing without a fresh survey, it gathers the missing observation and retries. After the operator opens the chest to enact the panel-triggered change, the robot returns to inspect the result. It then captures evidence, reports what it sees, and submits; all using onboard inference.

Testing agent designs through physical trials is costly, so our benchmark combines fast text-based comparisons with simulation tests of perception, motion, and skill behavior.

Here, the same task is executed in Habitat and in its symbolic text counterpart. Both use the navigation graph, but their observations differ: Habitat processes camera images, while text records symbolic events.

Success in Habitat usually also means success in text, but the reverse is less reliable.

Faster text evaluation supports iteration, while simulation tests execution under uncertainty.

During execution, skills must stay reactive. Each tick rechecks their guards.

A guard requests a stop locally, without waiting for the model. Commanded velocity reaches zero first; the robot then physically settles while the agent remains active.

After an injected timeout, the agent retries navigation and reaches the goal.

When we simulate a critical battery level, a local guard stops motion and requests a sit. The agent reports the constraint instead of retrying navigation.

BT-Agent brings adaptive reasoning and reactive robot skills together, with multi-fidelity evaluation providing a practical path to onboard deployment.