RoboAuto

Autonomous robot program search through multi‑level physical understanding

Execution feedback reveals whether a robot program succeeded, but rarely why it failed. RoboAuto guides program search with multi-level physical understanding: a hierarchical knowledge base, grounded in each scene, verified stage by stage, and refined through self-designed experiments.

Buzhao Liu^*, Jianing Ye^*, Xiu Yuan^*, Yangtong Feng, Chengsong Huang, Tyler Dinh, Yevgeniy Vorobeychik, Ning Zhang, Chongjie Zhang
Washington University in St. Louis

arXiv Code coming soon BibTeX

LIBERO-PRO tasks: 89.1%
LIBERO-PRO Long tasks: 73.4%
LIBERO-Plus tasks: 90.3%
Robosuite tasks: 90.6%

Overview

RoboAuto searches for robot programs with physical understanding that it retains, grounds in each scene, and refines through experiments.

Hierarchical knowledge base. Object-centric physical knowledge from demonstration videos, plus executable skills accumulated through online task solving.
Physical scene grounding. Measures the quantities each entry needs in the current scene, moving the arm to a more informative view when the first one is not enough.
Search and exploration. Plans in verified stages and, when physical understanding is uncertain, explores actively through self-designed experiments.

Real-robot tasks

Three long-horizon precision tasks on a WidowX AI arm.

Mean stage-completion score over 20 trials per task.

Method

Two kinds of experience, kept apart on purpose

The high-level layer holds object-centric physical knowledge: affordances, spatial relations, and action effects, each with the quantities that must be measured in every new scene. The low-level layer holds executable skills, programs verified during task solving.

Knowledge base examples

One stored entry of each type.

Measure first, then plan

A strategy that is right for an object class can still fail when its numbers are copied from another object or guessed from a segmentation mask. Before writing any code, RoboAuto identifies the task-relevant objects, retrieves their entries, and runs a measurement probe that establishes every open quantity from depth point clouds: circle fits for vessels and plates, oriented boxes for cartons, protruding-point fits for handles.

When the initial observation is not enough, active viewpoint selection moves the arm to get a more informative camera view. Estimates from the scene camera and the wrist camera must agree within one centimeter. Anything that fails its checks stays unresolved and is handed to search as a question, not a number.

Scene camera view of the plate-on-stove task with measured quantities

Quantities measured by the probe on scene 110000 of “put the plate on the stove”.

Stages, an auditor, and experiments

The coder plans the task as stages that each end in a checkable object state. An independent auditor watches failed runs and can overrule a verifier, and when its physical understanding is uncertain, the coder explores actively with a self-designed experiment before changing code.

Example run: put the plate on the stove

Results

RoboAuto achieves the highest mean success on the evaluated tasks from every benchmark family, and more than doubles ASPIRE on LIBERO-PRO Long tasks (73.4 vs. 30.8) and LIBERO-Plus tasks (90.3 vs. 44.7).

Mean final-program success (%) on test seeds; the number of tasks is in parentheses. Pos and Task denote position and task perturbation. A dash marks an unavailable result. The last row is RoboAuto's difference from ASPIRE.

Component ablation

Mean success on five tasks as search and exploration, the hierarchical knowledge base (KB), and scene grounding are added in turn.

Transfer to RoboTwin 2.0

New simulator and bimanual embodiment, without and with the hierarchical knowledge base.

No retained knowledgeHierarchical knowledge base

Token usage ↓

Geometric-mean total tokens per task over 16 LIBERO-PRO Long and LIBERO-Plus tasks.

RoboAutoASPIRECaP-Agent0

More tasks

Example tasks from five benchmark families: RoboTwin 2.0, Robosuite, short-horizon LIBERO-PRO (Object, Goal, Spatial), long-horizon LIBERO-PRO Long, and LIBERO-Plus.

BibTeX

@article{roboauto2026,
  title   = {RoboAuto: Autonomous Robot Program Search through Multi-Level Physical Understanding},
  author  = {Liu, Buzhao and Ye, Jianing and Yuan, Xiu and Feng, Yangtong and Huang, Chengsong and Dinh, Tyler and Vorobeychik, Yevgeniy and Zhang, Ning and Zhang, Chongjie},
  journal = {arXiv preprint arXiv:xxxx.xxxxx},
  year    = {2026}
}