ToyBlocks: A controlled world for robot learning and evaluation

Back to Articles ToyBlocks: A controlled world for robot learning and evaluation Enterprise Article Published November 6, 2026 Upvote - Kyle Wiggers Ai2Comms Follow allenai 💻 Code | 📊 Data | 🤖 Models | 📖 Documentation Robots are learning to fold laundry, sort parts, and handle unfamiliar objects.

But those demonstrations are often rather narrow and don't explain what a robot has learned or why it fails.To study a complex system, scientists often begin with a small, controlled environment, such as a petri dish or terrarium, where they can vary conditions and observe the effects.

Robotics needs environments that make this kind of investigation possible—researchers should be able to change the training data or model design, repeat a test, and trace the cause of a failure.Today we're introducing a framework for creating these environments; we call them FoundationWorlds.

Each pairs a controlled, simulated world targeted at studying robotics in a specific environment with tools for automatically generating tasks and training demonstrations, as well as metrics for measuring a robot's behavior over time.

Together, they provide a setting for studying questions such as how models improve as more training data becomes available, and how to best combine high-level reasoning with precise physical control.

Our first foundation world is ToyBlocks: an open simulation framework built around a robot arm and blocks on a tabletop.The setting is deliberately simple, but the robot must still reason about how the blocks interact and how to arrange them while pushing, flipping, and stacking.

ToyBlocks automatically generates demonstrations, which are recordings of a robot completing tasks in simulation, without requiring time-consuming teleoperation, in which a person remotely controls the robot.Researchers can easily vary these training experiences.

For example, a team could generate more pushing or flipping demonstrations, then evaluate performance on tasks kept out of training.Starting with ToyBlocks as our first FoundationWorld, we aim to extend the approach to increasingly complex environments.

How ToyBlocks works Building on our earlier work in open robot learning, including MolmoAct 2, MolmoBot, and MolmoSpaces, ToyBlocks gives researchers a way to study robotics questions in a deliberately constrained domain.

As a FoundationWorld, ToyBlocks is designed around four requirements: (1) realistic physics and appearance, (2) automatic generation of tasks and demonstrations at scale, (3) challenges that combine reasoning with dexterity, and (4) broad coverage of the tasks possible within a domain.

Open-ended domains require modeling a wide range of objects and interactions, making it infeasible to fulfill all these requirements.ToyBlocks’ narrow scope, on the other hand, makes it possible to fulfill them all.

Within ToyBlocks' smaller setting, researchers can work toward investigating how a robot can learn to perform any reasonable task while being confident that the simulated arm and blocks behave very similarly to their physical counterparts.

A physical robot gripper above wooden blocks in the ToyBlocks setup.ToyBlocks is not just about picking and stacking blocks: pictures and letters on the blocks' faces allow for searching, sorting, and arranging tasks.

A robot may need to turn a block to see a hidden face before deciding where it belongs.Domain randomization – varying block size, table height, and other physical properties from scene to scene – tests how well models generalize as the geometry changes.

ToyBlocks automatically generates skills (behaviors for short tasks) and task demonstrations that can be used to train models or adapt previously trained models through additional training (fine-tuning).

High-level reasoning models decide what to do; low-level action models control the robot’s movements.The framework can also be used to easily design experiments and validate hypotheses.

ToyBlocks uses reinforcement learning (RL) and motion planning, which computes how the robot should move, to generate skills for short-horizon tasks, which involve brief sequences of actions.Skill training takes place in Isaac Lab, Nvidia's open-source framework for robot learning in simulation.

Through trial and error, these policies – models that choose the robot’s actions – learn non-prehensile skills such as pushing a block across the table or flipping it onto another face without picking it up.

Skill generation can draw on privileged information – values the simulator can supply directly but a real robot cannot, such as a block's exact position and orientation – instead of estimating it from camera images.

This enables the automatic generation of skills that would require teleoperation in the real world.To generate demonstration data, a task-and-motion planner (TAMP) automatically selects and sequences the skills needed to solve a task and plans the robot’s movements between them.

Running each plan in simulation produces a dataset of recorded demonstrations along with labels describing which task or skill the robot performs.

ToyBlocks scales data generation through generating and executing hundreds of such plans in parallel on a single graphics processing unit (GPU), reaching up to 24 times real-time performance.

Acting from sensor observations adds a perception problem: a model has to estimate a block’s pose – its position and orientation – which the simulator could supply directly.To reflect real-world conditions, ToyBlocks trains models that map raw sensor observations to actions.

Execution of Molmo 2 (a high-level model) and ManiFlow (a low-level model) trained in ToyBlocks.Molmo 2 selects skills such as rotation and flipping, indicating the relevant block via a pointing interface, while ManiFlow controls the simulated arm to execute the commanded skills.

ToyBlocks' controllable data generator lets users create training mixtures that emphasize particular skills or fill gaps in existing datasets.Researchers can study how much data a model needs and which experiences help it generalize to unseen tasks.

For example, they could test whether adding rotation and pick-and-place demonstrations improves flipping, or vary the balance of examples inside and outside a defined workspace region—evaluating their choices on tasks kept out of training.

Importantly, evaluation of trained models generates metrics beyond success rates; it provides information such as execution speeds, forces exerted on objects, and task progress measured via the same specifications used by the planning module.

A webpage interface lets users connect Claude – or another frontier model – and use natural language to instruct the robot or try tasks, offering a way to explore how a model plans and acts without needing robotics expertise.

Research building on ToyBlocks The evaluations below use ToyBlocks to study how training data and methods affect performance, spatial reasoning, generalization to new letters and block appearances, multi-step planning, transfer to physical robots, and frontier models that act through tools.

Users can easily design their own tasks and measures to investigate other questions—for example, whether a model performs better when it selects objects by pointing or by issuing text commands such as "pick the yellow block.

" Low-level action models: VLA spatial generalization Low-level action models, such as vision-language-action models (VLAs), map language instructions and visual observations to robot actions.

The premise of VLAs is that, by pretraining on large-scale datasets, they can rapidly generalize to different scenarios or a specific workspace.However, evidence on how well these action models generalize remains conflicting.

Some studies report that these action models are brittle to changes in inputs, such as different object layouts and lighting, whereas others show that VLAs generalize well.We argue that these studies often rely on real-world datasets with limited flexibility in spatial variations.

ToyBlocks enables us to test how well action models generalize in a simple, easily reproducible setting with meaningful task-space diversity.

We design generalization tests that vary the skill sequences needed to complete an instruction, the letters or colors on the blocks, and the number of blocks in a scene.

A stacking study can withhold four-block stacks from training, then test whether a model can combine familiar movements to build them.Workspace studies distinguish performance at new positions within a training region from performance beyond its boundaries, where the model has to extrapolate.

We illustrate the study setup and findings below.We consider four tasks: Stack, Unstack, Flip, and Rotate.For these tasks, we study interpolation (new configurations within the training range) and extrapolation (configurations beyond that range).

For example, we evaluate generalization by training on scenarios where objects spawn only in the left region (green) and testing on the right region (orange), or by training on stack heights of 2 and 5 blocks and testing on heights 3 and 4.

Spatial generalization studies for VLA models (results based on ManiFlow, π0.5, and MolmoAct 2).Despite large-scale pretraining, these popular action models generalize poorly in the initial results.We improve generalization and analyze the contribution of each technique.

The chart’s LoRA label stands for low-rank adaptation, a method for fine-tuning a model with a small set of trainable weight updates.We believe action model generalization remains far from solved; ToyBlocks provides a valuable testbed for systematically exploring generalization techniques.

High-level reasoning: vision-language models Long-horizon tasks, which require many steps, allow us to measure how well models can plan a sequence of actions to solve them.

Our framework enables us to create such long-horizon tasks, and create oracles – reference components that provide correct plans or actions – at both the high and low levels.This allows us to isolate whether the high-level planning or low-level execution failed when a model fails at a task.

An example long-horizon task might be to spell “Ai2” with a block stack; because the letters may not be immediately visible, the agent must first choose actions that reveal information, and then plan and execute a sequence of actions to build the stack.

We evaluate hierarchical models with explicit high-level planning and low-level execution models.In this instance, we choose Molmo 2, a vision-language model (VLM), as the high-level planner, and ManiFlow as the low-level policy.

Molmo 2 chooses the skill to execute and the block to execute it on, while the low-level model executes the instruction.We train low-level models such as ManiFlow to pause after completing the execution and query the high-level model for the next instruction.

We evaluate the model’s planning abilities on three scenarios of increasing difficulty.The easiest scenario initializes with each character upright and facing the camera.

The medium scenario has characters that may not be upright, and the hardest scenario contains characters that are not visible at the start of the episode.

Success rates on three-letter block-spelling tasks using Molmo 2 as the high-level reasoning model and either ManiFlow or a motion planner for low-level manipulation.An example task is “Stack Ai2”.

Molmo 2 and ManiFlow are trained on ToyBlocks and evaluated across three difficulty settings and two generalization conditions.High-level reasoning: agentic systems Our experiments with agentic systems test how frontier models plan and act through tool calls.

We study three settings using Claude Opus 5.5.In the search-and-spell setting, we expose ToyBlocks skills through the Model Context Protocol (MCP), a standard for connecting AI applications to tools.

The agent sees the scene through camera images, calls skills such as pick, flip, or rotate by pointing a