Imagine watching someone open a screw-top water bottle. One hand holds the bottle steady while the other twists the cap and lifts it away. To us, this looks effortless. But that is because we already understand how objects work: we know that the cap follows a screw thread, that the bottle must remain stable, and that the cap can only be lifted once it is unscrewed. For a robot, however, the same few seconds of video leave nearly everything that matters unanswered. The motion is visible, but the physical interactions that make it possible are not.

Simply copying the human’s hand movements is not enough. The robot must infer how the cap moves, find suitable grasps for its own hands, and coordinate both arms while respecting physical constraints. Even if it successfully reproduces the demonstrated motion, that alone does not teach it how to handle a bottle in a different position. The real challenge is turning a single observed demonstration into diverse, physically consistent robot experiences from which a generalizable policy can be learned.

This is the idea behind DexAgent, our agentic framework that transforms a single egocentric human video and task description into diverse robot training data (Figure 1). Rather than directly imitating visible motions, DexAgent uses an AI agent to reconstruct the underlying physical interactions in simulation, verify that the reconstructed objects possess their expected physical properties, and generate feasible, optimized robot trajectories. It then generates diverse training demonstrations across different initial conditions, while continuously expanding a reusable library of skills and verifiers. By turning visual observations into physically grounded simulation and robot training data, DexAgent enables robots to learn complex manipulation tasks from just one human video. Across eleven real-world tasks, policies trained on DexAgent-generated data achieve 63.6% average success, compared with 18.2% for the strongest evaluated baseline—a 3.5× improvement.

Figure 1. From human demonstrations to robot skills. DexAgent generates training data for tasks involving rigid objects, moving mechanisms, deformable objects, and multi-step interactions. Each task starts from one human video and a task prompt.

An agent that builds and checks its own tools

DexAgent is an agentic workflow that organizes video-to-robot conversion into four stages: semantic understanding, object property-based simulation reconstruction, robot trajectory optimization, and robot data generation. A shared tool library supports the entire workflow.

Figure 2. The DexAgent workflow. The agent identifies what matters in the demonstration, reconstructs the relevant objects, generates robot motion, and expands verified trajectories into training data. Skills and verifiers are selected or developed as needed and retained in a shared library.

Let’s follow the bottle-opening example through the four stages (Figure 2).

1. Understand what the task requires

For the bottle, this means recognizing that the cap turns relative to the body before detaching, and that one hand stabilizes the bottle while the other manipulates the cap.

First, DexAgent identifies the objects, their relevant properties, and the smaller steps that make up the demonstration. It outputs a structured natural language description of the scene and its subgoals, including the actions of each hand and criteria for success. This structured description gives later stages of DexAgent the specific object interactions it needs to reproduce during the data collection. For a drawer task, for example, the generated description describes the need to open the drawer enough to place the toy inside.

2. Reconstruct properties, not just appearances

Next, DexAgent builds the objects in simulation. The key is to reproduce the properties needed for the task, not just the objects’ visible shapes. A screw cap needs a mechanism coupling rotation to upward movement; a drawer needs a sliding joint with the appropriate direction and range.

Based on the structured description, the agent selects or writes reconstruction skills, Python functions that build simulated objects, and verifiers that check whether those objects satisfy the required physical properties. For example, a verifier might check whether rotating the bottle cap produces the correct upward displacement.

The agent uses the structured description from the previous stage, which identifies the objects, their physical properties, and the interactions required to complete the task. Based on this description, it first selects or creates checks for the required properties, then selects a reconstruction skill or writes one when the library has no suitable tool. It iteratively refines the simulated objects until all checks pass, and tests newly created reconstruction functions on three additional versions with different parameters before retaining them. The goal is to build reusable methods for reconstructing a class of objects, rather than a model that only works for one demonstration.

3. Find motion that works for the robot

With the scene reconstructed, DexAgent generates robot motion for each subgoal. It references the demonstrated human pose, optimizes the robot’s fingers to reproduce relevant contacts, and generates a grasp based on the task’s requirements. Critically, our setup frames human motion as a guide, not as a requirement to copy every movement exactly.

Candidate hand poses are tested through simulated interactions. A grasp used to carry an object can be checked by lifting and shaking it; a grasp used to manipulate a mechanism must allow the required sliding or rotation. The agent then connects verified poses into a complete trajectory, combining human-motion-guided optimization with code-generated motion. When an early action leaves the scene in a state that prevents a later step, it can revise the earlier motion.

4. Turn one solution into many training examples

A single valid trajectory is a starting point, not a complete training set. DexAgent varies the initial object and robot states in simulation, adapts the trajectory to each configuration, and checks that the resulting episode remains feasible. In our policy experiments, this produces 500 verified robot episodes per task.

The visual observations also need to resemble what the real robot will see. Rather than generating these images directly, DexAgent renders the reconstructed objects and robot hands in simulation using the verified robot trajectories. It removes the demonstrator’s hands and manipulated objects from the original video, composites the simulated renderings into the cleared scene, and refines their appearance through retexturing. The agent coordinates this process using simulation and rendering tools, allowing it to generate visually realistic robot training data while preserving physically consistent interactions. (Figure 3).

Figure 3. One demonstration, many robot training examples. Bottle opening, rope knotting, and battery insertion illustrate the human demonstration, processed robot-view observation, synthesized wrist views, and examples with different random seeds. Source: selected rows from paper Figure 6.

We use these generated episodes to fine-tune the pretrained π0.5 policy2.

A tool library that grows across demonstrations

A key feature of DexAgent is its self-evolving tool library, which continuously grows as the agent encounters new tasks. Instead of starting from scratch for every video, DexAgent saves newly developed skills and verifiers as reusable code that can be applied to future demonstrations. Over time, the library accumulates capabilities for reconstructing objects, generating robot motions, and verifying physical interactions, making the framework increasingly efficient and adaptable.

Across our experiments, the library contains 103 skills and 188 verifiers. In a separate 100-video EgoDex3 experiment, processing time falls from 4.3 hours for the first video to 2.2 hours with skill reuse, and to 13.1 minutes when object assets can also be reused. The full run averages 2.1 hours per video; the 13.1-minute result reflects the asset-reuse setting, not an arbitrary new task (Figure 4).

Figure 4. Reusing skills and object assets reduces offline conversion time. This 100-video experiment accumulates 85 skills and 168 verifiers, a subset of the broader library.

Does the generated data help a real robot?

We evaluate policies on a bimanual robot with two multifingered Sharpa hands across eleven tasks, including battery insertion, scissor cutting, rope knotting, and computer installation. These tasks cover rigid, articulated, and deformable objects, as well as tool use and longer sequences. Each policy is tested over ten trials per task with randomized initial object placements (Figure 5).

Figure 5. Human demonstrations, reconstructed scenes, and successful real-robot rollouts for rope knotting, a biology experiment, battery insertion, and bottle opening.
Figure 6. Policy rollout success rates of DexAgent and other competing baselines across 11 real-world long-horizon dexterous manipulation tasks. Among all these tasks, DexAgent achieved a 63.6% average success rate compared to the best baseline of 18.2% average success rate, around a 3.5x improvement.

Policies trained on DexAgent-generated data achieve 63.6% average success, compared with 18.2% for SPIDER and 17.3% for V2D (Figure 6)—approximately 3.5× the strongest evaluated baseline. For these comparisons, we use the same policy model and a common data-augmentation and visual-processing protocol, retaining each method’s supported conversion components.

The gains extend beyond simple pick-and-place: DexAgent-trained policies achieve 80% success on placing a toy in a drawer and 50% on rope knotting. These results suggest that adapting reconstruction and motion generation to the interaction can produce useful training data for a wider range of dexterous tasks.

Looking ahead

Passing simulation checks does not guarantee success in the real world. DexAgent can misunderstand a demonstration or reconstruct an object incorrectly, and its verifiers can miss those errors. An undetected error can also be retained in the library and affect later videos.

Force-sensitive interactions are a particular challenge: a visually plausible reconstruction and a successful simulated motion may still fail on the physical object. Stronger verification and smaller simulation-to-reality gaps remain important directions. We also plan to explore reinforcement learning as a fallback when the available skills cannot produce a valid solution.

DexAgent treats learning from human video as more than translating one motion sequence into another. The video describes what needs to happen; simulation provides a place to test candidate solutions, and the tool library preserves useful ways to build and check those solutions.

Our results suggest that an adaptive, verification-guided workflow can turn human demonstrations into useful data for a range of dexterous tasks. We hope this points toward data-generation systems that do not just process more videos, but also reuse what they have built along the way.

Paper and website

This work is by Youhui Wang, Yunzhu Li, Fei-Fei Li, Jiajun Wu, Huang Huang (Stanford and Columbia).

References

  1. Youhui Wang, Yunzhu Li, Li Fei-Fei, Jiajun Wu, and Huang Huang. DexAgent: An Agentic Human2Sim2Robot Framework for Dexterous Manipulation with Self-Evolving Tool Library. 2026. arXiv:2609.35318.

  2. Physical Intelligence et al. π0.5: A Vision-Language-Action Model with Open-World Generalization. 2025. arXiv:2504.16054.

  3. Ryan Hoque, Peide Huang, David J. Yoon, Mouli Sivapurapu, and Jian Zhang. EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video. 2025. arXiv:2505.11709.