What Training Data Does a Humanoid Robot Actually Need?

If you are building a humanoid, the hardest question is often not the model but the data mix. In September 2026, Figure reported that pretraining on human video raised its robot's zero-shot success in 30 unseen homes from 9% to 56%, with the robot-specific data held fixed (Figure, Helix 2.5).

That result explains a lot of the current buying spree, which we covered in why robotics labs are buying egocentric video at scale. But it doesn't mean human video is all a humanoid needs. Every company in this guide still trains on its own robot data, and most lean on simulation too. Below is what each layer of humanoid robot training data does, using only what Figure, 1X, Agility, Apptronik and NVIDIA have published.

What kinds of training data does a humanoid robot use?

Humanoid teams combine four kinds of data: web-scale data, human video, simulation or synthetic data, and real robot data. Each one teaches something the others can't, and they are usually used in that order, from broad to specific.

NVIDIA's GR00T N1 paper describes this as a "data pyramid." Web data and human videos form the base, synthetic data from physics simulation or neural models forms the middle, and data collected on real robot hardware sits at the top (NVIDIA, GR00T N1). The paper says the lower layers give broad visual and behavioral priors, while the top layer grounds the model in real robot execution.

Agility Robotics frames it in terms of where the data comes from. It names three sources: generating data itself through teleoperation, simulation, and public internet data, which it notes does not yet exist for robot movement (Agility, Agility and AI).

• Web-scale data: teaches language, objects and general visual understanding, usually through a pretrained vision-language model.
• Human video: teaches how people move through tasks, handle objects and adapt to new rooms.
• Simulation and synthetic data: teaches balance and control, and multiplies a small number of demonstrations.
• Robot data: teaches the model its own body, cameras and joints, and grounds everything else in real execution.

Why does a humanoid need its own robot data?

Because only robot data records what the robot's own body did. Human video shows a hand closing on a mug; robot data shows which joints moved, how hard the hand pushed, and what the robot's cameras saw while it happened.

Agility describes its teleoperation setup plainly. Engineers wear VR headsets, see through Digit's cameras and use two controllers to show it a task. While they do, Digit records camera feeds, joint positions, force readings and end-effector location (Agility, Agility and AI).

This is still how many humanoid models start. Figure's first Helix model was trained on about 500 hours of teleoperated data collected across multiple robots and operators. That data drove a 35-degree-of-freedom upper body at 200 Hz, from individual fingers to head and torso (Figure, Helix).

Some companies are scaling robot-side collection directly. Apptronik opened an expanded, nearly 90,000-square-foot Robot Park in Austin in June 2026, where Apollo 2 robots collect data through a mix of teleoperation and autonomous work. Similar workflows run at Google DeepMind and at customers including Mercedes-Benz and GXO, and the data trains Google DeepMind's Gemini Robotics models (Apptronik).

The limit is speed. Agility says teleoperation is the most expensive way to get data and is bound by space and time, because a person has to drive a real robot for every hour.

What does simulation teach a humanoid?

Simulation is best at the parts of control that are easy to score and dangerous to practice on real hardware, like staying balanced. It can also turn a handful of demonstrations into thousands of training examples.

Agility's whole-body controller for Digit is a small neural network with fewer than one million parameters. It is trained in NVIDIA's Isaac Sim for decades of simulated time over three or four days, is learned purely in simulation, and transfers to the real robot without extra training (Agility, Whole-Body Control Foundation Model). Agility says it can then learn dexterous manipulation skills on top of that layer.

NVIDIA used simulation to multiply data instead. Its team generated 780,000 simulated trajectories, which it says equal 6,500 hours of human demonstration data, in 11 hours. It also used video-generation models to turn 88 hours of in-house teleoperation data into about 827 hours of new video, roughly a 10x increase (NVIDIA, GR00T N1).

Figure uses simulation too. The fast control part of the original Helix used a vision backbone pretrained entirely in simulation (Figure, Helix).

Why are humanoid companies pretraining on human video?

Because human video can be collected far faster than robot data, and published results now show it transfers to humanoids. It gives a model broad experience of real homes and workplaces before it ever touches a robot.

Figure's Helix 2.5 is the clearest test. Figure trained two policies on identical task data. One started from scratch; the other was pretrained on Index, Figure's dataset of human behavior. In blind trials across 30 Bay Area homes the robot had never seen, the scratch policy succeeded 9% of the time and the Index-pretrained policy 56% of the time (Figure, Helix 2.5).

Figure reports two more effects. Helix 2.5 needed half as much task-specific data as an earlier Helix 02 behavior. And loss fell predictably as Index data doubled, which Figure calls a human-to-humanoid transfer scaling law.

What makes the video useful is its range. Per 1,000 hours, Figure says Index contains 373 unique tasks, 1,146 unique manipulated objects and 116 unique environments. By September 2026, Figure said Index was generating roughly 35 minutes of new human experience every second (Figure, Introducing Index; Figure, Helix 2.5).

1X uses human video differently, inside a world model for its NEO humanoid. It starts from a 14-billion-parameter video model, trains on 900 hours of egocentric human video, then fine-tunes on 70 hours of robot data. In 1X's tests, adding the egocentric human data improved video generation quality on new tasks (1X, World Model).

What can't human video teach a humanoid robot?

Human video can't show a robot its own body. It carries no joint positions or force readings, and a human hand is not a robot hand, so every company above adapts the model with robot data afterward.

1X states the reason directly: its 70 hours of robot data adapt the model to NEO's visual appearance and kinematics. Figure still trains each Helix 2.5 behavior on task-specific data, just less of it. NVIDIA keeps real robot data at the top of its pyramid for the same reason.

There are coverage limits too. 1X notes that its NEO post-training data was 98.5% pick-and-place. Human video can widen what a model has seen, but the robot-side data still defines what the robot has practiced.

This is why human demonstration data collection is a pretraining and coverage tool, not a replacement for robot data. The open question is how far the robot share can shrink. We'll compare the two sources directly in a future post on human video versus teleoperation.

How much humanoid robot training data do you need?

There is no single number. The published amounts vary by orders of magnitude, depending on whether a team is pretraining a foundation model or adapting one to a task.

Here is what the companies above report:

• Figure Helix (Feb 2025): about 500 hours of teleoperated robot data, which Figure says is under 5% of the size of previous VLA datasets (Figure).
• 1X world model (Jan 2026): 900 hours of egocentric human video plus 70 hours of robot data, on top of a video model trained on web-scale video (1X).
• NVIDIA GR00T N1 (Mar 2025): 88 hours of in-house teleoperation expanded to about 827 hours with generated video, plus 780,000 simulated trajectories (NVIDIA).
• Figure Index (Sep 2026): about 35 minutes of new human experience every second, used to pretrain Helix 2.5 from random initialization (Figure).
• Agility motor cortex (Aug 2025): decades of simulated time, generated in three or four days (Agility).

The practical takeaway: robot data budgets in these reports run from tens to hundreds of hours, while Figure's Index rate works out to roughly 50,000 hours of human video a day. Most teams won't build the pretraining layer themselves. They need targeted data for the environments, objects and tasks their model handles badly.

How should you plan robot training data collection?

Start from the gap, not the volume. Work out which layer your model is weak in, then collect for that layer.

• Weak at control or balance: look at simulation and reinforcement learning first.
• Weak in new rooms or with new objects: add human video with more environments, objects and task variations per hour.
• Weak at a specific task on your robot: collect teleoperated demonstrations on your own hardware.
• Weak at rare situations: collect the edge cases and failures on purpose rather than waiting for them.

Quality checks matter at every layer. Figure runs its human video through five stages before training: filtering, fraud review, deduplication, rebalancing and annotation. It drops clips too similar to data it already accepted (Figure, Introducing Index). Embodied AI training data is only as useful as the pipeline that cleans and labels it.

Where does Firsthand fit, and what can't we do?

Firsthand, CloudPano's AI training data team, collects the human video layer. We run egocentric video data collection to a written spec, at three capture tiers: a phone in a chest mount, a head-mounted rig with an IMU, or synchronized head, wrist and depth cameras on one hardware clock. Annotation layers include actions, hand pose, objects and transcripts, and we deliver in formats such as RLDS and LeRobot. Faces, plates and screens are blurred and checked by a reviewer before delivery.

Here is what we can't do. Human demonstration is our default, so we don't supply the robot-side joint and force data that only your own hardware can record; if you need teleoperated robot trajectories, that has to be scoped as its own program. We don't offer simulation or synthetic data. And the results in this article (Figure's 9% to 56%, 1X's ablations, NVIDIA's data pyramid) were measured on those companies' own data and models. None of them used Firsthand footage, so they show what human video can do, not what our data will do for your robot.

The bottom line

A humanoid robot needs more than one kind of data. Web data gives it general understanding, human video gives it broad real-world experience, simulation teaches it balance and control, and robot data teaches it its own body. The newest results show human video is the layer that scales, but every published recipe still ends on the robot. If you are planning robot training data collection, find the layer where your model is weakest and spend there first.