Human Video vs. Teleoperation Data: Which Trains Better Policies?

If you are building a robot policy, the hardest question is often not the model. It is where the demonstrations come from. The large-scale DROID dataset took 50 collectors 12 months to gather 76,000 teleoperated trajectories, or 350 hours of robot interaction data.

That pace is why so many teams are now asking whether human demonstration data for robots, recorded as first-person video, can replace or support teleoperation. This guide compares both fairly, walks through what recent studies actually measured, and explains how leading teams combine them.

What is teleoperation data?

Teleoperation data is recorded when a human operator drives a robot directly, and the robot logs its own camera views and joint actions. Because the data comes from the same robot that will run the policy, every frame has precise action labels and no embodiment gap.

This is why teleoperation has been the default. The HumanScale authors describe teleoperated robot trajectories as the dominant pretraining source for embodied foundation models, thanks to their exact action supervision and embodiment alignment.

A typical teleoperation data collection setup needs:

• A physical robot: the operator controls it in real time, so every collector needs hardware on site.
• A trained operator: quality depends on smooth, consistent control.
• A suitable space: the scene has to be safe and set up for the robot.

What is human video demonstration data?

Human video demonstration data is first-person footage of people doing real tasks with their own hands, usually from a head, chest or glasses-mounted camera. It shows what the hands, tools and objects do from the actor's point of view, but it does not record robot joint commands.

To make it usable for robots, teams add labels such as 3D hand pose and action segments. EgoMimic, for example, paired egocentric video from Project Aria glasses with 3D hand tracking, then treated human and robot data equally as demonstration data. For a deeper look at why labs want this footage at scale, see why robotics labs are buying egocentric video.

Why is teleoperation data hard to scale?

Teleoperation is hard to scale because it is tied to a robot, a place and real time. You cannot record faster than the operator works, and you can only record where a robot is installed.

Skild AI makes this argument directly. Its team says teleoperation faces two problems:

• Diversity: most data is stuck in labs or the few places a robot is already deployed.
• Scale: even a global workforce driving robots around the clock would take too long. Skild calls reaching LLM-scale data this way "mathematically unfeasible."

The HumanScale paper lists the same limits: high collection cost, difficult acquisition, and low behavioral and environmental diversity in teleoperated real-robot trajectories.

What does human video lack?

Human video lacks the robot's own action signals. It shows what happened, not the exact motor commands a robot should send.

Skild names two gaps:

• Missing signals: videos do not show forces, torques or touch feedback.
• The embodiment gap: a human hand, a 7-DOF arm and a quadruped move very differently, so mapping a human grasp onto a robot is a hard translation problem.

Physical Intelligence adds that past methods often needed manual alignment, such as masking parts of the image or turning human hands into robot hands with generative models. Those tricks can help transfer but tend to cost generality.

What do head-to-head studies show?

Recent studies show human video can match or beat teleoperation data for pretraining, as long as it is processed well and paired with some robot data. Here is what each study measured.

• EgoMimic (October 2024): co-trained one policy on egocentric human video and robot data. Adding 1 hour of human hand data was significantly more valuable than adding 1 hour of robot data, and the policy generalized to entirely new scenes.
• HumanScale (June 2026): compared egocentric video and teleoperated trajectories as pretraining data, with the same data volume and the same post-training. Egocentric pretraining gave a 24% lower validation loss on real-robot action prediction, plus 52.5% and 90% higher success rates on in-distribution and out-of-distribution real-robot tasks.
• Physical Intelligence (December 2025): co-fine-tuned its π0.5 model on egocentric human video plus related robot data. Performance went up by about 2x across four generalization scenarios shown only in the human data.
• Skild AI (January 2026): reports fine-tuning its model on new skills from human videos plus under 1 hour of robot data.

Two caveats matter. HumanScale's result depended on a carefully designed filtering and labeling pipeline, not raw footage. And these are different models, tasks and robots, so the numbers are not directly comparable to each other.

Does human video replace teleoperation?

No. The evidence points to human video as the scaling layer and robot data as the alignment layer, not to dropping robot data altogether.

Physical Intelligence found the benefit of human video grew as robot pretraining grew. On its egg-sorting task, adding more robot pretraining stopped helping the robot-only model after about 60% of the dataset, but kept helping the model fine-tuned with human video. In other words, more robot data made the model better at absorbing human data.

HumanScale reaches a similar recipe: pretrain on egocentric human video to learn diverse world representations, then adapt with a small amount of labeled real-robot data for action-space alignment.

Which should you choose for your robot manipulation dataset?

Choose based on what your policy is missing. Most teams need both, in very different amounts.

• Lean on teleoperation when: you need exact action labels for one robot, force-sensitive skills, or the final alignment step for deployment.
• Lean on human video when: you need many homes, workplaces, objects and people, long-tail task coverage, or pretraining volume that teleop cannot reach.
• Use both when: you are training a generalist policy. Pretrain or co-train on large, diverse human video, then fine-tune on a smaller robot manipulation dataset.

If you are planning for a humanoid specifically, our guide to what training data a humanoid robot actually needs breaks down each data type in more detail.

What Firsthand can and can't do

Firsthand collects the human side of this recipe. Its egocentric video data collection service offers three capture tiers: a chest-mounted phone, a head rig with IMU, and a multi-camera rig with synchronized head, wrist and depth streams. Footage can be annotated for actions, hand pose, objects and transcripts, anonymized, and delivered in RLDS, LeRobot, WebDataset or MP4 + JSONL.

Here is what we can't do:

• No robot teleoperation: Firsthand captures people, not robot joint trajectories. You will still need your own robot data for action alignment.
• No force or torque labels: like all video, our footage does not record contact forces or tactile feedback.
• No borrowed results: the EgoMimic, HumanScale, Physical Intelligence and Skild results above were measured on those teams' own data, not on Firsthand data.

The bottom line

Teleoperation data is precise but slow, expensive and narrow. Human video is abundant, diverse and cheaper, but needs good labeling and some robot data to close the embodiment gap. The latest evidence says the best policies come from both: human video for scale, teleoperation for alignment.