Human Video vs. Teleoperation Data: Which Trains Better Policies?

If you are budgeting a robot learning project, the hardest call is not the model. It is where your human demonstration data for robots will come from. The best-known open teleoperation set, DROID, took 50 collectors 12 months to gather 350 hours of robot interaction. NVIDIA's EgoScale pretrained on 20,854 hours of egocentric human video.

That gap in scale is why so many teams are asking the same question. Should you keep paying for teleoperation data collection, or shift budget to human video? This guide compares both fairly, walks through what the research from 2024 to 2026 actually measured, and explains why the answer is usually "both, in the right order." For background on the market side, see why robotics labs are buying egocentric video at scale.

What is teleoperation data?

Teleoperation data is recorded when a person drives a real robot remotely, and the robot logs its own camera views, joint positions and commands. Every frame comes with the exact action the robot took, in the robot's own body.

That is why it has been the default for training manipulation policies. The HumanScale paper describes teleoperated real-robot trajectories as the dominant pretraining source, because they offer precise action supervision and match the robot's embodiment.

Typical examples of a teleoperated robot manipulation dataset include:

• DROID: 76,000 demonstration trajectories (350 hours) across 564 scenes and 84 tasks, collected in North America, Asia and Europe (arXiv).
• Open X-Embodiment: robot data pooled from 22 robots by 21 institutions, covering 527 skills (arXiv).

What is human video data for robot training?

Human video data is footage of people doing real tasks, usually filmed from a head, chest or glasses-mounted camera so it shows hands and objects from the actor's point of view. It is often paired with hand tracking so the motion can be turned into action labels.

Nobody has to operate a robot to collect it. A contributor with a camera can record in their own kitchen, workshop or warehouse. Physical Intelligence notes this kind of data is easy to record, which is the main reason it scales (Physical Intelligence).

Why is teleoperation data collection hard to scale?

Teleoperation runs in real time and needs a robot in the room, so hours of data cost hours of operator time plus hardware. The DROID team wrote that collecting robot data in diverse settings brings logistical and safety challenges and heavy investment in hardware and labor (arXiv).

Skild AI makes the same argument from the other direction. In its January 2026 post, the company says teleoperation is tied to places where a robot is already deployed, and that even a global workforce driving robots around the clock could not reach LLM-scale data. It calls that goal "mathematically unfeasible" (Skild AI).

The practical result is narrow coverage. Even large teleop sets tend to come from a limited number of labs, rooms and objects.

What are the weaknesses of human video?

Human video is missing two things robots need: the physical signals behind a motion, and a body that matches the robot. Skild AI lists both problems plainly: video does not show forces, torques or touch, and a human hand looks and moves differently from a robot arm or gripper (Skild AI).

Physical Intelligence calls this the domain gap. Humans and robots look different and move differently, so a robot cannot copy a human directly (Physical Intelligence).

Video quality also matters more than volume. HumanScale's positive result depended on a carefully designed filtering and labeling pipeline, not on raw footage (arXiv).

What does the research say about human video vs. teleoperation?

Recent studies point the same way: human video can match or beat robot data for pretraining, but it works best when a small amount of robot data is added for alignment. Here is what each study measured.

• HumanScale (June 2026): With the same amount of pretraining data, models pretrained on filtered egocentric video got 24% lower validation loss on real-robot action prediction than models pretrained on teleoperated robot data. Task success was 52.5% higher in-distribution and 90% higher out-of-distribution (arXiv).
• EgoMimic (October 2024): Co-training on egocentric human video and robot data improved long-horizon manipulation. The authors report that 1 extra hour of hand data was significantly more valuable than 1 extra hour of robot data (arXiv).
• EgoScale, NVIDIA (February 2026): Pretraining on 20,854 hours of action-labeled egocentric video, followed by a small aligned human-robot mid-training stage, raised average success by 54% over a no-pretraining baseline on a 22-DoF robot hand (NVIDIA).
• Physical Intelligence (December 2025): Adding egocentric human video during fine-tuning roughly doubled performance across 4 scenarios that appeared only in the human data. The gain grew as the robot pretraining data grew (Physical Intelligence).

Note what these numbers compare. HumanScale compares pretraining sources of equal size. EgoScale compares against no human pretraining at all. Physical Intelligence compares a policy with and without human video added. None of them shows robot data becoming unnecessary.

Does human video replace teleoperation data?

No. The strongest results use human video for breadth and robot data for alignment. HumanScale's own conclusion is to pretrain on egocentric video for diverse world knowledge, then adapt with a small amount of labeled robot data (arXiv). EgoScale uses the same pattern with a short mid-training stage (NVIDIA). Skild AI says it fine-tunes new skills from video plus less than 1 hour of robot data (Skild AI).

Physical Intelligence adds a useful twist. On its egg-sorting task, more robot pretraining stopped helping the robot-only policy after about 60% of the dataset, but kept improving the policy that also learned from human video (Physical Intelligence). In other words, robot data makes human video more useful, and human video keeps the gains going.

Which should you choose for your robot policy?

Choose by the job each data type does best.

• Use teleoperation data when: you need exact actions for one specific robot, contact-rich force control, or a final alignment set before deployment.
• Use human video when: you need scale, many environments, many objects and many people, or pretraining coverage you cannot reach with robots in a lab.
• Use both when: you are building a general manipulation policy. Pretrain on large, filtered human video, then align with a smaller teleop set on your target robot.

If your target is a humanoid, the trade-offs shift again, since hand dexterity and whole-body motion matter. We cover that in what humanoid robots actually need.

Where Firsthand fits, and what we can't do

Firsthand collects the human side of this recipe: first-person video of real people doing real tasks, to spec. Capture runs from a chest-mounted phone up to a synchronized head, wrist and depth rig, with annotation for actions, hand pose and objects, and delivery in RLDS, LeRobot or WebDataset formats. See our egocentric video data collection service for capture tiers and turnaround.

What we can't do matters just as much:

• We don't collect teleoperation data. Firsthand records people, not robots, so you will still need your own robot data for the alignment step.
• Video can't capture forces or touch. No camera tier replaces robot-side force, torque or tactile signals.
• The results above aren't ours. HumanScale, EgoMimic, EgoScale and Physical Intelligence ran their studies on their own data. None was measured on Firsthand data.

The bottom line

Teleoperation data gives robots exact actions in their own bodies, but it is slow and narrow. Human video is broad and scalable, and in equal-size tests it has matched or beaten teleop data for pretraining. The best policies today use both: human video to learn the world, then a small amount of robot data to learn the body.