Why Your Robot Policy Plateaus at 50%

Your policy picks up the cup in the lab demo, then fails every other run on the real line. Physical Intelligence, the company behind the π0.6 robot model, describes the same wall: with demonstrations alone, "it's not hard to get a robot to succeed half of the time," but succeeding every time is a different problem (Physical Intelligence).

That gap is rarely a model-size problem. It is a data problem, and specifically a missing-data problem: your training set shows the robot what success looks like, but almost never what to do one second after something goes wrong. If you have read Why Robotics Labs Are Buying 1,000,000 Hours of Egocentric Video, this is the other half of the story. Volume gets a policy started. Failure data for robot training is what gets it unstuck.

Why does a robot policy get stuck around 50% success?

Because small mistakes compound. A policy trained by imitation makes a tiny error, that error puts the robot in a state its training data never covered, and the next error is bigger.

Physical Intelligence explains it plainly in its write-up on π*0.6. A vision-language-action model controlling a robot will sometimes put the gripper in the wrong spot, miss a grasp or knock something over. Each slip creates a situation slightly different from anything in the demonstrations, where a bigger mistake becomes more likely. The small mistakes are fixable; the chain of them is what causes failure.

This is not a new finding. Back in 2010, Ross, Gordon and Bagnell showed that imitation learning breaks the usual assumption that training and test data come from the same distribution, because a policy's own actions shape what it sees next. They noted this leads to poor performance "in theory and often in practice" (arXiv:1011.0686).

Physical Intelligence also points out why this hits robots harder than chatbots. A language model produces a static output. A robot policy acts, sees the result, and acts again, so every error feeds into the next decision.

What is failure data for robot training?

Failure data is training data that keeps the moments where a task goes wrong, plus what happens next. That can mean a slipped grasp and the re-grasp, a dropped object and the pick-up, or a box that has to be refolded after a bad fold.

It comes in a few forms:

• Corrections: a human takes over when the robot makes a mistake and shows how to recover.
• Autonomous experience: the robot's own attempts, good and bad, scored by outcome.
• Labeled failure demonstrations: recorded failures with the cause written down.
• Human failure-and-recovery video: people performing the task, with natural slips and recoveries kept in the clip and labeled.

The common thread is that the recovery is the valuable part. A clip of a failure on its own teaches the policy what not to do. A clip of a failure and the fix teaches it how to get back on track.

Why don't more clean demonstrations fix the plateau?

Because clean demonstrations keep showing the policy states it already handles. They rarely show the off-track states the policy creates for itself in deployment, so adding more of them does little for the failures you actually see.

Physical Intelligence makes the point directly: if you train a policy to copy its own past behavior, you just teach it to repeat the same mistakes. The useful signal has to come from what should have happened after the mistake. That is why their RECAP method uses an expert teleoperator who takes over when the robot errs. The correction supervises exactly the situations the policy puts the robot into, which ordinary demonstrations never reach.

The results show what that missing data is worth. Training π*0.6 with corrections and on-robot experience more than doubled throughput on some of the hardest tasks and cut failure rates by 2x or more, compared with earlier training stages of the same model. On espresso making, both throughput and success rate more than doubled once on-robot experience was added (Physical Intelligence). The final model ran espresso drinks from 5:30am to 11:30pm and assembled 59 factory packaging boxes without interruption.

Note what that comparison measures: one model at different training stages, all on Physical Intelligence's own robots and tasks. It shows that data from off-track states moves the ceiling. It does not tell you how much any particular dataset will move yours.

What kinds of failure data actually help?

The best failure data covers the mistakes your robot really makes, keeps the recovery attached, and records why the failure happened. Each source does part of that job.

• Teleoperated corrections: Highest relevance, because they start from your policy's own errors. They cost operator time on your hardware, and Physical Intelligence notes their quality depends on a person spotting the right moment to step in.
• On-robot experience with rewards: Scales without a human at every step, but needs a way to tell which action caused the outcome. In Physical Intelligence's espresso example, a failed portafilter insertion was really caused by a bad grasp earlier on.
• Labeled failure datasets: RoboMIND, a teleoperation dataset of 107k trajectories across 479 tasks and four robot types, includes 5k real-world failure demonstrations, each with its cause recorded (arXiv:2412.13877). Public datasets are a good way to test whether failure examples help your setup before you pay for collection.
• Human failure-and-recovery demonstrations: First-person video of people doing the task, where slips, occlusions and recoveries stay in the episode. This is cheaper to scale than robot time and shows recovery behavior in varied real settings, but it is human data, not your robot's own mistakes.

Most teams will need more than one. Human and dataset sources broaden coverage of edge cases; on-robot corrections target the specific failures your policy produces.

How do you collect edge case data without wasting hours?

Decide which failures matter before anyone presses record. Edge case data collection for AI goes wrong when teams capture "messy footage" and hope the right moments are in it.

1. List your real failure modes. Pull them from deployment logs: slipped grasps, objects behind the hand, low light, items stuck together.
2. Turn each one into a named scenario. "Re-grasp after a mid-lift slip in dim light" can be counted. "Some failures" cannot.
3. Keep the failure and the recovery in one continuous episode. If the clip is cut at the drop, the policy loses the context for why the recovery was needed.
4. Label the recovery as its own step. A separate action segment lets you weight, filter or train on recoveries directly.
5. Record contact and visibility. Knowing when the hand lost contact, and which joints were hidden, tells the model what actually went wrong.
6. Check coverage per scenario, not in total. One hour across ten scenarios is not the same as six minutes in each. Count each cell.

What can Firsthand's failure data do, and what can't it?

Firsthand collects first-person human demonstrations to a written skill spec that names failure scenarios, such as occlusion angles, lighting conditions and recovery cases, before capture starts. Contributors don't pause or re-stage, so a drop and the re-grasp stay in one episode. The recovery gets its own labeled action segment, a contact_events stream flags loss of contact, and hand pose is tracked at 21 joints per hand with occluded joints flagged rather than guessed (Firsthand).

Streams are synchronized to under 2 ms, data ships in RLDS, LeRobot or WebDataset, and the license is buyer-owned. Where coverage is deep, the first validated batch arrives in a median of 48 hours from a signed spec.

Here is what it does not do:

• No robot or teleoperation data. Firsthand captures human demonstrations. It cannot give you corrections on your own robot's mistakes, which is the data behind Physical Intelligence's results.
• No force, pressure or tactile channels. If your failures are about grip force, this data will not show it.
• No success/failure flag or failure taxonomy. Recoveries are labeled as action segments; outcome labels are not included.
• No set failure ratio. How much failure data you need depends on your failure modes, and it is planned in the spec, not promised.
• Thin settings take longer. Environments with little coverage typically take three to four weeks, and long-tail failure coverage takes longer still.

The third-party results in this article, including Physical Intelligence's and RoboMIND's, were not measured on Firsthand data.

If human failure-and-recovery video fits your pipeline, start with our failure-case data collection spec.

The bottom line

A policy that plateaus near 50% is usually telling you it has never seen its own mistakes. More clean demonstrations add little. Data from off-track states, whether robot corrections, labeled failures or human recovery demonstrations, is what raises the ceiling. Name the failures you need, keep the recovery in the episode, and count coverage per scenario.