Self-Improving Vision-Language-Action Models with Data Generation via Residual RL

Wenli Xiao et al. · ICLR 2026 conference paper · Primary paper

A robot generalist may know roughly what to do yet still fail in the awkward states created by its own mistakes. This paper asks whether those failures can become useful training material—without asking people to demonstrate every recovery.

The frozen generalist reveals where help is actually needed

The starting VLA turns a camera view and a language goal into robot actions. It is frozen while the system watches where its own attempts go wrong: a cube pushed into a corner, a gripper approaching at a bad angle, or a drawer left half open. These are deployment-shaped failures, not mistakes imagined in advance by a human demonstrator. [Paper §§1, 3.1; Fig. 8]

This matters because polished human demonstrations often skip those awkward states. The paper’s claim is narrower than “human data is bad”: human data initializes the generalist, while PLD targets the gaps left by that data. [Paper §1]

A small residual specialist learns corrections, not the whole job

The real question is: if the generalist already moves roughly toward the goal, what is the smallest change that could rescue it? PLD keeps the base action $a_b$ and learns a correction $a_\delta$:

$$\bar{a}=a_b+a_\delta$$

The inputs are the current robot state and the base action. The operation adds the specialist’s correction to that base action, and the output $\bar a$ is the command sent to the robot. If $a_\delta=0$, the specialist changes nothing; as its magnitude grows, the robot can depart further from the generalist. For example, if the base proposes a 2 cm move right and the residual proposes 1 cm left, the combined move is 1 cm right. This lets lightweight reinforcement learning search near an already useful behavior instead of relearning vision, language, and control from scratch. [Paper Eq. 2; §3.1]

The specialist receives only a sparse success signal at the end of an attempt. PLD therefore mixes prior successful base-policy experience with new online trials while learning. The paper reports above 95% specialist performance on every one of more than 120 tasks used for later fine-tuning, but these are task-specific specialists rather than one new generalist. [Paper §§2.1, 3.1, 4.1]

Hybrid rollouts turn mistakes into recoveries

An expert that always starts from a clean setup produces narrow, polished trajectories. PLD instead lets the base policy act for a randomly chosen number of steps, then hands control to the residual specialist:

$$\pi(s_t)= \begin{cases} a_{\text{base}}, & t<T_{\text{base}}\\ a_{\text{base}}+a_{\delta}, & t\geq T_{\text{base}} \end{cases}$$

Here $t$ is the current step and $T_{\text{base}}$ is the handoff step. Before the handoff, the base model creates the kind of state it would encounter in deployment; after it, the combined policy demonstrates a recovery. A later handoff gives the base more time to drift and usually creates longer, more varied episodes. The paper finds that added diversity helps until performance saturates. [Paper §3.2; Algorithm 1; Fig. 4]

Only successful, deployment-shaped trajectories enter the lesson

The useful dataset is neither raw failure footage nor specialist-only perfection. It contains successful hybrid trajectories: the generalist’s approach, the difficult state it produced, and the specialist-assisted recovery. In the paper’s trajectory visualizations, specialist-only data is concentrated and far from base behavior, while PLD data stays nearer the base distribution and covers varied recoveries. [Paper §§3.2, 4.5; Fig. 10]

That alignment is a measured pattern; the explanation remains a hypothesis. The authors propose that fine-tuning on behavior close to the base policy causes less forgetting, but they do not prove that distribution distance alone causes the generalization gains. [Paper §4.5]

Distillation folds many local fixes back into one VLA

PLD’s last step is ordinary supervised fine-tuning: show the base VLA the curated observation–goal–action triples and train it to reproduce their actions. The temporary specialists are no longer needed at deployment. This makes the recipe compatible with both tested action-head families: the flow-based $\pi_0$ model and the autoregressive OpenVLA model. [Paper §§2.2, 3, 4.2]

The chart shows the strongest controlled simulation summary. On LIBERO, PLD raised average success from 93.4% to 97.2% for $\pi_0$, and from 91.8% to 99.2% for OpenVLA. On SimplerEnv, Octo rose from 71.8% to 96.6%. These are absolute gains of 3.8, 7.4, and 24.9 percentage points under the paper’s matched-budget comparisons. [Paper Tables 1–2]

Real robots recover better, but the evidence is still bounded

In 30 randomized cube-pickup trials, the PLD-trained policy succeeded 30/30 times, compared with 16/30 for RLPD-generated data and 10/30 for additional human data. All three reached 30/30 on peg insertion. The observed difference appeared when the cube entered a corner state: PLD had collected recoveries from that region. [Paper §4.4; Fig. 8]

The real-world study began with 200 teleoperated trajectories, then collected 200 successful trajectories per learned data source. The separate two-arm GPU system ran a full insertion-and-removal loop for at least one hour after at most eight hours of training per subtask, but individual stages were not always successful and a reward-classifier state machine coordinated them. The evidence supports robustness on these setups, not unrestricted autonomous improvement across robots, tasks, or safety-critical environments. [Paper §4.4]

Research appendix

What the paper contributes

PLD stands for Probe, Learn, Distill. “Probe” uses the frozen base VLA to visit its own likely states. “Learn” trains task-specific residual policies with off-policy reinforcement learning. “Distill” converts successful hybrid rollouts into supervised training data for the generalist. The contribution is mainly a data-generation and post-training pipeline, not a new VLA backbone.

Evidence ledger

Claim	Exact evidence	Scope and caveat
Specialist acquisition is sample-efficient	250k online interaction steps, 3 seeds, 8 selected LIBERO-90 tasks; above 95% on each of 120+ reported fine-tuning tasks	The 8-task curves are selected tasks; the broader statement is reported in text
LIBERO improves	$\pi_0$: 93.4→97.2 average; OpenVLA: 91.8→99.2	50 evaluation episodes per task; three 10-task suites
SimplerEnv improves	Octo-SFT: 71.8→96.6 average	Four tasks; “over 50% gains” refers to the largest task-level absolute increase, +50.6 points
Unseen-task transfer survives	At 10% LIBERO-90 task coverage, PLD unseen success is 24.4%; human data 21.4%; base rollout 5.6%	Four randomly sampled disjoint subsets per coverage level; unseen success is not high in absolute terms
Long-horizon transfer is incomplete	PLD beats self-bootstrapped data but remains below human demonstrations	One-shot evaluation on held-out LIBERO-10 after LIBERO-90 training
Franka cube pickup	PLD 30/30; RLPD 16/30; human 10/30	One robot setup and two task families; all methods got 30/30 on peg insertion
Two-arm long run	At least 1 hour without human reset	Four-stage state machine; each subtask trained up to 8 hours; stages can still fail

The value update behind the residual learner

The learner must answer: “After this action, how promising is the rest of the attempt?” The paper updates its action-value estimate with:

$$Q^{\bar{\pi}}(s_t,\bar a_t)\leftarrow r(s,a)+\gamma\, \mathbb{E}_{s_{t+1}\sim p(\cdot\mid s_t,\bar a_t)} \left[Q_{\text{target}}^{\bar{\pi}}(s_{t+1},\bar a_{t+1})\right], \qquad \bar a=a_b+a_\delta .$$

The inputs are the present state $s_t$, combined action $\bar a_t$, immediate reward $r$, and an estimate of future value at the next state. The expectation averages over possible next states; $\gamma$ controls how much future success matters. The output is a revised score $Q$ for the current choice. If $\gamma=0$, only immediate reward matters. As $\gamma$ rises toward 1, later success matters more—important when a grasp needs several unrewarded setup moves. PLD trains this critic from equal samples of an offline buffer of successful base rollouts and an online buffer of new interaction. [Paper Eq. 2; §3.1; Algorithm 1]

What supervised distillation optimizes

For an autoregressive action head, the paper gives:

$$\mathcal{L}_{\mathrm{AR}}(\theta)=- \mathbb{E}_{k\sim[K]} \left[\log p_\theta(u_k\mid u_{<k},x)\right].$$

The practical question is: “How strongly did the model expect the demonstrated next action token?” The input $x$ is the observation and language goal; $u_{<k}$ are earlier action tokens; $u_k$ is the demonstrated next token. Taking the logarithm turns a product of probabilities into an additive training score, and the minus sign makes high probability produce low loss. Training lowers this loss, so demonstrated PLD actions become more likely. The paper also supports diffusion and flow-matching heads with their corresponding standard losses; their full derivations are not needed to understand PLD. [Paper §2.2]

Ablations and causal boundaries

The paper compares PLD with methods that remove or alter ingredients such as residual guidance, offline-data replay, and base-policy probing. These ablations support the claim that policy-guided residual exploration and distribution-aware replay matter within the tested settings. They do not isolate every possible explanation: trajectory quality, diversity, closeness to the base distribution, and recovery coverage change together.

The authors explicitly frame lower forgetting as a hypothesis based on trajectory clustering and analogy to prior language-model observations. No formal guarantee says PLD must improve every unseen task. On long-horizon LIBERO transfer, human demonstrations still perform better.

Terminology

Term	Plain meaning here
VLA	A model that maps vision and a language instruction to robot actions
Generalist / base policy	The broad pretrained robot policy PLD starts from
Residual policy	A small policy that adds a corrective action to the base action
Off-policy RL	Learning from stored attempts, including attempts made by earlier policies
Sparse binary reward	One success/failure signal for an episode rather than guidance at every step
Rollout / trajectory	One time-ordered robot attempt
Distribution alignment	Training examples resemble states the deployed base policy actually visits
Distillation	Folding specialist-generated behavior into the generalist through supervised training

Explicit predecessors and adopted components

The base $\pi_0$ VLA comes from Black et al. (2024). The equal offline/online replay design builds on RLPD (Ball et al., 2023). Conservative critic initialization uses Cal-QL (Nakamoto et al., 2024), while the warm-up with a policy prior is compared with WSRL (Zhou et al., 2024). These links are stated in §§3.1, 4.1, and the implementation appendix; they should not be read as claims that those papers already contained the full PLD loop.

Source grounding and publication status

The official title, authors, abstract, paper text, tables, and equations were checked against arXiv:2511.00091 and its HTML full text. Publication status was checked against the ICLR 2026 paper page, which lists it as an ICLR 2026 Poster, and the published conference-paper record.

Verification questions

1. Why does a specialist-only rollout miss states that matter to the deployed generalist?
2. What changes before and after $T_{\text{base}}$ in a hybrid rollout?
3. Why can $a_\delta=0$ be useful when the base action is already good?
4. Which real-robot task separated PLD from the comparison data sources, and which did not?
5. What evidence suggests distribution alignment helps, and why is that evidence not a proof of causation?
6. Which result shows that PLD does not always beat human demonstrations?