Robotics / Humanoid

OmniH2OUniversal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning

Original paper PaperBridge Editorial7/26/2026

Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi · arXiv:2406.08858 · submitted 13 June 2024 · paper

OmniH2O asks how a person can direct a full-size humanoid without wearing a motion-capture suit or planning every joint. Its answer is to make body pose a shared language between several kinds of human input and one learned whole-body controller.

Human motion becomes practice material a humanoid can survive

Start at the source of the training signal. The researchers take human motion from AMASS, reshape it for the humanoid’s different body, and reject motions the robot cannot perform. They also create standing and squatting variants by holding the root and lower body fixed. The point is not to copy every human motion perfectly; it is to give the controller extra practice staying planted while the upper body reaches and manipulates. §3 and Fig. 2 of the paper

This is simulated practice built from retargeted motion, not a record of the robot safely performing every source motion in the physical world. The simulation evaluation covers about 14,000 augmented sequences, while the real-world tracking test later uses only 20 standing sequences. §4.1

A well-informed teacher trains a sensor-limited student

In simulation, the teacher can see the robot’s full pose and motion. The deployable student gets much less: the target positions of the head and two hands, plus a short history of joint readings, body orientation, and past actions. The teacher supplies the action it would take; the student adjusts until its own action is close. This lets difficult reinforcement learning happen with rich simulated information while the final controller uses signals available on a real robot. §3, “Teacher,” “Student,” and “Policy Distillation”

The one essential equation asks: how far is the student’s motor command from the teacher’s?

\mathcal{L}=\|{\bm{a}_{t}}^{\text{privileged}}-{\bm{a}_{t}}\|^{2}_{2}

At time $t$ , ${\bm{a}_{t}}^{\text{privileged}}$ is the teacher’s list of target joint angles, and ${\bm{a}_{t}}$ is the student’s list. Subtracting compares each joint; squaring makes opposite errors count the same and makes large misses matter more; adding the squared misses produces one loss $\mathcal{L}$ . If two joints miss by 1° and 2°, the loss contribution is $1^2+2^2=5$ . Training changes the student to lower this number. Zero means the commands match for that example, not that the robot is guaranteed safe. §3, Eq. under “Policy Distillation”

Three tracked points and recent memory become whole-body action

At deployment, VR, an RGB camera, or a motion generator can all produce the same compact request: where the head and hands should go. The student combines those three target points with 25 steps of recent robot readings, then outputs target joint angles for the body controller. The recent history acts like a few frames of video: motion can be inferred from change over time, so the policy does not need a separately measured global linear velocity. §3 and Table 1

This design trades information for accessibility. In simulation, 22 tracked points give lower position error than three, but the three-point policy still reaches a 94.10% tracking success rate and works with consumer VR hardware. “Universal” therefore means several interfaces can speak the same pose language; it does not mean every command is unambiguous or every sensor setup works equally well. Table 1(c) and §3

The robot demonstrations are broad, but the controlled tests are small

The real robot performs teleoperated sports, object handling, and human interaction, and the same tracking policy accepts VR, RGB, and language-generated motion goals. In the controlled 20-motion real-world test, OmniH2O lowers global joint-position error from H2O’s 87.33 mm to 47.94 mm. The authors also collect 40 minutes of teleoperated data across six tasks; on four evaluated tasks over 10 runs, their diffusion-policy setup averages 8 successes out of 10, versus 1 out of 10 for basic behavior cloning. §4.1–4.3, Tables 2–3

These results show a working research system, not general-purpose humanoid autonomy. The real-world tracking set contains only standing motions; the task trials are few; root odometry can be noisy; stairs remain future work; and the paper provides no safety guarantee for extreme pushes or unfamiliar motion goals. §5

Research appendix

What the system learns

The problem is framed as goal-conditioned reinforcement learning. At each time step, the controller receives robot information ${\bm{s}^{\text{p}}_{t}}$ , a motion goal ${\bm{s}^{\text{g}}_{t}}$ , and produces an action ${\bm{a}_{t}}$ . The action specifies target joint angles, which a proportional–derivative controller turns into motor actuation. The teacher is trained to maximize discounted reward:

\mathbb{E}\left[\sum_{t=1}^{T}\gamma^{t-1}r_t\right].

The question here is: which policy produces useful behavior over an entire attempt, rather than at one instant? $r_t$ scores step $t$ ; $T$ is the attempt length; and $0\leq\gamma\leq1$ controls how much later scores matter. Multiplying by $\gamma^{t-1}$ gives later rewards less weight when $\gamma<1$ , then the sum combines the attempt and the expectation averages over uncertain rollouts. Raising $\gamma$ makes distant consequences matter more; lowering it makes training more short-sighted. This equation defines the teacher’s learning objective, while the paper’s contribution lies mainly in the state, reward, data, and teacher–student design rather than a new RL optimizer.

Exact inputs and outputs

Component	Available input	Output	Role
Privileged teacher	Full simulated body positions, orientations, velocities, previous action, and full reference pose differences	Target body joint angles	Finds a strong action with information unavailable on hardware
Deployable student	25-step histories of joint positions and velocities, root angular velocity, gravity direction, past actions; three target points for head and hands	Target body joint angles	Reproduces teacher actions from real-world-accessible signals
Hand controller	Hand pose estimated from VR	Finger and wrist joint targets	Uses inverse kinematics and a separate low-level controller
Learning-from-demonstration policy	Egocentric image	A sequence of sparse motion goals, including hand commands	Chooses a short future motion; OmniH2O converts it to body actions

The real-system appendix reports a 50 Hz policy loop and about 20 ms latency. Finger and wrist targets are mapped separately from Vision Pro data. Appendix A

Numeric evidence

Simulation method, about 14k sequences	Success ↑	Global MPJPE ↓	Root-relative MPJPE ↓
Privileged teacher	94.77%	126.51 mm	70.68 mm
H2O	87.52%	148.13 mm	81.06 mm
OmniH2O student	94.10%	141.11 mm	77.82 mm

The student nearly matches the teacher’s success rate but not its position error. It improves over H2O on all listed measures. Success means the average reference deviation never exceeds 0.5 m during a sequence; it is not a task-completion or safety metric. Table 1 and §4.1

Real-world method, 20 standing motions	Global MPJPE ↓	Root-relative MPJPE ↓	Acceleration error ↓	Velocity error ↓
H2O	87.33 mm	53.32 mm	6.03 mm/frame²	5.87 mm/frame
OmniH2O	47.94 mm	41.87 mm	1.84 mm/frame²	2.20 mm/frame

The VIO linear-velocity variant could not complete the test because the robot fell. Learned MLP and GRU velocity estimators also performed slightly worse than the history-only OmniH2O policy. Table 2

Learning-from-demonstration comparison, average over four tasks	Successes per 10 runs ↑	MSE ↓
Behavior cloning	1.0	5.63×10⁻³
Diffusion Policy with DDIM	7.75	1.90×10⁻³
Diffusion Policy with DDPM	8.0	5.25×10⁻⁴
Sequence observations → sequence actions	8.75	9.91×10⁻⁴

The best success number and the best mean-squared error come from different ablations, so they should not be combined into one “best model.” Results average only 10 runs per task. Table 3 and Appendix J

Why standing and squatting data matter

Earlier H2O behavior could take small adjustment steps instead of standing still. OmniH2O fixes the root and lower body of selected retargeted sequences to create standing and squatting variants. The reward design also introduces a maximum-foot-height term with a curriculum, intended to help the policy distinguish standing from taking a deliberate step. These are paper-specific engineering choices supported by ablations, not general laws for every humanoid.

Source grounding and uncertainties

Title, authors, date, abstract, and subject areas: arXiv record.
Method and equations: paper §§3–4.
Exact tables, hardware details, reward weights, randomization ranges, and limitations: paper PDF.
The paper calls OmniH2O-6 the first released full-size humanoid whole-body loco-manipulation dataset “to the best of our knowledge.” This is an author-scoped priority claim, not independently established here.
Demonstrations of sports, outdoor terrain, GPT-4o-selected primitives, and disturbance recovery are largely qualitative. They establish observed examples, not population-level reliability.
The main paper alternates between a PD target-angle description and an appendix hardware description that says torque information is dispatched. This report keeps the higher-level, consistently stated interpretation: the learned action supplies joint targets and the low-level control stack actuates the motors.

Terms

Retargeting: reshaping a human motion so a robot with different limb lengths and joint limits can attempt it.
Proprioception: measurements of the robot’s own joints and body orientation.
Privileged information: simulator-only measurements that are accurate or unavailable on the physical robot.
Distillation: training a smaller or more constrained student to match a stronger teacher’s outputs.
DAgger: repeatedly letting the student visit its own states, asking the teacher what to do there, and adding those examples to training.
Domain randomization: varying simulated friction, mass, delay, terrain, and disturbances so the controller is less tied to one exact simulator.
MPJPE: average distance between corresponding target and robot joints; lower is better.
Loco-manipulation: moving the body while using the arms or hands to affect objects.

Explicitly supported research relationships

OmniH2O extends H2O: Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation (arXiv:2403.04436): the paper reuses a similar AMASS retargeting process, treats H2O as the main teleoperation baseline, and removes H2O’s test-time motion-capture/global-velocity dependence. It adopts motion data from AMASS, teacher–student privileged learning supported by Lee et al.’s quadrupedal-locomotion work, and the DAgger data-aggregation framework for student supervision. For autonomous learning from collected demonstrations, it adopts Diffusion Policy as one evaluated high-level policy.

Verification questions

Why can three target points control more than three robot joints?
What extra information does the teacher have, and why is it withheld from the student?
What does the distillation loss measure, and what does a loss of zero fail to guarantee?
Why can a short history partly replace an explicit global velocity estimate?
Which result is based on about 14,000 simulated sequences, and which uses only 20 real motions?
Why do the demonstrations support breadth more strongly than reliability or safety?

Three key questions about this paper

What problem does OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning address?

Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi · arXiv:2406.08858 · submitted 13 June 2024 · paper

What evidence supports the main claim in OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning?

What limitation should readers know about OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning?

The one essential equation asks: how far is the student’s motor command from the teacher’s?