Grasp-MPCClosed-Loop Visual Grasping via Value-Guided Model Predictive Control

Jun Yamada, Adithyavairavan Murali, Ajay Mandlekar, Clemens Eppner, Ingmar Posner, and Balakumar Sundaralingam · ICRA 2026 · arXiv:2509.06201

A robot usually gets one predicted grasp and commits to it. Grasp-MPC gives the final approach a steering wheel: it repeatedly looks, tests possible short motions, and chooses the one that seems safest and most likely to end in a successful grasp.

A good plan can fail after the object moves

Notice the break between seeing and acting. An open-loop system predicts a grasp, plans once, and then follows that path. If the prediction is slightly wrong—or the object shifts—the robot has no way to repair the final approach. The paper identifies these two cases as central failure sources in clutter. [Paper §§I, IV-D]

This is not evidence that open-loop planning is useless. Grasp-MPC still uses a grasp predictor and a motion planner to reach a collision-free pre-grasp pose. Its new part controls only the last approach, where small errors matter most. [Paper Fig. 2 and §IV-D]

Two million attempts turn success and failure into a map

The training loop begins in simulation, with many digital objects and many perturbed starting poses. A motion planner produces paths from a pre-grasp pose toward an annotated grasp. Paths aimed at physically feasible grasps are marked as successes; the rest become failures. Together they form about 2 million trajectories. [Paper abstract and §IV-A]

The important teaching signal is the contrast, not just the scale. Successful and failed paths show the learner which nearby robot–object arrangements lead toward or away from a grasp. The authors did not physically execute every generated trajectory in simulation; labels come from grasp feasibility and planning, which speeds collection but leaves a simulation-to-reality gap. [Paper §IV-A]

The value function paints a “lower is better” landscape

The learner sees two things: a point-cloud view of the target object and the gripper’s pose relative to that object. It outputs one number: the expected remaining cost. Low cost means “this arrangement resembles a route toward success”; high cost means “success looks less likely from here.” [Paper §§IV-B, IV-E and Fig. 3]

The training question is: should the current estimate agree with what happens one step later? The paper uses a Bellman target, whose essential idea is

$$y_t = c_t + \gamma V_{\bar{\theta}}(o_{t+1}).$$

Here $o_{t+1}$ is the next visual-and-robot observation, $c_t$ is the immediate cost label, $V_{\bar{\theta}}$ estimates the cost still ahead, and $\gamma$ controls how much that future estimate matters. Add the immediate cost to the discounted future cost, and the result $y_t$ becomes the training target for the current observation. If the predicted future cost rises, the target rises too; the current state should therefore look less attractive. [Paper Eqs. 2–4 and §IV-B]

MPC balances grasp promise, collision risk, and smooth motion

At each control step, model predictive control (MPC) samples many short future motion sequences. It gives each sequence a combined bill: learned grasp cost, collision costs, and costs for violating joint limits or moving roughly. It executes only the first action from the cheapest sequence, then looks again. [Paper §§III, IV-C, Appendix G]

Think of three candidate motions with grasp, collision, and roughness costs of $(2,0,1)$, $(1,5,1)$, and $(3,0,0)$. If we add them with equal weight, their totals are 3, 7, and 3. The second motion looks best for grasping alone, but collision risk makes it unacceptable. The paper uses tuned weights rather than this toy equal-weight sum; the example only shows the job of combining costs. [Paper Eq. 6 and Appendix G]

Look, move a little, and look again

This loop is the paper’s central move. The camera updates the object point cloud; the value function rescoring changes which short motion looks promising; MPC sends one action; then the process repeats. A moved object can therefore bend the remaining path instead of making the original plan permanently wrong. [Paper §§IV-C–D and Fig. 9]

The controller also receives a signed-distance representation of the surrounding world, so correction is not “chase the object at any cost.” It must stay within robot limits and avoid self-collision and scene obstacles. The paper assumes the robot’s lower-level controller can accurately track the commanded joint targets. [Paper §III, §IV-D, Appendix G]

The clearest gain appears when the grasp prediction is imperfect

With grasp poses predicted by the off-the-shelf M2T2 model in FetchBench, Grasp-MPC reaches 67.2% success. The open-loop OSC controller reaches 63.6%, while the reported imitation-learning policies reach 36.5%. This is the most direct test of the intended setting: the initial grasp is useful but imperfect. [Paper §V-C and Fig. 5]

The boundary matters. Simulation covers 5,400 grasping problems, and real-robot tests use novel objects in empty-table, cluttered-table, and cluttered-shelf scenes. The abstract reports improvements of up to 32.6 percentage points in simulation and 33.3 points in noisy real-world conditions. But real-world closed-loop learning baselines were excluded for safety, dynamic perturbation tests evaluated only Grasp-MPC, training trajectories cover only the near-grasp phase, and validation is limited to grasping. [Paper abstract, §§VI–VII]

Research appendix

What exactly is trained?

The value network receives a centered segmented object point cloud plus the end-effector pose relative to the point-cloud centroid. PointNet++ encodes the point cloud; a multilayer perceptron encodes robot state; a final network produces a positive scalar value. Training mixes full and partial point clouds and adds Gaussian noise. The paper reports six days of training on one RTX 4090. [Paper §§IV-B, IV-E]

The sparse label marks terminal and near-terminal states of feasible grasp trajectories as low cost and other states as high cost. The current value estimate is fitted to the one-step target $y_t$ above by minimizing squared Bellman error. A slowly updated target network $V_{\bar{\theta}}$ makes that moving target more stable. The main text omits the paper’s nonessential parameter-update equations because they do not change the reader’s picture of the loop. [Paper Eqs. 1–4 and §IV-B]

What does the controller optimize?

The deployed controller uses MPPI, a sampling-based form of MPC implemented in CuRobo. It samples joint-acceleration sequences, rolls the robot state forward with Euler integration, scores the candidates, and applies the first action from the best sequence. Its cost contains the learned grasp value plus CuRobo terms for world collision, self-collision, joint bounds, and smoothness. [Paper §§III, IV-C, IV-E, Appendix G]

The paper’s main cost-combination equation is conceptually:

$$\text{total cost}=\text{standard CuRobo costs}+\lambda_{\text{grasp}}\times\text{learned grasp cost}.$$

This prose-preserving form is used because the accessible primary-source HTML drops several mathematical identifiers. The paper states a grasp-cost weight of 1000. Raising that weight makes the controller care more about the learned success landscape relative to the safety and smoothness terms; lowering it gives the standard motion costs more influence. [Paper Eq. 6 and §IV-C]

Evidence ledger

Claim	Exact evidence	Scope
Synthetic training scale	about 2 million grasp trajectories	Simulation-generated, success and failure
Predicted-grasp simulation	Grasp-MPC 67.2%; OSC 63.6%; IL policies 36.5%	M2T2 grasp poses in FetchBench
Benchmark size	5,400 grasping problems	Adapted FetchBench in Isaac Sim
Maximum reported gain	+32.6 points simulation; +33.3 points noisy real world	“Up to,” not every setting
Training compute	1 RTX 4090 for 6 days	Value-function training
Real-world setup	UR10 + Robotiq 2F-140; two RealSense L515 cameras	Three scene types

[Sources: paper abstract; §§IV-A, IV-E, V-C, VI-A–C, VII.]

What the experiments do and do not establish

All simulation methods share the same planner to reach the pre-grasp pose; the compared policy controls the subsequent grasp attempt. Grasp-MPC is tested with ground-truth, perturbed, and M2T2-predicted grasp poses. The paper reports that open-loop performance drops more under perturbation, while Grasp-MPC retains more of its ground-truth performance. [Paper §§V-A–C]

Real-world comparison is against CuRobo-GraspAPI, an open-loop pipeline. Other learned closed-loop baselines were not run on the real robot because the authors judged their lack of collision avoidance unsafe. The moving-object study has no competing method because open-loop control cannot respond by design. These choices support a practical demonstration, but they do not establish superiority over every safe closed-loop robot system. [Paper §§VI-A–C]

The paper also says real-world success remains below 100%, training labels do not come from physically executing every synthetic trajectory, the learned motions begin near the grasp, and evaluation covers grasping rather than broader manipulation. [Paper §§IV-A, VI-C, VII]

Terminology

Term	Plain meaning here
6-DoF grasp	A grasp whose position and three-dimensional orientation can both vary
Point cloud	A set of 3D points outlining the visible object surface
Value function	A learned score for how much cost likely remains before a successful grasp
MPC	Repeatedly compare short future motion plans, act once, then replan
Pre-grasp pose	A nearby staging pose reached before the final approach
Signed distance field	A map that tells how far locations are from obstacles
Open loop	Plan once and execute without using new observations to revise the plan
Closed loop	Observe during execution and use the new information to correct motion

Explicitly supported predecessor relationships

• M2T2: Multi-Task Masked Transformer for Object-Centric Pick and Place supplies the off-the-shelf grasp predictions used at deployment and in evaluation. [Paper refs. 1; §§IV-D, V-C]
• FetchBench: A Simulation Benchmark for Robot Fetching supplies the benchmark structure adapted for the simulation evaluation. [Paper ref. 14; §§V-A, VII]
• ACRONYM: A Large-Scale Grasp Dataset Based on Simulation supplies the antipodal-grasp sampling precedent used in data generation and evaluation annotations. [Paper ref. 12; §§IV-A, V-B]
• Neural Motion Fields: Encoding Grasp Trajectories as Implicit Value Functions is identified as a prior grasp-value approach based on distance to a predicted grasp; Grasp-MPC revises that idea with a visual value learned from success/failure trajectories. [Paper ref. 21; §II]
• Model Predictive Path Integral Control provides the sampling-based MPC method used through CuRobo. [Paper ref. 44; §IV-E]

Verification questions

1. Why can an accurate first grasp prediction still fail during execution?
2. What information goes into the value function, and what does a lower output mean?
3. Why are failed trajectories useful during training?
4. Why does MPC execute only the first action before planning again?
5. Which safety terms prevent the value function from pulling the gripper through an obstacle?
6. Which comparison is missing from the real-world experiments?