GRACE: Gradient-Free Robot Action Generation via Combined Diffusion–MPPI Posterior Mean Estimation
Abstract
Diffusion policies generate multimodal robot action sequences from demonstrations, but steering them toward deployment-time constraints typically relies on differentiable guidance costs. This excludes many practical safety constraints, such as binary collision checks, joint limits, and black-box rollout costs, that are nondifferentiable. We propose Gradient-free Robot Action generation via Combined diffusion–MPPI posterior mean Estimation (GRACE), which guides a pretrained diffusion policy with Model Predictive Path Integral (MPPI) control using only forward cost evaluations. Building on the common score-ascent structure of diffusion and MPPI, GRACE constructs a cost-conditioned guidance posterior at each reverse step and estimates its mean with a single MPPI update centered at the diffusion reverse mean. For differentiable costs, GRACE recovers conventional gradient guidance under a first-order, matched-covariance approximation. GRACE attains higher success rates and fewer total constraint violations than both gradient-guided diffusion policies and sampling-based planners in simulation, and we further validate it on real hardware.
Contributions
- MPPI-guided reverse diffusion. GRACE uses the score-ascent correspondence between diffusion and MPPI to incorporate sampling-based guidance into each reverse step, enabling adaptation to unseen deployment-time constraints without retraining.
- Posterior-mean guidance. GRACE estimates the mean of a cost-conditioned guidance posterior from cost-weighted rollouts, supporting nondifferentiable costs and recovering one-step gradient guidance under differentiability and matched covariance.
- Improved planning performance. GRACE achieves the highest success rates in both simulated tasks, reduces constraint violations relative to gradient-guided diffusion baselines, preserves prior multimodality, and transfers to real hardware.
Guidance on real hardware. The pretrained prior supplies the demonstrated manipulation behavior, while GRACE reweights forward-evaluated rollouts inside each reverse step to shift the posterior mean around an unseen obstacle—without retraining or differentiating the cost or dynamics.
Method
GRACE combines a guidance-independent diffusion prior learned offline with gradient-free, deployment-time cost evaluation. At every denoising step, the method performs the following posterior-mean update:
- Predict the diffusion reverse mean. The pretrained denoiser produces the reverse mean for the current noisy action sequence.
- Sample MPPI rollouts. Candidate action sequences are drawn from an exploration kernel centered at the diffusion reverse mean, with a guidance covariance that is controlled independently of the diffusion schedule.
- Estimate the guidance posterior mean. The rollouts are evaluated with the deployment-time cost and reweighted to estimate the mean of the cost-conditioned guidance posterior.
- Project and sample the next iterate. The estimated mean defines a projected reverse kernel that retains the diffusion schedule covariance, and the next denoising iterate is sampled from this kernel.
The construction follows from the shared score-ascent form of diffusion and MPPI. Under differentiable costs, a first-order approximation with matched guidance and reverse covariances recovers conventional one-step gradient guidance as a special case.
Simulation Experiments
Point-Mass Navigation
A planar point mass must reach the goal across a cluttered map without collision or timeout, where the gray obstacles were present during training but the red ones appear only at inference, and every clip below shares the same map, differing only in the planner.
CEM
MPPI
DA-MPPI
PO-DP
GG-DP
Diffusion-ES
GRACE (CNN, ours)
GRACE (UNet, ours)
| Method | Success [%] ↑ | Collision Fail ↓ | Path Length [m] ↓ |
|---|---|---|---|
| CEM | 8/30 (26.7%) | 0/30 | 1.247 ± 0.143 |
| MPPI | 8/30 (26.7%) | 0/30 | 1.241 ± 0.123 |
| DA-MPPI | 8/30 (26.7%) | 0/30 | 1.233 ± 0.120 |
| PO-DP | 18/30 (60.0%) | 12/30 | 2.657 ± 1.021 |
| GG-DP | 18/30 (60.0%) | 12/30 | 2.302 ± 1.115 |
| Diffusion-ES | 7/30 (23.3%) | 23/30 | 2.700 ± 0.832 |
| GRACE (CNN) | 20/30 (66.7%) | 0/30 | 3.681 ± 1.659 |
| GRACE (UNet) | 25/30 (83.3%) | 2/30 | 3.372 ± 1.538 |
Bold marks the best value per column over 30 random start–goal pairs, ties included. CEM, MPPI, and DA-MPPI remain collision-free but reach only 26.7% success because their uninformed proposals rarely discover the required detours. PO-DP and GG-DP improve success to 60.0% but each incurs 12 collision failures from local gradient refinement. Diffusion-ES reaches 23.3% success with 23 collision failures because its outer evolutionary loop continues to mutate clean samples from a prior that never observed the added obstacles. GRACE performs guidance in-loop: GRACE (CNN) reaches 66.7% with no collisions, while GRACE (UNet) achieves the best success rate at 83.3% with only two collisions.
FR3 Goal-Reaching
A 7-DoF Franka Research 3 (FR3) must reach goal joint configurations without violating obstacle, self-collision, floor-contact, or joint-limit constraints introduced only at inference, and every clip below shares the same start (green) and goal (red), differing only in the planner.
CEM
MPPI
DA-MPPI
PO-DP
GG-DP
Diffusion-ES
GRACE (CNN, ours)
GRACE (UNet, ours)
| Method | Success [%] ↑ | Computation Time [ms] ↓ | Guidance Time [ms] ↓ |
|---|---|---|---|
| CEM | 24/30 (80.0%) | 153.45 ± 2.55 | — |
| MPPI | 23/30 (76.7%) | 29.88 ± 0.49 | — |
| DA-MPPI | 23/30 (76.7%) | 154.22 ± 2.49 | — |
| PO-DP | 11/30 (36.7%) | 647.30 ± 15.83 | 220.01 ± 8.76 |
| GG-DP | 15/30 (50.0%) | 671.59 ± 9.00 | 244.72 ± 5.45 |
| Diffusion-ES | 12/30 (40.0%) | 1664.35 ± 6.72 | 217.87 ± 0.70 |
| GRACE (CNN) | 27/30 (90.0%) | 179.23 ± 1.91 | 90.53 ± 0.71 |
| GRACE (UNet) | 27/30 (90.0%) | 497.64 ± 7.08 | 85.88 ± 0.52 |
Bold marks the best value per column over 30 random obstacle layouts, and guidance does not apply to diffusion-free planners (—). CEM reaches 80.0% success, while MPPI and DA-MPPI each reach 76.7%. PO-DP and GG-DP reach 36.7% and 50.0%, respectively, as a single gradient direction must balance several competing safety penalties. Diffusion-ES reaches 40.0% and has the largest total computation time, 1664.35 ms, because it generates and refines multiple complete trajectories in an outer loop. GRACE reaches the best success rate, 90.0% for both backbones, with the lowest guidance times among diffusion-guided methods.
Real-World Experiments
A physical 7-DoF FR3, running the simulation prior with no retraining, must place a grasped cube at a randomly sampled spot on the shelf without hitting a red box that appeared in no demonstration, scored over 10 trials, and both clips below share the same prior and layout, differing only in the guidance.
Diffusion Policy. Reproduces the demonstrated placing motion faithfully, and drives the cube straight into the box it never saw.
GRACE (ours). The same prior, reweighted at each reverse step by the binary collision cost, detours around the box and still completes the placement.
| Method | Success [%] ↑ |
|---|---|
| Diffusion Policy | 0/10 (0.0%) |
| GRACE (Ours) | 9/10 (90.0%) |
Bold marks the best value per column over the 10 trials. The unguided policy hits the box in every trial while GRACE places the cube in 9 of 10, and since both rows run the identical prior on the identical hardware, that gap is the gradient-free guidance alone.
Appendix: Experimental Details
The following tables summarize the diffusion-policy training configuration and the principal hyperparameters used for each planning and guidance baseline.
A. Diffusion-Policy Training
A.1 Dataset and Sequence Configuration
| Domain | Training data | Observation horizon | Action horizon | Prediction horizon | Action dimension | Observation features |
|---|---|---|---|---|---|---|
| Point-Mass 2D | 10,000 demonstrations | 2 | 8 | 16 | 2 | Goal displacement, position, velocity |
| Franka Research 3 (FR3) | 20,000 trajectories | 1 | 16 | 32 | 7 | Goal joint displacement, joint position, joint velocity |
A.2 Shared Training Configuration
| Parameter | Value |
|---|---|
| Training epochs | 2,000 |
| Batch size | 256 |
| Optimizer | AdamW |
| Initial learning rate | 1 × 10−4 |
| Learning-rate schedule | Cosine |
| Warm-up steps | 500 |
| Weight decay | 1 × 10−6 |
| Observation normalization | Enabled |
| Action normalization | Min–max |
| Exponential moving average | Enabled |
| Diffusion scheduler | DDPM |
| Training diffusion steps | 100 |
| Inference denoising steps | 100 |
| Beta schedule | Squared cosine |
| Prediction target | Noise ε |
A.3 Denoiser Architectures
| Backbone | Architecture |
|---|---|
| UNet | Channel dimensions 256–512–1024; diffusion-time embedding 256; kernel size 5; GroupNorm with 8 groups |
| CNN | 3 residual blocks; hidden and conditioning dimensions 256; diffusion-time embedding 64; kernel size 5; GroupNorm with 8 groups |
B. Point-Mass 2D
B.1 Task Settings
| Parameter | Value |
|---|---|
| Action representation | Normalized 2D planar control action |
| Action dimension | 2 |
| Agent radius | 0.02 |
| Goal tolerance | 0.05 |
B.2 Sampling-Based Planners
CEM
| Parameter | Value |
|---|---|
| Population size | 1,000 |
| Optimization iterations | 5 |
| Elite fraction | 0.1 |
| Initial sampling std | 1.0 |
| Minimum sampling std | 0 |
MPPI
| Parameter | Value |
|---|---|
| Rollout samples | 1,000 |
| Optimization iterations | 1 |
| Sampling std | 1.0 |
DA-MPPI
| Parameter | Value |
|---|---|
| Rollout samples | 1,000 |
| Annealing iterations | 5 |
| Initial sampling std | 1.0 |
| Minimum sampling std | 0 |
| Inter-iteration std decay | 0.5 |
B.3 Diffusion-Based Methods
Gradient-Guided Diffusion
| Method | Backbone | Guidance stage | Gradient updates |
|---|---|---|---|
| PO-DP | UNet | After denoising | 25 |
| GG-DP | UNet | Final 5 denoising steps | 5 per guided step |
Diffusion-ES
| Parameter | Value |
|---|---|
| Backbone | UNet |
| Population size | 100 |
| Evolutionary search iterations | 3 |
GRACE
| Method | Backbone | Guided denoising steps | Rollouts per guided step | Guidance noise scale, σg |
|---|---|---|---|---|
| GRACE-CNN | CNN | Final 5 steps | 1,000 | 0.3 |
| GRACE-UNet | UNet | Final 5 steps | 1,000 | 0.3 |
C. Franka Research 3 (FR3)
C.1 Task Settings
| Parameter | Value |
|---|---|
| Goal joint-space tolerance | 0.03 rad |
| Action representation | delta_q |
| Action dimension | 7 |
C.2 Sampling-Based Planners
CEM
| Parameter | Value |
|---|---|
| Population size | 1,000 |
| Optimization iterations | 5 |
| Elite fraction | 0.1 |
| Initial sampling std | 0.03 rad |
| Minimum sampling std | 0 |
MPPI
| Parameter | Value |
|---|---|
| Rollout samples | 1,000 |
| Optimization iterations | 1 |
| Sampling std | 0.007 rad |
DA-MPPI
| Parameter | Value |
|---|---|
| Rollout samples | 1,000 |
| Annealing iterations | 5 |
| Initial sampling std | 0.03 rad |
| Minimum sampling std | 0 |
| Inter-iteration std decay | 0.5 |
C.3 Diffusion-Based Methods
Gradient-Guided Diffusion
| Method | Backbone | Guidance stage | Gradient updates |
|---|---|---|---|
| PO-DP | UNet | After denoising | 15 |
| GG-DP | UNet | Final 3 denoising steps | 5 per guided step |
Diffusion-ES
| Parameter | Value |
|---|---|
| Backbone | UNet |
| Population size | 100 |
| Evolutionary search iterations | 3 |
GRACE
| Method | Backbone | Guided denoising steps | Rollouts per guided step | Guidance noise scale, σg |
|---|---|---|---|---|
| GRACE-CNN | CNN | Final 3 steps | 1,000 | 0.3 |
| GRACE-UNet | UNet | Final 3 steps | 1,000 | 0.3 |
BibTeX
@article{anonymous2026grace,
title={GRACE: Gradient-Free Robot Action Generation via Combined Diffusion-MPPI Posterior Mean Estimation},
author={Anonymous Authors},
journal={IEEE Robotics and Automation Letters (RA-L)},
note={Submitted},
year={2026},
url={https://anonymous.4open.science/w/grace-70BB/}
}