Sparse ActionGen: Accelerating Diffusion Policy with Real-time Pruning

K. Ji, J. Zhou, Y. Meng, Y. Li, H. Cui, Z. Wang

📄 Paper  ·  💻 Code  ·  🔗 Project

Diffusion Policy dominates action generation thanks to its strong modeling of multi-modal action distributions, but its multi-step denoising makes it impractical for real-time visuomotor control. Existing caching-based accelerators rely on static schedules that fail to adapt to the dynamics of robot-environment interactions, leading to suboptimal performance.

SAG (Sparse ActionGen) enables extremely sparse action generation:

• Rollout-adaptive prune-then-reuse. It first identifies prunable computations globally, then reuses cached activations to substitute them during action diffusion.
• Observation-conditioned diffusion pruner. A highly parameter- and inference-efficient pruner adapts to each rollout’s observations for environment-aware, real-time prediction.
• One-for-all reuse. Activations are reused across both timesteps and blocks in a zig-zag manner, minimizing global redundancy.

Across multiple robotic benchmarks, SAG achieves up to 4× generation speedup without sacrificing performance.