Lishan Yang @catchieftain

🚀 We are excited to introduce 4D-WAM: Infusing Spatiotemporal Awareness into World Action Models through Trajectory Fields. Can WAMs understand not only how a scene looks, but also how its 3D structure evolves during robotic interaction? Our exploration of 4D-aware WAMs (1) Most WAMs jointly predict videos and robot actions in 2D latent space. However, robots operate in a dynamic 3D world. Frame-wise geometry alone is insufficient: manipulation requires understanding how 3D structures move and evolve over time—i.e., 3D + time, or 4D. (2) Interestingly, directly aligning WAM features with a 4D foundation model performs poorly because their representation spaces differ substantially. Our key finding is that their temporal feature changes exhibit much stronger correspondence than their absolute features. (3) Based on this insight, 4D-WAM transfers knowledge from 3D trajectory fields through two objectives: 🔹 Motion Alignment captures frame-to-frame dynamics. 🔹 Destination Alignment connects moving source regions with their final target regions. (4) 4D-WAM is a lightweight, model-agnostic post-training strategy. It aligns intermediate WAM representations without introducing extra reconstruction tasks or changing the original video-action objectives. The trajectory teacher is removed after training, adding no inference overhead. (5) 4D-WAM consistently improves FastWAM and Lingbot-VA across LIBERO and RoboTwin 2.0, with especially strong gains under OOD shifts: 📈 LIBERO-Plus: 62.21% → 71.01% 📈 Camera perturbations: +17.26 points 📈 RoboTwin Clean2Rand: 57.7% → 61.7% (6) It also generates more accurate and structurally consistent future videos on LIBERO-Plus: 🎥 PSNR: 19.22 → 21.44 🎥 SSIM: 0.75 → 0.83 4D-WAM better preserves robot motion, manipulated objects, target regions, and temporal consistency. (7) We further validate 4D-WAM on four challenging real-world manipulation tasks using an ARX LIFT2 robot, improving average task progress from 20.3% to 31.8%. WAMs should not only perceive 3D geometry—they should understand how it evolves toward a goal. 📄 arxiv.org/abs/2608.08023 💻 github.com/lishanyqy/4DWAM 🤗 huggingface.co/lishanyang/4dw…

keyboard_arrow_left Previous keyboard_arrow_right Next