Abstract
Vision-Language-Action (VLA) models show strong generalization for robotic control, but finetuning them with reinforcement learning (RL) is constrained by the high cost and safety risks of real-world interaction. Training VLA models in interactive world models avoids these issues but introduces several challenges, including pixel-level world modeling, multi-view consistency, and compounding errors under sparse rewards. Building on recent advances across multimodal models and model-based RL, we propose VLA-MBPO, a practical world model-based RL framework to tackle these problems in VLA finetuning. Our approach is guided by three key design choices: (i) adapting unified multimodal models (UMMs) to VLA settings, leveraging rich multimodal priors to enable world modeling with limited data; (ii) introducing an interleaved view decoding mechanism to enforce consistency across views; and (iii) employing chunk-level branched rollout to limit rollout horizons and mitigate error compounding during policy optimization. Our theoretical analysis shows a reduction in value gap of VLA-MBPO, and experiments in both simulated and real-world tasks demonstrate that our method effectively improves policy performance and sample efficiency for VLA finetuning.
World Model Performance
UMM-World performance comparison and ablation studies.
As presented in the table above, UMM-World demonstrates superior performance across nearly all metrics. Not only does it outperform the video world model baseline on the prediction fidelity of both head and wrist views, but also achieves a 2× faster inference speed owing to the frame-skipping scheme. When it comes to reward prediction, our model also matches the specialist Qwen3-VL-8B. We credit these substantial gains to two key innovations: Interleaved View Decoding and Unified Pretraining. Ablation studies confirm that removing either component leads to a severe degradation in both dynamics modeling and reward prediction performance. We also present qualitative results of UMM-World in the figures above.
LIBERO Benchmark Performance
Performance comparison on LIBERO benchmark.
As shown in the table above, VLA-MBPO achieves consistent and substantial performance gains over all baselines across every suite of the LIBERO benchmark. It attains the highest average success rate, improving the initial SFT policy by +9.1%. Notably, the most pronounced improvement occurs on the LIBERO-Long suite, which comprises the most challenging long-horizon tasks. Under an identical real-world data budget, VLA-MBPO substantially outperforms both offline and online RL methods on this suite, highlighting its superior sample efficiency and effectiveness in solving complex manipulation tasks.
Real-World Performance
Performance comparison of real-world tasks on the Arx-X5 and Galaxea-R1 platforms.
As shown in the figure above, VLA-MBPO delivers consistent performance gains across both robotic platforms, validating its robustness in physical environments where dynamics are inherently stochastic. On Arx-X5, gains in Fold Towel confirm effective modeling of deformable-object dynamics, while improved Plug Cable performance highlights success in fine-grained, contact-rich manipulation. On Galaxy-R1, VLA-MBPO excels in high-DoF whole-body control, notably in Wipe Board task with severe partial observability. Strong results on both seen and unseen conditions further validate its real-world generalization.
Qualitative comparison of UMM-World prediction against Ground Truth across both simulated (LIBERO) and real-world tasks. The leftmost image of each task is the ground truth.
The learning dynamics of value models. The blue lines represent the ground-truth return, while the orange lines represent the return estimated by the value model.
Real World Tasks
fold the yellow towel
Insert the plug into the power strip
move to the whiteboard and clean the whiteboard
pick the pen and put it into the box
pick the cup and put it into the coaster
BibTeX
@article{zhang2026practicalworldmodelbasedreinforcement,
title={Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models},
author={Zhilong Zhang and Haoxiang Ren and Yihao Sun and Yifei Sheng and Haonan Wang and Haoxin Lin and Zhichao Wu and Pierre-Luc Bacon and Yang Yu},
year={2026},
eprint={2603.20607},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2603.20607},
}