UniFlow: A Unified Multi-Modal Policy for Robot Manipulation

Abstract

Visuomotor manipulation policies face a persistent trade-off: image-only policies inherit the semantic priors of large-scale 2D pretraining but are brittle to lighting, texture, and viewpoint shift, whereas 3D-only policies are robust to such factors yet are often trained from scratch. Some recent multi-modal policies combine both streams but typically process 3D through a from-scratch encoder, preserving this asymmetry. We present UniFlow, a unified 2D+3D co-training recipe for manipulation in which 3D observations are represented as image-shaped pointmaps (per-pixel 3D coordinates) so that pretrained 2D backbones can be used to encode both modalities, and RGB and pointmap streams are fused by a unified diffusion transformer with per-modality modulation. On average across thirteen dexterous manipulation tasks from the challenging Large Behavior Model (LBM) simulation benchmark, UniFlow outperforms a strong RGB-only baseline in a distribution-shifted setting. UniFlow also outperforms the generalist vision-language-action (VLA) model π_0.5 when averaged over five RoboTwin bimanual tasks as well as four real-world tasks on the bimanual YAM platform.

An image of geometry

A pointmap stores, at every pixel, the 3D point that the pixel sees. It has the same shape as the camera image and lines up with it pixel for pixel, so the same kind of pretrained 2D encoder that encodes RGB can encode geometry, with no point sampling or neighbor search. Below, every frame of a real demonstration is a pointmap: drag the divider to compare it with the RGB image, then lift it into 3D.

How it works

RGB and pointmap tokens stay separate streams all the way into a unified diffusion transformer. Each stream keeps its own modulation and feed-forward weights, and all of them meet in one joint self-attention with the noisy action tokens. Hover a block for details.

Simulation · LBM benchmark

Robust when the scene shifts

Thirteen dexterous tasks from the Large Behavior Model study (Barreiros et al., Science Robotics 2026), seven bimanual and six single-arm, in the Drake simulator. Every policy runs the same seeded scenes in-domain and under a domain shift that adds a second light, swaps the environment map, perturbs camera poses and intrinsics, re-textures objects and table, and in some scenes adds distractors. The LBM baseline is a four-camera variant of that study's RGB-only diffusion policy, trained on the same demonstrations.

Under shift, the RGB-only policy loses more than half of its success. UniFlow, which also encodes geometry, stays close to its in-domain level.

Same scene, four policies

Pick a task and a scene: all four policies start from the same randomized state. Besides LBM and UniFlow, the panel shows UniFlow's two single-modality variants, trained with the pointmap or the RGB stream removed.

Task by task

The gap is not carried by a few tasks: under shift, UniFlow is ahead of the RGB-only policy on every one of the thirteen.

One shift at a time

The full shift stacks several perturbations, and a camera-frame pointmap could plausibly be fragile to camera motion. Re-running UniFlow with one axis at a time on the same scenes shows otherwise: no single axis, the camera included, costs it more than a few points. Drag the divider to compare a nominal rollout with a shifted one.

Simulation · RoboTwin 2.0

One checkpoint, five bimanual tasks

RoboTwin 2.0 randomizes scene appearance, object placement and lighting across rollouts. Each method trains one multi-task checkpoint on 100 demonstrations per task: π_0.5 is fine-tuned on them from its large-scale robot data pretraining, while UniFlow has no robot data pretraining, only off-the-shelf pretrained encoders. UniFlow still has the best average, ahead of both the generalist VLA π_0.5 and the 3D policy ManiFlow, and it is best of the three on Handover block, Lift pot and, narrowly, Move can pot, which suggests its edge is on tasks that need precise 3D placement.

Real world · bimanual YAM

On a real robot

One multi-task UniFlow policy, trained on 191 teleoperated demonstrations (about an hour) across four rack tasks, against π_0.5 fine-tuned on the same data. Apart from its off-the-shelf pretrained encoders, UniFlow has seen no robot data beyond these demonstrations; π_0.5 starts from large-scale robot data pretraining. UniFlow reads a ZED 2i head camera and two ZED M wrist cameras, with pointmaps from on-device stereo depth. It reaches a higher average task score than π_0.5, and the difference is clearest where geometry matters most: passing a thin plate from one arm to the other.

What makes it work

Two ablations test the idea of treating 3D as an image: the pointmap stream beats point-cloud encoders, pretrained or not, and its 2D pretraining matters, since the same encoder started from random weights does much worse.

Conclusion

UniFlow represents 3D observations as pointmaps so that pretrained 2D backbones encode both RGB and geometry, and fuses the two streams with per-modality modulation. Across the LBM benchmark it holds its in-domain performance under domain shift, where an RGB-only policy does not, and with no robot data pretraining it outperforms the large-scale pretrained π_0.5 on average in RoboTwin and on four real-world bimanual tasks. Treating 3D as an image is a simple way to bring large-scale 2D pretraining to robust multi-modal robot policies.

BibTeX

@inproceedings{yan2026uniflow,
  title     = {UniFlow: A Unified Multi-Modal Policy for Robot Manipulation},
  author    = {Yan, Ge and Iwase, Shun and Fan, Yuzhi and Fox, Dieter and Zakharov, Sergey and Liu, Katherine},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026}
}