D3Fields: Dynamic 3D Descriptor Fields
for Zero-Shot Generalizable Robotic Manipulation

Yixuan Wang1*, Zhuoran Li2, 3*, Mingtong Zhang1, Katherine Driggs-Campbell1, Jiajun Wu2, Li Fei-Fei2, Yunzhu Li1, 2
1University of Illinois Urbana-Champaign, 2Stanford University, 3National University of Singapore
(* indicate equal contribution)
Paper ArXiv Video Code


D3Fields is a 3D, dynamic, and semantic scene representation using foundation models,
which supports zero-shot generalizable robotic manipulation.


D3Fields supports goal specification using 2D images with varied styles, contexts, and instances
and zero-shot generalizable manipulation applicable to a broad spectrum of household tasks.

Abstract

Scene representation has been a crucial design choice in robotic manipulation systems. An ideal representation should be 3D, dynamic, and semantic to meet the demands of diverse manipulation tasks. However, previous works often lack all three properties simultaneously. In this work, we introduce D3Fields — dynamic 3D descriptor fields. These fields capture the dynamics of the underlying 3D environment and encode both semantic features and instance masks. Specifically, we project arbitrary 3D points in the workspace onto multi-view 2D visual observations and interpolate features derived from foundational models. The resulting fused descriptor fields allow for flexible goal specifications using 2D images with varied contexts, styles, and instances. To evaluate the effectiveness of these descriptor fields, we apply our representation to a wide range of robotic manipulation tasks in a zero-shot manner. Through extensive evaluation in both real-world scenarios and simulations, we demonstrate that D3Fields are both generalizable and effective for zero-shot robotic manipulation tasks. In quantitative comparisons with state-of-the-art dense descriptors, such as Dense Object Nets and DINO, D3Fields exhibit significantly better generalization abilities and manipulation accuracy.



Video

D3Fields



Overview of the proposed framework. (a) The fusion process fuses RGBD observations from multiple views. Each view is processed by foundation models to obtain the feature volume \(\mathcal{W}\). Arbitrary 3D points are processed through projection and interpolation. (b) After fusing information from multiple views, we obtain an implicit distance function to reconstruct the mesh form. We also have instance masks and semantic features for evaluated 3D points, as shown by the mask field and descriptor field in the top right subfigure. (c) Given a 2D goal image, we use foundation models to extract the descriptor map. Then we correspond 3D features to 2D features and define the planning cost based on the correspondence.



Interactive Visualization

Visualize D3Fields for

Mask Field

Descriptor Field


Visualize tracking for

3D Tracking Visualization (Projected to Image Space)

3D Tracking Trace Visualization

BibTeX

@article{wang2023d3fields,
      title={D$^3$Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Robotic Manipulation},
      author={Wang, Yixuan and Li, Zhuoran and Zhang, Mingtong and Driggs-Campbell, Katherine and Wu, Jiajun and Fei-Fei, Li and Li, Yunzhu},
      journal={arXiv preprint arXiv:2309.16118},
      year={2023}
    }