Unsupervised Discovery of Object Radiance Fields
arXiv 2021

Abstract

Input image

Reconstruction

Object removal

Object insertion

Rearrangement

Background and object radiance fields

We study the problem of inferring an object-centric scene representation from a single image, aiming to derive a representation that explains the image formation process, captures the scene's 3D nature, and is learned without supervision. Most existing methods on scene decomposition lack one or more of these characteristics, due to the fundamental challenge in integrating the complex 3D-to-2D image formation process into powerful inference schemes like deep networks. In this paper, we propose unsupervised discovery of Object Radiance Fields (uORF), integrating recent progresses in neural 3D scene representations and rendering with deep inference networks for unsupervised 3D scene decomposition. Trained on multi-view RGB images without annotations, uORF learns to decompose complex scenes with diverse, textured background from a single image. We show that uORF performs well on unsupervised 3D scene segmentation, novel view synthesis, and scene editing on three datasets.

Video

Results

Unsupervised 3D segmentation

Input view

Novel view

GT color image

GT segmentation

Slot Attention

w/o background

w/o prog. trn.

uORF (ours)

Novel view synthesis

Input view

Novel view

Ground truth

NeRF-AE

w/o background

w/o prog. trn.

uORF (ours)

Scene manipulation

I. Move obj.

Input image

Input view

Novel view

Ground truth

Slot Attention

NeRF-AE

w/o bg.

w/o prog. trn.

uORF (ours)

II. Change bg.

Bg. image (input image is same as above)

Input view

Novel view

Ground truth

Slot Attention

NeRF-AE

w/o bg.

w/o prog. trn.

uORF (ours)

Acknowledgements

The website template was borrowed from Michaƫl Gharbi.