GScream: Learning 3D Geometry and Feature Consistent
Gaussian Splatting for Object Removal
ECCV 2024

Object Removal Demos

Left: Sampled multi-view images & masks. Right: Novel view synthesis with object removed.

img
img
img
img
img
img

Supplementary Video

Please refer to our supplementary video for more comparison and ablation studies. (It may take a few seconds to load. )

You could also watch the video on Youtube or Bilibili.




Abstract

This paper tackles the intricate challenge of object removal to update the radiance field using the 3D Gaussian Splatting. The main challenges of this task lie in the preservation of geometric consistency and the maintenance of texture coherence in the presence of the substantial discrete nature of Gaussian primitives. We introduce a robust framework specifically designed to overcome these obstacles. The key insight of our approach is the enhancement of information exchange among visible and invisible areas, facilitating content restoration in terms of both geometry and texture. Our methodology begins with optimizing the positioning of Gaussian primitives to improve geometric consistency across both removed and visible areas, guided by an online registration process informed by monocular depth estimation. Following this, we employ a novel feature propagation mechanism to bolster texture coherence, leveraging a cross-attention design that bridges sampling Gaussians from both uncertain and certain areas. This innovative approach significantly refines the texture coherence within the final radiance field. Extensive experiments validate that our method not only elevates the quality of novel view synthesis for scenes undergoing object removal but also showcases notable efficiency gains in training and rendering speeds.

Pipeline

img

Citation

@inproceedings{wang2024gscream,
     title={GScream: Learning 3D Geometry and Feature Consistent Gaussian Splatting for Object Removal},
     author={Wang, Yuxin and Wu, Qianyi and Zhang, Guofeng and Xu, Dan},
     booktitle={ECCV},
     year={2024}
     }
                

Acknowledgements

The website template was borrowed from Jon Barron Mip-NeRF.