Abstract

This paper tackles the intricate challenge of object removal to update the radiance field using the 3D Gaussian Splatting. The main challenges of this task lie in the preservation of geometric consistency and the maintenance of texture coherence in the presence of the substantial discrete nature of Gaussian primitives. We introduce a robust framework specifically designed to overcome these obstacles. The key insight of our approach is the enhancement of information exchange among visible and invisible areas, facilitating content restoration in terms of both geometry and texture. Our methodology begins with optimizing the positioning of Gaussian primitives to improve geometric consistency across both removed and visible areas, guided by an online registration process informed by monocular depth estimation. Following this, we employ a novel feature propagation mechanism to bolster texture coherence, leveraging a cross-attention design that bridges sampling Gaussians from both uncertain and certain areas. This innovative approach significantly refines the texture coherence within the final radiance field. Extensive experiments validate that our method not only elevates the quality of novel view synthesis for scenes undergoing object removal but also showcases notable efficiency gains in training and rendering speeds.

Pipeline

Concurrent works

Here are some concurrent works we found that also focus on 3D scene in-painting based on NeRF/3DGS (sorted by arxiv date):

• MALD-NeRF: Taming Latent Diffusion Model for Neural Radiance Field Inpainting
• RefFusion: Reference Adapted Diffusion Models for 3D Scene Inpainting
• InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior

Citation

@inproceedings{wang2024gscream,
     title={GScream: Learning 3D Geometry and Feature Consistent Gaussian Splatting for Object Removal},
     author={Wang, Yuxin and Wu, Qianyi and Zhang, Guofeng and Xu, Dan},
     booktitle={ECCV},
     year={2024}
     }
                

Acknowledgements

The website template was borrowed from Jon Barron Mip-NeRF.