CVPR 2023 - Unsupervised Continual Semantic Adaptation through Neural Rendering

In this episode we discuss Unsupervised Continual Semantic Adaptation through Neural Rendering by Zhizheng Liu, Francesco Milano, Jonas Frey, Roland Siegwart, Hermann Blum, Cesar Cadena. The paper proposes a method for continual multi-scene adaptation for semantic segmentation tasks, in which no ground-truth labels are available during deployment and performance on previous scenes must be maintained. The method involves training a Semantic-NeRF network for each scene by fusing the predictions of a segmentation model and using the view-consistent rendered semantic labels as pseudo-labels to adapt the model. The Semantic-NeRF model enables 2D-3D knowledge transfer and can be stored in long-term memory to reduce forgetting. The proposed approach outperforms both a voxel-based baseline and a state-of-the-art unsupervised domain adaptation method on the ScanNet dataset.