360MVSNet: Deep Multi-view Stereo Network with 360° Images for Indoor Scene Reconstruction

Ching-Ya Chiu1     Yu-Ting Wu2     I-Chao Shen3     Yung-Yu Chuang1
National Taiwan University1     National Taipei University2     The University of Tokyo3    

Qualitative comparison of a real-world scene. We compared results generated by our method and other methods on a real-world scene. Our method uses only 11 equirectangular 360◦ images to reconstruct the scene, while other methods use 66 images. Compared to other methods, our method reconstructs the scene more completely.

Abstract
Recent multi-view stereo methods have achieved promising results with the advancement of deep learning techniques. Despite of the progress, due to the limited fields of view of regular images, reconstructing large indoor environments still requires collecting many images with sufficient visual overlap, which is quite labor-intensive. 360° images cover a much larger field of view than regular images and would facilitate the capture process. In this paper, we present 360MVSNet, the first deep learning network for multi-view stereo with 360° images. Our method combines uncertainty estimation with a spherical sweeping module for 360° images captured from multiple viewpoints in order to construct multi-scale cost volumes. By regressing volumes in a coarse-to-fine manner, high-resolution depth maps can be obtained. Furthermore, we have constructed EQMVS, a large-scale synthetic dataset that consists of over 50K pairs of RGB and depth maps in equirectangular projection. Experimental results demonstrate that our method can reconstruct large synthetic and real-world indoor scenes with significantly better completeness than previous traditional and learning-based methods while saving both time and effort in the data acquisition process.

Publication
Ching-Ya Chiu, Yu-Ting Wu, I-Chao Shen, Yung-Yu Chuang
360MVSNet: Deep Multi-view Stereo Network with 360° Images for Indoor Scene Reconstruction.
Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2023. BibTex
WACV 2023 Paper (8.3MB PDF)
Digital library

Supplemental
WACV 2023 supplementary document (18.5MB PDF)