Multi-Resolution Shared Representative Filtering for Real-Time Depth Completion

Multi-Resolution Shared Representative Filtering for
Real-Time Depth Completion

Yu-Ting Wu¹ Tzu-Mao Li² I-Chao Shen³ Hong-Shiang Lin⁴ Yung-Yu Chuang¹
National Taiwan University¹ MIT CSAIL² The University of Tokyo³ FIH Mobile Limited⁴

Qualitative comparison of Multi-Resolution Joint Bilateral Upsampling (M-JBU, Richardt et al. 2012), The Fast Bilateral Solver (FBS, Barron and Poole 2016), and our Multi-Resolution Shared Representative Filtering (M-SRF). M-JBU and M-SRF takes 7.5 ms. The Python version FBS takes 320.2 ms. Our method (M-SRF) can better adapt to hole shapes and preserve depth discontinuities than previous methods.

Abstract
We present shared representative filtering for real-time high-resolution depth completion with RGB-D sensors. Conventional filtering-based methods face a dilemma when the missing regions of the depth map are large. When the filter window is small, the filter fails to include enough samples. On the other hand, when the window is large, the method could oversmooth depth boundaries due to the error introduced by the extra samples. Our method adapts the filter kernels to the shape of the missing regions to collect a sufficient number of samples while avoiding oversmoothing. We collect depth samples by searching for a small set of similar pixels, which we call the representatives, using an efficient line search algorithm. We then combine the representatives using a joint bilateral weight. Experiments show that our method can filter a high-resolution depth map within a few milliseconds while outperforming previous filtering-based methods on both real-world and synthetic data in terms of both efficiency and accuracy, especially when dealing with large missing regions in depth maps.

Publication
Yu-Ting Wu, Tzu-Mao Li, I-Chao Shen, Hong-Shiang Lin, Yung-Yu Chuang.
Multi-Resolution Shared Representative Filtering for Real-Time Depth Completion.
Proceedings of High-Performance Graphics 2021. BibTex
HPG 2021 Paper (36.3MB PDF)
Digital library

Supplemental
HPG 2021 supplementary document (67.7MB PDF)
HPG 2021 presentation (11.2MB PPTX)
Web interactive comparison on synthetic data and Intel RealSense data
Web interactive comparison on Middlebury 2014 data

Last Update: July 2021