目的 深度相机能够对场景的深度信息进行实时动态捕捉，但捕获的深度图像分辨率低且容易形成空洞。利用高分辨率彩色图像作为引导，是深度图超分辨率重建的重要方式。现有方法对彩色边缘与深度不连续区域的不一致性问题难以有效解决，在深度图超分辨率重建中引入了纹理复制伪影。针对这一问题，本文提出了一种鲁棒的彩色图像引导的深度图超分辨率重建算法。方法 首先，利用彩色图像边缘与深度图像边缘的结构相关性，提出RGB-D结构相似性度量，检测彩色图像与深度图像共有的边缘不连续区域，并利用RGB-D结构相似性度量自适应选取估计像素点邻域的最优图像块。接着，通过提出的定向非局部均值权重，在图像块区域内建立多边引导下的深度估计，解决彩色边缘和深度不连续区域的结构不一致性。最后，利用RGB-D结构相似性度量与图像平滑性之间的对应关系，对多边引导权重的参数进行自适应调节，实现鲁棒的深度图超分辨率重建。结果 在Middlebury合成数据集、ToF和Kinect数据集以及本文自建数据集上的实验结果表明，相比其他先进方法，本文方法能够有效抑制纹理复制伪影。在Middlebury、ToF和Kinect数据集上，本文方法相较于次优算法，平均绝对偏差平均降低约63.51%、39.47 %和7.04 %。结论 对于合成数据集以及真实场景的深度数据集，本文方法均能有效处理存在于彩色边缘和深度不连续区域的不一致性问题，更好地保留深度边缘的不连续性。

Objective Depth cameras can capture depth images of the dynamic scene in real time, which exhibits unique advantages in depth information acquisition. However, the depth images are often sensitive to noise and subject to low spatial resolution, and depth values are missing in some areas. Depth information and color edges are two complementary aspects that describe the same scene and therefore have a strong correlation in structure. Depth discontinuous transitions often coincide with color transitions. The color-depth correlation can be used for depth image reconstruction due to the blurred edges of low resolution depth images. Utilizing a high resolution color image as a reference is an important approach for reconstructing a high-resolution depth image. However, rich texture regions can be found in color images, except depth images. Among the highly challenging problems in color-guided depth image reconstruction is the inconsistency between color edges and depth discontinuities in the texture region. Simply passing structural information in the color image to the target one could introduce significant errors. Existing methods tend to consider only color images and ignore the correlation with depth images, which ineffectively solves the inconsistency, resulting in texture copying artifacts and even blurred depth edges. In this paper, we propose a color-guided depth image super-resolution reconstruction algorithm that is robust to inconsistency. Method We propose RGB-D structure similarity measure to predict the color edges that are most likely to coincide with depth discontinuities by using the structural correlation between the color and depth images. We examine local structural gradients rather than the gradient magnitude of individual pixel to measure the inconsistency effectively. Result shows that the proposed RGB-D structure similarity measure is less affected by color texture. We use the proposed RGD-D structure similarity measure as an image patch adaptive selection indicator, which can effectively reflect the discontinuity of the depth edges. The conventional image patch is centered on the pixel to be estimated. However, when the pixel is located in the depth edge region, the depth estimation is blurred due to the changes in the nearby depth gradients. In contrast with the conventional image patches, we select the optimal image patch that is least likely to contain prominent depth edges among all the image patches in the pixel neighborhood, which helps preserve sharp depth edges. Then, the multilateral guided estimation of depth values is performed in the selected optimal patch. We propose an oriented nonlocal means weighting scheme using high-quality structural gradients and directional information by utilizing the nonlocal characteristics of the color and the depth images. The weighting scheme combines with spatial and range kernel as the multilateral guidance for depth estimation, which effectively solves the structural inconsistency, preserves depth discontinuities, and is robust to depth holes. Finally, the three bandwidth parameters of our multilateral guidance weighting scheme are important benchmarks of our depth image super-resolution reconstruction model. The proposed RGB-D structure similarity measure is related to the depth image smoothness, which corresponds to depth discontinuity and is less affected by the incoherent texture. Small bandwidth parameters can effectively preserve depth discontinuities but poorly perform in smoothing noise. Large bandwidth parameters can effectively smooth noise but may blur depth discontinuities. We can adaptively adjust the multilateral guidance weight parameters in accordance with the RGB-D structure similarity measure to achieve robust depth image construction. The framework of our depth image super-resolution reconstruction is based on the multilateral guidance. Moreover, the corresponding relationship between the proposed RGB-D structure similarity measure and image smoothness is used to select the position of the neighborhood image patch and the size of the guide weight parameters adaptively. Result Quantitative and qualitative evaluation results show that our method has promising performance compared with other state-of-the-art methods on Middlebury synthetic, ToF real, and Kinect real datasets and our own dataset. Our method can effectively suppress texture copying artifacts, restore the depth hole image, and preserve depth discontinuities. We use the mean absolute difference as an evaluation metric, which is also the commonly used evaluation analysis metric for the depth image reconstruction. In particular, the mean absolute difference of the proposed method is decreased by approximately 63.51%, 39.47%, and 7.04% on the Middlebury, ToF, and Kinect datasets on average compared with the suboptimal algorithm. Furthermore, when the up-sampling factor of depth image reconstruction increases, the results of our reconstruction are more evident than those of other state-of-the-art methods as we fully utilize the structural information of color images. The other methods ineffectively solve the influence of color textures, whereas the depth image information is no longer reliable. For the depth hole image, most of the previous methods can only restore the depth image but cannot increase the depth image resolution, or the two can be separated. Our method can effectively restore the depth hole image, and the depth image super-resolution reconstruction and experiments on the NYU raw dataset verify the effectiveness of our method. Conclusion Our method can effectively handle the inconsistency between color edges and depth discontinuities in color-guided depth image super-resolution reconstruction, effectively restoring depth holes. In particular, our method can be used for not only synthetic datasets but also real-world depth datasets to preserve depth discontinuities effectively.