Current Issue Cover
使用线约束运动最小二乘法的视差图像拼接

樊逸清1, 李海晟1, 楚东东2(1.华东师范大学计算机科学技术系, 上海 200062;2.中国银行软件中心上海分中心, 上海 201201)

摘 要
目的 图像配准是影响拼接质量的关键因素。已有的视差图象拼接方法没有解决匹配特征点对间的错误配准问题,容易引起不自然的拼接痕迹。针对这一问题,提出了使用线约束运动最小二乘法的配准算法,减少图像的配准误差,提高拼接质量。方法 首先,计算目标图像和参考图像的SIFT(scale-invariant feature transform)特征点,应用RANSAC(random sample consensus)方法建立特征点的匹配关系,由此计算目标到参考图像的最佳单应变换。然后,使用线约束运动最小二乘法分别配准两组图像:1)第1组是目标图像和参考图像;2)第2组是经单应变换后的目标图像和参考图像。第1组用逐点仿射变换进行配准,而第2组配准使用了单应变换加上逐点仿射变换。最后,在重叠区域,利用最大流最小割算法寻找最优拼接缝,沿着拼接缝评估两组配准的质量,选取最优的那组进行融合拼接。结果 自拍图库和公开数据集上的大量测试结果表明,本文算法的配准精度超过95%,透视扭曲比例小于17%。与近期拼接方法相比,本文配准算法精度提高3%,拼接结果中透视扭曲现象减少73%。结论 运动最小二乘法可以准确地配准特征点,但可能会扭曲图像中的结构对象。而线约束项则尽量保持结构,阻止扭曲。因此,线约束运动最小二乘法兼顾了图像结构的完整性和匹配特征点的对准精度,基于此配准模型的拼接方法能够有效减少重影和鬼影等人工痕迹,拼接结果真实自然。
关键词
Parallax image stitching using line-constraint moving least squares

Fan Yiqing1, Li Haisheng1, Chu Dongdong2(1.Department of Computer Science and Technology, East China Normal University, Shanghai 200062, China;2.Software Center Shanghai Branch, Bank of China, Shanghai 201201, China)

Abstract
Objective Image alignment is a key factor in assessing stitching performance. Image deformation is a critical step of the alignment model for parallax image stitching and directly determines the alignment quality. Accurately aligning all points in an overlapping region of parallax images is difficult. Thus, an alignment strategy that can produce visually satisfying stitching results must be developed. Recent state-of-the-art stitching methods practically combine homography with content-preserving warping. Either homography is first used to pre-align two images and is followed by content-preserving warping to refine alignment, or the mesh deformation is globally optimized by solving an energy function, which is a weighting linear combination of homography and content-preserving warping. Both approaches commonly use homography in the aligning phase and therefore easily produce perspective distortion. At the same time, these approaches possibly misalign the object edges of images with several dominant structural objects. To address these problems, this paper presented a novel stitching method that combines homography, deformation using moving least squares (MLS), and line constraint. The deformation method based on MLS has an interpolation property and can therefore accurately align matching feature points. However, this deformation method may distort structural regions; thus a line constraint item was added to the deformation model to preserve the structure. Method To attain a clear depiction, we considered a two-image stitching as an example. The two input images are called the target and reference images, respectively, and are denoted by T and R. First, feature detection and matching estimation were conducted using SIFT and RANSAC, followed by distance similarity to check the matching accuracy of the feature points. The homography (denoted by H) with the best geometric fit was selected. Then, H was applied to the target image T, and the transformed image was denoted as TH. Afterward, the two group images (T, R) and (TH, R) were aligned using a line constraint MLS. To eliminate perspective distortion in the deformation image, affine transformation was used in MLS. However, a simple affine transformation was insufficient to handle the parallax. Thus, an additional pair of images (TH, R) was processed as a candidate stitching result for the pair of images (T, R). The test experiments revealed that many examples obtained a more natural stitching result when only affine transformation rather than the composite transformation of homography and affine transformation was applied, implying that the alignment between T and R was better than that between TH and R. Taking the deformation from the target image T to the reference image R as an example, the line constraint MLS was outlined as follows. First, the four corner points of T were deformed to the coordinate system of R by using matching feature points as control points based on MLS. Then, we deformed the remaining points on the four border lines (top, bottom, left, and right boundaries) of T by using line constraint MLS. Here, the line constraint was constructed by preserving the relative position of each point of a border line, based on which a deformation objective function was developed. Similarly, we handled the internal points of T by using vertical and horizontal grid lines as constraint conditions, and the vertical and horizontal grid lines are consisted of the constraint lines of their intersection point. Finally, the quality of each alignment was evaluated, and the best one was chosen to blend them. In the overlapping regions, the max-flow min-cut algorithm was used to find the best stitching seam-cut of two alignments and assess the alignment quality along the seam-cut. The assessment of the alignment quality mainly considered the color and structural differences between overlapping regions of two images, and the structure was reflected by a gradient. Then, feathering approach was utilized to blend the two images of the best alignment. Result To test our stitching algorithm, 23 pairs of pictures, which cover commonly seen natural and man-made scenes, were captured. In addition, we conducted several experiments on publicly published data provided by recent related works. The experimental results demonstrated that the alignment accuracy of our method exceeded 95%, and the ratio of perspective distortion was lower than 17%. Compared with recent state-of-the-art methods, our method's alignment accuracy was higher by 3%, and the ratio of perspective distortion was lower by 73%. Therefore, our method exhibits a better performance in handling image stitching with a large parallax, and the stitching result is authentic and natural. Conclusion This paper presented a hybrid transformation for aligning two images that combines line constraint with MLS. In addition, an alignment quality evaluation rule was introduced by computing the weighted differences of the points along the stitching seam-cut and the remaining points in the overlapping region. As the proposed method can balance alignment accuracy and structure preservation, it can address the misalignment issues easily caused by current stitching approaches for parallax images and effectively reduce stitching artifacts, such as ghosting and distortion.
Keywords

订阅号|日报