Dominant sub-plane registration algorithm for large parallax image stitching

Jiale Xue; Meng Zhao; Zhe Zhang; Xu Cheng; Shengyong Chen

doi:10.11834/jig.170518

Image Processing and Coding | Views : 0 下载量: 32 CSCD: 4

PDF
Export
Share
Collection
Album

Dominant sub-plane registration algorithm for large parallax image stitching
Vol. 23, Issue 3, Pages: 323-332(2018)
Received：20 September 2017，

Revised：2017-11-15，

Published：16 March 2018
DOI： 10.11834/jig.170518
稿件说明：

移动端阅览

DOI：

Jiale Xue, Meng Zhao, Zhe Zhang, Xu Cheng, Shengyong Chen. Dominant sub-plane registration algorithm for large parallax image stitching[J]. Journal of Image and Graphics, 2018, 23(3): 323-332. DOI： 10.11834/jig.170518.

摘要

目的

针对图像拼接中大视差图像难以配准的问题，提出一种显性子平面自动配准算法。

方法

假设大视差图像包含多个显性子平面且每个平面内所含特征点密集分布。对该假设进行了验证性实验。所提算法以特征点分布为依据，通过聚类算法实现子平面分割，进而对子平面进行局部配准。首先，使用层次聚类算法对已匹配的特征点聚类，通过一种本文设计的拼接误差确定分组数目，并以各组特征点的聚类中心为新的聚类中心对重叠区域再聚类，分割出目标图像的显性子平面。然后，求解每个显性子平面的投影参数，并采用就近原则分配非重叠区域的单应性矩阵。

结果

采用公共数据集对本文算法进行测试，并与Auto-Stitching、微软Image Composite Editor两种软件及全局投影拼接方法（Baseline）、尽可能投影算法（APAP）进行对比，采用均方根误差作为配准精度的客观评判标准。实验结果表明，该算法在拼接大视差图像时，能有效地配准局部区域，解决软件和传统方法由误配准引起的鬼影、错位等问题。其均方根误差比Baseline方法平均减小55%左右。与APAP算法相比，均方根误差平均相差10%左右，但可视化配准效果相同且无需调节复杂参数，可实现自动配准。

结论

提出的显性子平面自动配准算法，通过分割图像所含子平面进而实现局部配准。该方法具有较高的配准精度，在大视差图像配准方面，优于部分软件及算法，可应用于图像拼接中大视差图像的自动配准。

Abstract

Objective

Image stitching technology can be used to combine multiple narrow-angle images with overlapping regions into a one wide-angle image and has a wide range of applications. Image registration is the core method of image mosaic technology. The traditional registration method requires shooting a picture to meet the ideal condition in which the photographing device is located in a fixed point in a three-dimensional space. However

the ideal condition is often hard to achieve

which results in a certain parallax between images. Image registration with a large parallax is highly difficult

which affects the quality of image mosaic. This paper proposes a novel dominant sub-plane automatic registration algorithm to achieve a large parallax image registration for image stitching.

Method

We propose the hypothesis that large parallax images contain multiple dominant sub-planes and that feature points contained in each plane are densely distributed. To prove the hypothesis

we register the images after clustering feature points. Experimental result shows that the hypothesis is applicable to large parallax images. The proposed algorithm based on the distribution of feature points performs the local registration on the sub-plane segmented by clustering. First

feature points of the input images are extracted by scale-invariant feature transform

and the nearest neighbor principle is used to pre-match the feature points. The random sample consensus algorithm is used to remove the mismatched feature points

and the overlapping area of the image is delimited by feature points. A canvas that is the same as the overlap region is initialized

and all correctly matched feature points are mapped to the canvas. The hierarchical clustering algorithm is used to group the matched feature points. The grouping number is determined by the stitching error designed in this study. The overlapping regions are re-clustered by taking the clustering centers of each group feature point as new clustering centers to obtain the dominant sub-plane. The projection parameters of each sub-plane are solved through their feature points

and the projection matrix of the non-overlapping region is assigned by the nearest neighbor principle. The proposed algorithm needs to project the image multiple times for different regions. Thus

the use of traditional bilinear interpolation in the projection process has a significant boundary effect

and the nearest neighbor interpolation method can effectively overcome this problem. Visual effects are improved significantly.

Result

The algorithm is tested on public datasets and compared with two commercial software

auto-stitching and Microsoft Image Composite Editor (ICE)

and two common methods

namely

baseline and as-projective-as-possible algorithm (APAP). Root mean square error (RMSE) was used as an

objective

criterion for registration accuracy. Experimental results show that

compared with the baseline algorithm

the average RMSE of the proposed algorithm is reduced by approximately 55%. Many misaligned areas are aligned. Compared with the APAP algorithm

the accuracy is close

and the visual effect of registration is the same. The method in this paper can realize automatic registration without adjusting complex parameters. Compared with auto-stitching and Microsoft ICE

the method can effectively register the local area and eliminate the ghosting and dislocation caused by misalignment when stitching large parallax images.

Conclusion

The proposed dominant sub-plane automatic registration algorithm achieves local registration by segmenting sub-planes contained in the image. Using the sub-plane local registration method can effectively overcome the global registration error. The area prone to ghosting and other issues can often be separated as a separate plane registration area that improves the accuracy of the local registration. This method has high registration accuracy for large parallax images and is better than some commercial software and advanced algorithms. The method can be applied to the registration of large parallax images in image mosaic. The proposed method is based on plane approximation and allocates the projection matrix of the non-overlapping region using the proximity principle. Thus

some accumulative distortion may be observed in the non-overlapping region of the images because of projection transformation

which affects the mosaic of multiple images. This paper combines some excellent image transformation methods to address projection distortion in non-overlapping regions and apply the algorithm in a panoramic image mosaic.

关键词

Keywords

references

Kim D H, Yoon Y I, Choi J S. An efficient method to build panoramic image mosaics[J]. Pattern Recognition Letters, 2003, 24(14):2421-2429.[DOI:10.1016/S0167-8655(03)00071-0]

Wang Z C, Chen Y F, Zhu Z W, et al. An automatic panoramic image mosaic method based on graph model[J]. Multimedia Tools and Applications, 2016, 75(5):2725-2740.[DOI:10.1007/s11042-015-2619-0]

Zaragoza J, Chin T J, Brown M S, et al. As-projective-as-possible image stitching with moving DLT[C]//Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland, OR, USA: IEEE, 2013: 2339-2346. [ DOI: 10.1109/CVPR.2013.303 http://dx.doi.org/10.1109/CVPR.2013.303 ]

Lin C C, Pankanti S U, Ramamurthy K N, et al. Adaptive as-natural-as-possible image stitching[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA: IEEE, 2015: 1155-1163. [ DOI: 10.1109/CVPR.2015.7298719 http://dx.doi.org/10.1109/CVPR.2015.7298719 ]

Gao J H, Kim S J, Brown M S. Constructing image panoramas using dual-homography warping[C]//Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition. Colorado Springs, CO, USA: IEEE, 2011: 49-56. [ DOI: 10.1109/CVPR.2011.5995433 http://dx.doi.org/10.1109/CVPR.2011.5995433 ]

Zaragoza J, Chin T J, Tran Q H, et al. As-projective-as-possible image stitching with moving DLT[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(7):1285-1298.[DOI:10.1109/TPAMI.2013.247]

Chang C H, Sato Y, Chuang Y Y. Shape-preserving half-projective warps for image stitching[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH, USA: IEEE, 2014: 3254-3261. [ DOI: 10.1109/CVPR.2014.422 http://dx.doi.org/10.1109/CVPR.2014.422 ]

Chen Y S, Chuang Y Y. Natural image stitching with the global similarity prior[C]//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, Netherlands: Springer, Cham, 2016: 186-201. [ DOI: 10.1007/978-3-319-46454-1_12 http://dx.doi.org/10.1007/978-3-319-46454-1_12 ]

Liu H, Gu X Q, Gu X S. Thermal image stitching based on robust feature matching[J]. Infrared Technology, 2016, 38(1):10-20.

刘欢, 谷小婧, 顾幸生.基于鲁棒特征匹配的热成像全景图生成方法[J].红外技术, 2016, 38(1):10-20.[DOI:10.11846/j.issn.1001_8891.201601003]

蓝鲸. 层次聚类算法的原理及实现Hierarchical Clustering[EB/OL]. (2016-04-19)[2017-06-07] . http://bluewhale.cc/2016-04-19/hierarchical-clustering.html http://bluewhale.cc/2016-04-19/hierarchical-clustering.html .

Lowe D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2):91-110Hingham, MA:Kluwer Academic Publishers, 2004:91-110.[DOI:10.1023/B:VISI.0000029664.99615.94]

Bay H, Tuytelaars T, Van Gool L. SURF: speeded up robust features[C]//Proceedings of the 9th European Conference on Computer. Graz, Austria: Springer, 2006: 404-417. [ DOI: 10.1007/11744023_32 http://dx.doi.org/10.1007/11744023_32 ]

Szeliski R. Image alignment and stitching:a tutorial[J]. Foundations and Trends ® in Computer Graphics and Vision, 2006, 2(1):1-104.[DOI:10.1561/0600000009]

Li Q, Jiang Z T. Binary quantized SIFT feature descriptors for optimized image stitching[J]. Journal of Image and Graphics, 2016, 21(12):1593-1601.

李倩, 江泽涛.二值化的SIFT特征描述子及图像拼接优化[J].中国图象图形学报, 2016, 21(12):1593-1601.[DOI:10.11834/jig.20161204]

Levin A, Zomet A, Peleg S, et al. Seamless image stitching in the gradient domain[C]//Proceedings of the 8th European Conference on Computer Vision. Prague, Czech Republic: Springer-Verlag, 2004: 377-389. [ DOI: 10.1007/978-3-540-24673-2_31 http://dx.doi.org/10.1007/978-3-540-24673-2_31 ]

Gribbon K T, Bailey D G. A novel approach to real-time bilinear interpolation[C]//Proceedings of the 2nd IEEE International Workshop on Electronic Design, Test and Applications. Perth, WA, Australia: IEEE, 2004: 126-131. [ DOI: 10.1109/DELTA.2004.10055 http://dx.doi.org/10.1109/DELTA.2004.10055 ]

Ni K S, Nguyen T Q. Adaptable K-nearest neighbor for image interpolation[C]//Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. Las Vegas, NV, USA: IEEE, 2008: 1297-1300. [ DOI: 10.1109/ICASSP.2008.4517855 http://dx.doi.org/10.1109/ICASSP.2008.4517855 ]

Autostitching: a new dimension in automatic image stitching[EB/OL]. [2017-06-07] . http://matthewalunbrown.com/autostitch/autostitch.html http://matthewalunbrown.com/autostitch/autostitch.html .

Microsoft. Image Composite Editor[EB/OL]. 2015-02-25[2016-8-20] . https://www.microsoft.com/en-us/download/details.aspx?id=52459 https://www.microsoft.com/en-us/download/details.aspx?id=52459 .

Zaragoza J, Chin T J, Brown M, et al. As-projective-as-possible image stitching with moving DLT[EB/OL]. [2016-06-10] . http://cs.adelaide.edu.au/~tjchin/apap http://cs.adelaide.edu.au/~tjchin/apap .

Alert me when the article has been cited

提交

High-precision parallax image stitching method using feature points clustering

Feature detection algorithm of musculoskeletal ultrasound image and its application of image stitching

Multi-modality feature fusion-based wide field-of-view image generation

Survey of texture optimization algorithms for 3D reconstructed scenes

Multimodal remote sensing image registration： a survey