Stitching of parametric-free images based on coordinated image transformation and seam-line generation

Jiongli Gao; Jun Wu; Qichang Liu; Gang Xu

doi:10.11834/jig.190392

Image Understanding and Computer Vision | Views : 0 下载量: 44 CSCD: 3

PDF
Export
Share
Collection
Album

Stitching of parametric-free images based on coordinated image transformation and seam-line generation
Vol. 25, Issue 5, Pages: 1007-1018(2020)
Received：16 August 2019，

Revised：2019-10-11，

Accepted：18 October 2019，

Published：16 May 2020
DOI： 10.11834/jig.190392
稿件说明：

移动端阅览

DOI：

Jiongli Gao, Jun Wu, Qichang Liu, Gang Xu. Stitching of parametric-free images based on coordinated image transformation and seam-line generation[J]. Journal of Image and Graphics, 2020, 25(5): 1007-1018. DOI： 10.11834/jig.190392.

摘要

目的

为解决小重叠度、大视角影像自然拼接问题，提出一种统筹图像变换和缝合线生成过程的影像拼接新方法。

方法

主要包括两部分：1）将单应性映射引入到薄板样条变换（thin plate spline，TPS）并建立二次图像变换模型，在同一变换框架下进行全局单应性透视变换及基于径向基函数的局部映射调整，实现拼接影像透视特性良好保持与减少局部变形，且有效满足缝合线控制点无误差配准计算需要；2）根据控制点像素配准精度优于非控制点像素配准精度，对基准影像控制点集进行三角构网，并从中搜索初始缝合线，结合二次配准模型参数和动态规划匹配过程，获得控制点更密集且配准误差总体最小的最终缝合线，有效抑制图像融合时鬼影现象的产生，且计算实现上更简单。

结果

对两组标准影像进行拼接测试，并与3种方法（双单应性变换（dual-homography warping，DHW）、平滑变化的仿射拼接（smoothly varying affine stitching，SVA）、尽可能投影（as-projective-as-possible，APAP）的移动直接线性变换拼接（image stitching with moving direct linenr transform（DLT））对比，本文方法拼接影像局部变形失真最小、匹配控制点配准精度最高：在railtracks影像训练集（trarning set，TR）、测试集（test set）上分别为0和2.27；在temple影像训练集TR、测试集TE上分别为0和1.46；不同场景多路采集视频影像拼接透视特性保持良好，视觉效果自然。

结论

本文方法影像拼接时无先验知识要求、参数可线性求解，整体视觉效果流畅、重叠区域与非重叠区域过渡平滑，具有较好的应用价值。

Abstract

Objective

Image stitching technology aims to resample a group of interrelated (overlapping) images into a single image with a large angle of view and high resolution. This technology effectively overcomes the density limitation of a sensor array on the focal plane of common image acquisition equipment. Currently

numerous applications

such as realize virtual reality

security monitoring

and remote sensing detection

use the image stitching technology to obtain visual information of macro scenes and achievements have been made in the development of image stitching technology. However

due to the complexity of scene content

illumination conditions

and the changeability of camera pose

the acquisition of natural-looking visual effects of stitching image still depends mainly on post-processing interactively. Especially for applications with low measurement requirements and emphasis on visual experience

the parameter-free image stitching technology ignoring prior knowledge (scene 3D structure

camera parameters) is more valuable. To this end

this study presents a novel approach of stitching parametric-free images by comprehensively considering the process of image registration and fusion.

Method

The ghost phenomenon originates from an inaccurate image transformation model. Potential parameter estimation error has the greatest impact on the visual experience of stitching images. At present

the effective way to solve the problem is to find an optimal seam-line in overlapping regions of images to be stitched and take only one image pixel on each side of the seam-line for image fusion operation. Usually

the parameters of image transformation model are estimated using a set of control points located in overlapping regions of images. Thus

the accuracy of transformation at control points is better than that at non-control points. Accordingly

this study proposes to automatically generate an optimal seam-line from matched feature points for image stitching purpose. This technology mainly involves two aspects. The first one is the twice registration framework based on modified TPS (thin plate spline) transformation. It is established to meet the needs of error-free control point registration and automatic generation of reliable seam-line. In this step

the global homograph mapping and local mapping adjustment based on radial basis function are ingeniously incorporated into the uniform spatial transformation framework. Thus

it maintains perspective geometry and reduces local distortion in stitching image as much as possible. The second one is the reliable seam-line generated based on the fact that the registration accuracy of control point pixels is better than that of non-control point pixels in the process of image transformation. In this step

the initial seam-line is primarily searched out from triangulated control points in the reference image. Then

it is refined by adding more control points through dynamic programming matching process and remapping non-control point pixels with the transformation parameters estimated in advance. With the small registration error of pixels on seam-line

troublesome ghost phenomenon in stitching images can be effectively suppressed with computationally simpler image fusion operation.

Result

The experiments on two sets of standard images from the Internet are performed and compared with state-of-the-art methods in extant literature. The transformation accuracy of training set and test set of matching control points is found to be the highest in this study

whereas the local distortion of mosaic images is the smallest. Moreover

the stitching test is also performed on multiple videos with two different scenes. Its visual mosaic effect is quite natural.

Conclusion

The proposed method has a natural transition between images' overlapping and non-overlapping areas and achieves better visual effect in stitching images. Furthermore

it requires no prior knowledge on images

and its parameters can be solved linearly. Thus

it exhibits huge potential for application.

关键词

Keywords

references

Bookstein F L. 1989. Principal warps:thin-plate splines and the decomposition of deformations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(6):567-585[DOI:10.1109/34.24792]

Chang C H, Sato Y and Chuang Y Y. 2014. Shape-preserving half-projective warps for image stitching//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus: IEEE: 3254-3261[ DOI:10.1109/CVPR.2014.422 http://dx.doi.org/10.1109/CVPR.2014.422 ]

Chen H, Long A Q and Peng Y H. 2009. Building panoramas from photographs taken with an uncalibrated hand-held camera. Chinese Journal of Computers, 32(2):328-335

陈辉, 龙爱群, 彭玉华. 2009.由未标定手持相机拍摄的图片构造全景图.计算机学报, 32(2):328-335)[DOI:10.3724/SP.J.1016.2009.00328]

Cheng Z G and Zhang L. 2016. An aerial image mosaic method based on UAV position and attitude information. Acta Geodaetica et Cartographica Sinica, 45(6):698-705

程争刚, 张利. 2016.一种基于无人机位姿信息的航拍图像拼接方法.测绘学报, 45(6):698-705)[DOI:10.11947/j.AGCS.2016.20150567]

Chon J, Kim H and Lin C S. 2010. Seam-line determination for image mosaicking:a technique minimizing the maximum local mismatch and the global cost. ISPRS Journal of Photogrammetry and Remote Sensing, 65(1):86-92[DOI:10.1016/j.isprsjprs.2009.09.001]

Dornaika F and Chung R. 2004. Mosaicking images with parallax. Signal Processing:Image Communication, 19(8):771-786[DOI:10.1016/j.image.2004.06.008]

Du W Z, Song Z H, Yan W B and Wu L G. 2016. Fast image stitching under single camera rotate monitoring. Journal of Image and Graphics, 21(2):245-254

杜往泽, 宋执环, 闫文博, 吴乐刚. 2016.单摄像头旋转监控下的快速图像拼接.中国图象图形学报, 21(2):245-254)[DOI:10.11834/jig.20160214]

Duplaquet M L. 1998. Building large image mosaics with invisible seam lines//Proceedings of the SPIE 338, Visual Information Processing Ⅶ. Orlando: SPIE: 369-377[ DOI:10.1117/12.316427 http://dx.doi.org/10.1117/12.316427 ]

Fang X Y, Zhang M M, Pan Z G, Luo B and Wang P. 2007. Research of image mosaic based on graph cut. Journal of Image and Graphics, 12(12):2050-2056

方贤勇, 张明敏, 潘志庚, 罗斌, 王鹏. 2007.基于图切割的图像拼接技术研究.中国图象图形学报, 12(12):2050-2056)[DOI:10.11834/jig.20071203]

Gao J H, Kim S J and Brown M S. 2011. Constructing image panoramas using dual-homography warping//Proceedings of CVPR 2011. Colorado Springs: IEEE: 49-56[ DOI:10.1109/CVPR.2011.5995433 http://dx.doi.org/10.1109/CVPR.2011.5995433 ]

Guo D B, Lu Z Y, Jiao W D and Cui L L. 2009. Dynamic programming in segments for occlusion detection/stereo matching. Acta Electronica Sinica, 37(7):1516-1521

郭大波, 卢朝阳, 焦卫东, 崔玲玲. 2009.遮挡检测/立体匹配中的分段动态规划法.电子学报, 37(7):1516-1521)[DOI:10.3321/j.issn:0372-2112.2009.07.022]

Kerschner M. 2001. Seamline detection in colour orthoimage mosaicking by use of twin snakes. ISPRS Journal of Photogrammetry and Remote Sensing, 56(1):53-64[DOI:10.1016/S0924-2716(01)00033-8]

Kwatra V, Schödl A, Essa I, Turk G and Bobick A. 2003. Graphcut textures:image and video synthesis using graph cuts. ACM Transactions on Graphics (TOG), 22(3):277-286[DOI:10.1145/882262.882264]

Li J, Sheng Y H, Zhang K, Duan P and Wu H. 2013. Panoramic stitching method based on uncalibrated ordinary camera. Journal of System Simulation, 25(9):2070-2074

李佳, 盛业华, 张卡, 段平, 吴辉. 2013.基于未标定普通相机的全景图像拼接方法.系统仿真学报, 25(9):2070-2074)[DOI:10.16182/j.cnki.joss.2013.09.032]

Lin C C, Pankanti S U, Ramamurthy K N and Aravkin A Y. 2015. Adaptive as-natural-as-possible image stitching//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE: 1155-1163[ DOI:10.1109/CVPR.2015.7298719 http://dx.doi.org/10.1109/CVPR.2015.7298719 ]

Lin W Y, Liu S Y, Matsushita Y, Ng T T and Cheong L F. 2011. Smoothly varying affine stitching//Proceedings of CVPR 2011. Provi dence: IEEE: 345-352[ DOI:10.1109/CVPR.2011.5995314 http://dx.doi.org/10.1109/CVPR.2011.5995314 ]

Lowe D G. 2004. Distinctive image features from scale-invariant Keypoints. International Journal of Computer Vision, 60(2):91-110[DOI:10.1023/B:VISI.0000029664.99615.94]

Marzotto R, Fusiello A and Murino V. 2004. High resolution video mosaicing with global alignment//Proceedings of 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington: IEEE: #8161456[ DOI:10.1109/CVPR.2004.1315099 http://dx.doi.org/10.1109/CVPR.2004.1315099 ]

Qin X J, Wang Q, Wang H L, Zheng H B and Chen S N. 2015. Image fusion method based on best seam-line for serial remote sensing images mosaic. Computer Science, 42(10):306-310

秦绪佳, 王琪, 王慧玲, 郑红波, 陈胜男. 2015.基于最佳缝合线的序列遥感图像拼接融合方法.计算机科学, 42(10):306-310)[DOI:10.11896/j.issn.1002-137X.2015.10.062]

Rui Y K and Wang J C. 2007. A new study of compound algorithm based on sweepline and divide-and-conquer algorithms for constructing delaunay triangulation. Acta Geodaetica et Cartographica Sinica, 36(3):358-362

芮一康, 王结臣. 2007. Delaunay三角形构网的分治扫描线算法.测绘学报, 36(3):358-362)[DOI:10.3321/j.issn:1001-1595.2007.03.020]

Su M S, Hwang W L and Cheng K Y. 2004. Analysis on multiresolution mosaic images. IEEE Transactions on Image Processing, 13(7):952-959[DOI:10.1109/TIP.2004.828416]

Szeliski R. 1996. Video mosaics for virtual environments. IEEE Computer Graphics and Applications, 16(2):22-30[DOI:10.1109/38.486677]

Xiao F, Wu H Z, Xiao L and Tang Y. 2007. An ambient light independent image mosaic algorithm. Journal of Image and Graphics, 12(9):1671-1675

肖甫, 吴慧中, 肖亮, 汤杨. 2007.一种光照鲁棒的图像拼接融合算法.中国图象图形学报, 12(9):1671-1675)[DOI:10.11834/jig.20070929]

Zaragoza J, Chin T J, Brown M S and Suter D. 2013. As-projective-as-possible image stitching with moving DLT//Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland: IEEE: 2339-2346[ DOI:10.1109/CVPR.2013.303 http://dx.doi.org/10.1109/CVPR.2013.303 ]

Zaragoza J, Chin T J, Tran Q H, Brown M S and Suter D. 2014. As-projective-as-possible image stitching with moving DLT. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1285-1298[DOI:10.1109/TPAMI.2013.247]

Zhang Z X and Zhang J Q. 2012. Digital Photogrammetry. 2nd ed. Wuhan:Wuhan University Press

张祖勋, 张剑清.数字摄影测量学(第二版).武汉:武汉大学出版社, 2012