Mesh-based image stitching algorithm with linear structure protection

Chuan He; Jun Zhou

doi:10.11834/jig.170653

Image Processing and Coding | Views : 0 下载量: 137 CSCD: 10

PDF
Export
Share
Collection
Album

Mesh-based image stitching algorithm with linear structure protection
Vol. 23, Issue 7, Pages: 973-983(2018)
Received：22 December 2017，

Revised：2018-2-1，

Published：16 July 2018
DOI： 10.11834/jig.170653
稿件说明：

移动端阅览

DOI：

Chuan He, Jun Zhou. Mesh-based image stitching algorithm with linear structure protection[J]. Journal of Image and Graphics, 2018, 23(7): 973-983. DOI： 10.11834/jig.170653.

摘要

目的

基于网格变形的图像配准方式

针对待拼接图片重叠区域的视差具有一定的容忍性

并且能够适应更复杂的图像拼接场景。在NISwGSP (natural image stitching with the global similarity prior)算法基础上提出了一种具有直线结构保护的图像拼接算法(MISwLP)

该算法通过提取图片中的直线结构并施加约束

可以得到视觉效果自然、畸变较小的图像拼接结果。

方法

首先对图片进行网格划分

建立网格优化模型

针对网格顶点坐标集定义能量函数

在保证图片重叠区域高度对齐的同时

对网格进行相似性连续约束

并辅以直线结构约束

最后使用共轭梯度最小二乘法求解得到最优网格顶点集

指导网格变形。

结果

针对不同场景下的图片进行拼接实验

同时和几种比较流行的图像拼接软件和算法进行比较。结果表明

同经典拼接算法

比如Autostitch相比

基于网格优化的图像拼接算法能够适应更加复杂的多平面场景

在减小投影失真和对齐误差方面表现更好; 同现在比较好的几种网格拼接算法

比如SPHP (shape-preserving half-projective warps for image stitching)、APAP (as-projective-as-possible image stitching with moving DLT)、NISwGSP等的比较

MISwLP算法不仅能够很好地对齐图像和避免投影失真

并且能够保持图像重叠区域到非重叠区域的一致性

即保护原图中的直线结构。

结论

提出了一种基于网格优化的直线约束方法

对于具有显著几何结构的图像拼接场景

能够较好地保护拼接后图像中原有的直线结构

具有较好的应用价值。

Abstract

Objective

The image registration method based on mesh deformation can handle some parallax in the overlapping area of input images and can adapt to more complex scenarios where the scenery is not in the same plane.A new mesh-based image stitching with linear structure protection (MISwLP) is proposed.This algorithm applies constraints to the lines extracted from images to protect them from being distorted by the mesh deformation process

thus obtaining natural panoramas with reduced distortion.

Method

MISwLP is based on mesh deformation.Images are meshed with a set of vertices

and image deformation is guided by the indexed vertices.The algorithm can be implemented with four steps.The first step is called APAP (As-Projective-As-Possible Image Stitching with Moving DLT) pre-registration.The APAP algorithm is applied to align the images

and the feature matching pairs obtained by the APAP algorithm can be used to obtain all the vertex matching pairs in the overlapped area of the image matching pairs

which are called matching points.These matching points are distributed evenly and can be used to obtain good alignment capability for the mesh optimization model.The second step is called global similarity estimation.The relative 2D rotation angle and the relative scale between two images are estimated in this step.Then

a similarity transform between the two images can be constructed.In the third step

a mesh optimization model is established for the input vertices of the images.The mesh optimization process is implemented in two stages.In the first stage

the energy function includes three terms

namely

the alignment

local similarity

and global similarity terms

and the original vertices are used as the input for this function.It is solved by the least-squares conjugate gradient method.The first stage aims to align the images.Then

the outputs of the first stage are used as the input vertices of the second stage.In the second stage

a new term called line protection is added for further optimization.The lines are extracted by the LSD algorithm with a threshold or user-guided interface and then sampled across the grid.The line protection term constrains the sample points in a straight line.The optimization solution is computed with a sparse matrix effectively.At this time

the distorted lines in the first stage of this step are straightened.In the fourth step

a texture mapping method is applied by affine transforming the input grids into the output grids.All images are blended with a linear blending method.

Result

The performance of MISwLP is verified using images captured from different sceneries by handheld devices

such as mobile phones and digital cameras

and several open datasets.The scenes include urban and nature sceneries.MISwLP can handle more complicated image stitching tasks

in which the scenery consists of two planes

than image-stitching algorithms

which use only one global holography

such as AutoStitch.Furthermore

MISwLP produces a natural stitching result and reduced projective distortion.In addition

MISwLP outperforms several state-of-the-art methods

such as SPHP (Shape-Preserving Half-Projective Warps for Image Stitching)

APAP

and NISwGSP (Natural Image Stitching with the Global Similarity Prior).These algorithms use similarity transform to protect the non-overlapping area from projective distortion.Consequently

inconsistency is introduced between the overlapped and non-overlapped areas.The human eyes can perceive the destruction of certain geometry structures of the transitional area.MISwLP handles this problem with a line protection term and provides a good result with only a few geometry distortions.The proposed method works especially well for urban sceneries that contain many linear structures.For sceneries with no evident geometry

a user-guided auxiliary method is provided for selecting lines to protect.MISwLP is based on the NISwGSP algorithm

but the experiments show that their time complexities are nearly the same.

Conclusion

The performance of the proposed method is superior to those of state-of-the-art image-stitching methods.MISwLP protects the linear structure in the image stitching process

thereby providing a good stitching result with no geometry and projective distortions.Therefore MISwLP has good application value.

关键词

Keywords

references

Brown M, Lowe D G.Automatic panoramic image stitching using invariant features[J].International Journal of Computer Vision, 2007, 74(1):59-73.[DOI:10.1007/s11263-006-0002-3]

Joshi H, Sinha K L.A survey on image mosaicing techniques[J].International Journal of Advanced Research in Computer Engineering&Technology, 2013, 2(2):1-11.

Igarashi T, Igarashi Y.Implementing as-rigid-as-possible shape manipulation and surface flattening[J].Journal of Graphics, Gpu, and Game Tools, 2009, 14(1):17-30.[DOI:10.1080/2151237x.2009.10129273]

Gao J H, Kim S J, Brown M S. Constructing image panoramas using dual-homography warping[C]//Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition. Springs, Colorado, USA: IEEE, 2011: 49-56. [ DOI:10.1109/cvpr.2011.5995433 http://dx.doi.org/10.1109/cvpr.2011.5995433 ]

Lin W Y, Liu S Y, Matsushita Y, et al. Smoothly varying affine stitching[C]//Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition. Colorado Springs, USA: IEEE, 2011: 345-352. [ DOI:10.1109/cvpr.2011.5995314 http://dx.doi.org/10.1109/cvpr.2011.5995314 ]

Zaragoza J, Chin T J, Brown M S, et al. As-projective-as-possible image stitching with moving DLT[C]//Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland, OR, USA: IEEE, 2013: 2339-2346. [ DOI:10.1109/CVPR.2013.303 http://dx.doi.org/10.1109/CVPR.2013.303 ]

Li S W, Yuan L, Sun J, et al. Dual-feature warping-based motion model estimation[C]//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE, 2015: 4283-4291. [ DOI:10.1109/iccv.2015.487 http://dx.doi.org/10.1109/iccv.2015.487 ]

Joo K, Kim N, Oh T H, et al. Line meets as-projective-as-possible image stitching with moving DLT[C]//Proceedings of 2015 IEEE International Conference on Image Processing. Québec City, Canada: IEEE, 2015: 1175-1179. [ DOI:10.1109/icip.2015.7350985 http://dx.doi.org/10.1109/icip.2015.7350985 ]

Liu F, Gleicher M, Jin H L, et al. Content-preserving warps for 3D video stabilization[C]//Proceedings of 2009 ACM SIGGRAPH. New Orleans, Louisiana, USA: ACM, 2009: #44. [ DOI:10.1145/1576246.1531350 http://dx.doi.org/10.1145/1576246.1531350 ]

Gao J H, Li Y, Chin T J, et al. Seam-driven image stitching[C]//Proceedings of 2013 EUROGRAPHICS. Girona, Spain: EUROGRAPHICS, 2013. [ DOI:10.2312/conf/eg2013/short/045-048 http://dx.doi.org/10.2312/conf/eg2013/short/045-048 ]

Zhang F, Liu F. Parallax-tolerant image stitching[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH, USA: IEEE, 2014: 3262-3269. [ DOI:10.1109/cvpr.2014.423 http://dx.doi.org/10.1109/cvpr.2014.423 ]

Lin K M, Jiang N J, Cheong L F, et al. SEAGULL: seam-guided local alignment for parallax-tolerant image stitching[C]//Proceedings of the 14th European Conference on Computer Vision-ECCV 2016. Amsterdam, The Netherlands: Springer, 2016: 370-385. [ DOI:10.1007/978-3-319-46487-9_23 http://dx.doi.org/10.1007/978-3-319-46487-9_23 ]

Achanta R, Shaji A, Smith K, et al.SLIC superpixels compared to state-of-the-art superpixel methods[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(11):2274-2282.[DOI:10.1109/tpami.2012.120]

Chang C H, Sato Y, Chuang Y Y. Shape-preserving half-projective warps for image stitching[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, Ohio, USA: IEEE, 2014: 3254-3261. [ DOI:10.1109/cvpr.2014.422 http://dx.doi.org/10.1109/cvpr.2014.422 ]

Lin C C, Pankanti S U, Ramamurthy K N, et al. Adaptive as-natural-as-possible image stitching[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, Massachusetts, USA: IEEE, 2015: 1155-1163. [ DOI:10.1109/CVPR.2015.7298719 http://dx.doi.org/10.1109/CVPR.2015.7298719 ]

Chen Y S, Chuang Y Y. Natural image stitching with the global similarity prior[C]//Proceedings of the 14th European Conference on European Conference on Computer Vision. Amsterdam, Netherlands: Springer, 2016: 186-201. [ DOI:10.1007/978-3-319-46454-1_12 http://dx.doi.org/10.1007/978-3-319-46454-1_12 ]

Li N, Xu Y F, Wang C.Quasi-homography warps in image stitching[J].IEEE Transactions on Multimedia, 2017.[DOI:10.1109/tmm.2017.2771566](in Press)

Xiang T Z, Xia G S, Bai X, et al. Image stitching by line-guided local warping with global similarity constraint[J]. arXiv: 1702. 07935, 2017.

Yao D, Zhou J, Xue Z. Homography matrix genetic consensus estimation algorithm[C]//Proceedings of 2010 International Conference on Audio Language and Image Processing. Shanghai, China: IEEE, 2010: 1139-1143. [ DOI:10.1109/icalip.2010.5685084 http://dx.doi.org/10.1109/icalip.2010.5685084 ]

Alert me when the article has been cited

提交

Multi-modality feature fusion-based wide field-of-view image generation

Survey of texture optimization algorithms for 3D reconstructed scenes

High-precision parallax image stitching method using feature points clustering

Feature detection algorithm of musculoskeletal ultrasound image and its application of image stitching

Efficient 3D animation shape manipulation with region smoothness preservation