Objective

2

Image alignment is a key factor in assessing stitching performance. Image deformation is a critical step of the alignment model for parallax image stitching and directly determines the alignment quality. Accurately aligning all points in an overlapping region of parallax images is difficult. Thus

an alignment strategy that can produce visually satisfying stitching results must be developed. Recent state-of-the-art stitching methods practically combine homography with content-preserving warping. Either homography is first used to pre-align two images and is followed by content-preserving warping to refine alignment

or the mesh deformation is globally optimized by solving an energy function

which is a weighting linear combination of homography and content-preserving warping. Both approaches commonly use homography in the aligning phase and therefore easily produce perspective distortion. At the same time

these approaches possibly misalign the object edges of images with several dominant structural objects. To address these problems

this paper presented a novel stitching method that combines homography

deformation using moving least squares (MLS)

and line constraint. The deformation method based on MLS has an interpolation property and can therefore accurately align matching feature points. However

this deformation method may distort structural regions; thus a line constraint item was added to the deformation model to preserve the structure.

Method

2

To attain a clear depiction

we considered a two-image stitching as an example. The two input images are called the target and reference images

respectively

and are denoted by

$$\mathit{\boldsymbol{T}}$$

and

$$\mathit{\boldsymbol{R}}$$

. First

feature detection and matching estimation were conducted using SIFT and RANSAC

followed by distance similarity to check the matching accuracy of the feature points. The homography (denoted by

$$\mathit{\boldsymbol{H}}$$

) with the best geometric fit was selected. Then

$$\mathit{\boldsymbol{H}}$$

was applied to the target image

$$\mathit{\boldsymbol{T}}$$

and the transformed image was denoted as

$${\mathit{\boldsymbol{T}}_H}$$

. Afterward

the two group images (

$$\mathit{\boldsymbol{T}}$$

$$\mathit{\boldsymbol{R}}$$

) and (

$${\mathit{\boldsymbol{T}}_H}$$

$$\mathit{\boldsymbol{R}}$$

) were aligned using a line constraint MLS. To eliminate perspective distortion in the deformation image

affine transformation was used in MLS. However

a simple affine transformation was insufficient to handle the parallax. Thus

an additional pair of images (

$${\mathit{\boldsymbol{T}}_H}$$

$$\mathit{\boldsymbol{R}}$$

) was processed as a candidate stitching result for the pair of images (

$$\mathit{\boldsymbol{T}}$$

$$\mathit{\boldsymbol{R}}$$

). The test experiments revealed that many examples obtained a more natural stitching result when only affine transformation rather than the composite transformation of homography and affine transformation was applied

implying that the alignment between

$$\mathit{\boldsymbol{T}}$$

and

$$\mathit{\boldsymbol{R}}$$

was better than that between

$${\mathit{\boldsymbol{T}}_H}$$

and

$$\mathit{\boldsymbol{R}}$$

. Taking the deformation from the target image

$$\mathit{\boldsymbol{T}}$$

to the reference image

$$\mathit{\boldsymbol{R}}$$

as an example

the line constraint MLS was outlined as follows. First

the four corner points of

$$\mathit{\boldsymbol{T}}$$

were deformed to the coordinate system of

$$\mathit{\boldsymbol{R}}$$

by using matching feature points as control points based on MLS. Then

we deformed the remaining points on the four border lines (top

bottom

left

and right boundaries) of

$$\mathit{\boldsymbol{T}}$$

by using line constraint MLS. Here

the line constraint was constructed by preserving the relative position of each point of a border line

based on which a deformation objective function was developed. Similarly

we handled the internal points of

$$\mathit{\boldsymbol{T}}$$

by using vertical and horizontal grid lines as constraint conditions

and the vertical and horizontal grid lines are consisted of the constraint lines of their intersection point. Finally

the quality of each alignment was evaluated

and the best one was chosen to blend them. In the overlapping regions

the max-flow min-cut algorithm was used to find the best stitching seam-cut of two alignments and assess the alignment quality along the seam-cut. The assessment of the alignment quality mainly considered the color and structural differences between overlapping regions of two images

and the structure was reflected by a gradient. Then

feathering approach was utilized to blend the two images of the best alignment.

Result

2

To test our stitching algorithm

23 pairs of pictures

which cover commonly seen natural and man-made scenes

were captured. In addition

we conducted several experiments on publicly published data provided by recent related works. The experimental results demonstrated that the alignment accuracy of our method exceeded 95%

and the ratio of perspective distortion was lower than 17%. Compared with recent state-of-the-art methods

our method's alignment accuracy was higher by 3%

and the ratio of perspective distortion was lower by 73%. Therefore

our method exhibits a better performance in handling image stitching with a large parallax

and the stitching result is authentic and natural.

Conclusion

2

This paper presented a hybrid transformation for aligning two images that combines line constraint with MLS. In addition

an alignment quality evaluation rule was introduced by computing the weighted differences of the points along the stitching seam-cut and the remaining points in the overlapping region. As the proposed method can balance alignment accuracy and structure preservation

it can address the misalignment issues easily caused by current stitching approaches for parallax images and effectively reduce stitching artifacts

such as ghosting and distortion.