Remote sensing image registration is a key technology in the field of remote sensing image processing and has been widely used in the military and civil fields

such as natural disaster damage assessment

resource census

military damage assessment

environment monitoring

and ground target identification.Remote sensing image registration is the fundamental task in image processing that aligns two or more images of the same scene (i.e.

reference and sensed images)

which can be multitemporal (taken at different times)

multisource (derived from different sensors)

and multi-view (obtained from different viewpoints).It aims to acquire the geometric transformation relationship between the reference and sensed images accurately.Non-rigid geometric distortion occurs among multi-viewpoint remote sensing images because of the changes in aerial view and the complexities of the spatial distribution and geometric shapes of terrestrial objects

especially in roads

rivers

viaduct

buildings

mountains

and other linear areas

thereby increasing the difficulty of remote sensing image registration.Few methods have attempted to solve the distortion problem in multi-viewpoint remote sensing image registration；however

most of these methods mainly focused on solving the rigid and affine geometric distortion problems.Furthermore

only single feature was applied for point correspondence estimation. A new multi-viewpoint remote sensing image registration algorithm is proposed to address the multi-viewpoint remote sensing image registration problem and achieve high registration accuracy.The proposed algorithm includes four steps：1) The scale-invariant feature transform (SIFT) detector is employed to extract the feature points from the reference and sensed images.Two sets of extracted feature points are considered as salient image features and are used to represent putative feature correspondences.2) Global and local mixed features are identified based on the non-rigid point set registration algorithm for registering the two aforementioned sets of SIFT feature points

which consists of an alternating two-step process (i.e.

correspondence estimation and transformation update).Global and local geometric structure features are initially defined using the shape context and the local distance descriptors for measuring global and local structural differences between SIFT feature point sets

respectively.The two features are then combined to form global and local mixed features based on cost matrix.This step provides a flexible way to estimate point correspondences by minimizing global or local geometric structural differences using a linear assignment solution.An annealing scheme is designed to change the cost minimization from local to global gradually and the thin plate spline (TPS) transformation from rigid to non-rigid during registration to improve the correspondence estimation and enhance the interaction between the two steps.3) After point correspondences are calculated in Step 2

the backward approach and the TPS transformation model are applied to estimate the image transformation model between the reference and sensed images.4) After estimating the TPS transformation model

the transformed image is calculated using the TPS mapping functions that are constructed in the previous step.A bicubic interpolation algorithm is used in the sensed image on a regular grid.Each pixel from the sensed image can be directly transformed using the estimated mapping functions and the backward approach.Image interpolation occurs in the sensed image on the regular grid.Neither holes nor overlaps (due to the discretization and rounding) will occur in the output image (i.e.

the transformed image) by using the backward approach.The proposed algorithm can enhance the registration accuracy for multi-viewpoint remote sensing images with non-rigid distortion by improving the geometric structure feature descriptions and utilizing the complementary relationship between the global and local geometric structure features of SIFT feature point sets. The main process of the proposed algorithm is implemented in Matlab

and the Jonker-Volgenant algorithm in C++ is considered as a Matlab mex function.Two series of remote sensing image data are used to evaluate the performances of the proposed algorithm.These remote sensing image data are 1) the satellite image pairs of Google Earth with different viewpoints

which contain nine image pairs taken from London

Las Vegas

Paris

Mekong River

Tokyo

Hawaii

Beijing

Niulanjiang River

and New York；and 2) aerial image pairs of unmanned aerial vehicle (UAV) with different viewpoints

which contains five image pairs taken from the Yunnan Normal University

Chenggong Campus.Moreover

the proposed algorithm is compared against three state-of-the-art methods (i.e.

SIFT

speeded-up robust features

and coherent point drift.For quantitative comparisons

a reliable and fair criterion was employed to evaluate the performances of all methods.Nine pairs of corresponding points between the transformed and reference images are manually determined as ground truth

and all corresponding points were well-distributed and selected on the easily identified landmarks in the transformed and reference images.The registration error is defined as the root of mean square coordinate error among the determined corresponding points.Results show that the proposed multi-viewpoint remote sensing image registration algorithm has effectively improved the registration accuracies of SIFT feature point sets and provided accurate image registration results in multi-viewpoint remote sensing images with non-rigid distortion. In this study

a multi-viewpoint remote sensing image registration algorithm is realized by mixing global and local geometric structure features of SIFT feature point sets.The proposed algorithm includes 1) SIFT feature extraction

2) feature point set registration

3) image transformation model estimation

and 4) image transformation.The main contributions of this work are shown by the successful solution of the problems of remote sensing image registration by the proposed four-step algorithm in moderate ground relief variations and imaging viewpoint changes；the algorithm also shows the best alignments in most cases compared with the three state-of-the-art methods.The experimental results on the satellite images of Google Earth and UAV images show that the proposed algorithm can accurately register multi-viewpoint remote sensing images with non-rigid distortion and is applicable to address the problems in general multi-viewpoint remote sensing image registration.