浏览全部资源
扫码关注微信
收稿：2020-10-12，
修回：2021-1-12，
录用：2021-1-19，
纸质出版：2021-12-16
移动端阅览
目的
2
遥感图像配准是对多组图像进行匹配和叠加的过程。该技术在地物检测、航空图像分类和卫星图像融合等方面发挥着重要作用，主要有传统方法和基于深度学习的方法。其中，传统遥感图像配准算法在进行配准时会耗费大量人力，并且运行时间过长。而基于深度学习的遥感图像配准算法虽然减少了人工成本，提高了模型自适应学习的能力，但是算法的配准精度和运行时间仍有待提高。针对基于深度学习的配准算法存在的问题，本文提出了参数合成的空间变换网络对遥感图像进行双向一致性配准。
方法
2
通过增加空间变换网络的深度、合成网络内部的参数对空间变换模型进行改进，并将改进后的模型作为特征提取部分的骨干网络，有效地提高网络的鲁棒性。同时，将单向配准方法改为双向配准方法，进行双向的特征匹配和特征回归，保证配准方向的一致性。然后将回归得到的双向参数加权合成，提高模型的可靠性和准确性。
结果
2
将本文实验结果与两种经典的传统方法SIFT（scale-invariant feature transform）、SURF（speeded up robust features）对比，同时与近3年提出的CNNGeo（convolutional neural network architecture for geometric matching）、CNN-Registration（multi-temporal remote sensing image registration）和RMNet（robust matching network）3种最新的方法对比，配准结果表明本文方法不仅在定性的视觉效果上较为优异，而且在定量的评估指标上也有不错的效果。在Aerial Image Dataset数据集上，本文使用"关键点正确评估比例"与以上5种方法对比，精度分别提高了36.2%、75.9%、53.6%、29.9%和1.7%；配准时间分别降低了9.24 s、7.16 s、48.29 s、1.06 s和4.06 s。
结论
2
本文所提出的配准方法适用于时间差异变化（多时相）、视角差异（多视角）与拍摄传感器不同（多模态）的3种类型的遥感图像配准应用。在这3种类型的配准应用下，本文算法具有较高的配准精度和配准效率。
Objective
2
Remote sensing image registration is a process of matching and superimposing multiple sets of images. It plays an important role in many fields such as climate change
urban change and crustal movement. Currently
most remote sensing registration methods can be generally divided into two categories: traditional based methods and deep learning based methods. The traditional remote sensing registration algorithms can be labor-cost and weaken adaptive learning to cause time-consuming registration. Even though the remote sensing image registration algorithms based on deep learning reduce the labor cost and improve the ability of model adaptive learning
the accuracy and the running time still need to be improved. A parametric synthesized spatial transformation network has been proposed that can be probably used for bidirectional consistent registration of remote sensing images.
Methods
2
An end-to-end method is proposed for registration
which mainly includes feature extraction
feature matching and parameter regression. First
the feature extraction network has been designated based on the spatial transformation network model: the local network in the context of spatial transformation network has been more deepening via jumping connection. Four sets of full convolution modules are added
each of which is composed of four full convolution layers. Meanwhile
every two sets of four full convolution layers in each module are connected based on the same jumping connection structure. In order to ensure the integrity of data transmission
the beginning and the ending of each module are connected by jumping structure as well. Then two parameters have been synthesized which are regressed by local network. Following the process of grid generator and sampler
the input images are transformed to generate two saliency images with the same region based on affine transformation. Thus
fine-tuning residual structure has been used for feature extraction to obtain the targeted feature map. Next
a feature matching structure is designed to conduct bidirectional consistent matching. A matching branch is added to obtain the correlation from the source image to the target image and the correlation originated from the target image to the source image via Pearson correlation coefficient. The parameter regression network with two regression parameters have been leaked out based on the regression of matching relationship in two directions to maintain the consistency of registration. At last
the grid loss function has been iterated in consistency. The optimized bidirectional consistency parameters have been calculated via weighted and synthesized regression. The final registration is completed after sampling.
Result
2
The experimental results have been compared with two classical methods
which are scale-invariant feature transform (SIFT) and speeded up robust features(SURF).Simultaneously the latest methods proposed in recent three years have been compared as well
such as convolutional neural network architecture for geometric matching (CNNGeo)
CNN-Registration (multi-temporal remote sensing image registration) and robust matching network (RMNet). Registration results have illustrated that our research is qualified in qualitative visual effects and has good results in quantitative evaluation indexes. Based on the Aerial Image Dataset
"the percentage of correct key points" compared with the above five methods have been implemented
and the accuracy is increased by 36.2%
75.9%
53.6%
29.9% and 1.7%
respectively. Registration time is reduced by 9.24 s
7.16 s
48.29 s
1.06 s and 4.06 s. Since the gap between CNNGeo method
RMNet method and the method proposed
it cannot be clearly identified via the percentage of correct keypoints(PCK) evaluation index
the grid loss and the average grid loss for further comparison. Compared with the above two methods
the grid loss in this research has been increased by 3.48% and 2.66%
the average grid loss has been increased by 2.67% and 0.2% respectively. The gradient of the research method and RMNet method has decreased fastest in the grid loss line chart and average grid loss line chart. It has demonstrated that the accuracy of proposed method is higher via the histogram comparison between the method proposed and the RMNet method. The improved feature extraction network has been used to replace the feature extraction network in the CNNGeo method
and the PCK index is increased by 4.6% compared with the original benchmark network (CNNGeo). The improved matching relationship is replaced via the matching relationship in the CNNGeo method.The PCK index is improved by 3.9% compared with the original benchmark network. Bidirectional parameter of weighted synthesis has been further improved. The PCK index is increased by 14.1% compared with the original benchmark network. The experimental results have shown that the method proposed has its advantages in accuracy and efficient operation.
Conclusion
2
The registration method is applicable for three types of remote sensing image registration applications
such as temporal variation (multi-temporal)
visual diversity (multi-viewpoints) and different sensors (multi-modal). The proposed algorithm has illustrated more qualified registration accuracy and registration efficiency.
Bay H, Tuytelaars T and Van Gool L. 2006. SURF: speeded up robust features//Proceedings of the 9th European Conference on Computer Vision. Graz, Austria: Springer: 404-417[ DOI: 10.1007/11744023_32 http://dx.doi.org/10.1007/11744023_32 ]
Gong X Y, Liu Y Y and Yang Y. 2020. Robust stepwise correspondence refinement for low-altitude remote sensing image registration. IEEE Geoscience and Remote Sensing Letters, 18(10): 1736-1740[DOI:10.1109/LGRS.2020.3008446]
He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceeding of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 770-778[ DOI: 10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]
Jaderberg M, Simonyan K, Zisserman A and Kavukcuoglu K. 2015. Spatial transformer networks[EB/OL]. [2020-09-15] . https://arxiv.org/pdf/1506.02025.pdf https://arxiv.org/pdf/1506.02025.pdf
Ji K X, Guo C, ZouS F, Gao Y and Zhao H W. 2015. Image retrieval algorithm based on feature fusion and bidirectional image matching//Proceedings of the 4th National Conference on Electrical, Electronics and Computer Engineering. Xi'an, China: Atlantis Press: 1634-1639[ DOI: 10.2991/nceece-15.2016.295 http://dx.doi.org/10.2991/nceece-15.2016.295 ]
Kim B, Kim J, Lee J G, Kim D H, Park S H and Ye J C. 2019a. Unsupervised deformable image registration using cycle-consistent CNN//Proceedings of the 22nd International Conference on Medical Image Computing and Computer Assisted Intervention. Shenzhen, China: Springer: 166-174[ DOI: 10.1007/978-3-030-32226-7_19 http://dx.doi.org/10.1007/978-3-030-32226-7_19 ]
Kim D G, Nam W J and Lee S W. 2019b. A robust matching network for gradually estimating geometric transformation on remote sensing imagery//Processing of 2019 IEEE International Conference on Systems, Man, and Cybernetics. Bari, Italy: IEEE: 3889-3894[ DOI: 10.1109/SMC.2019.8913881 http://dx.doi.org/10.1109/SMC.2019.8913881 ]
Li H Y, Li C G, An J B and Ren J L. 2019. Attention mechanism improves CNN remote sensing image object detection. Journal of Image and Graphics, 24(8): 1400-1408
李红艳, 李春庚, 安居白, 任俊丽. 2019. 注意力机制改进卷积神经网络的遥感图像目标检测. 中国图象图形学报, 24(8): 1400-1408[DOI:10.11834/jig.180649]
Liang Y, Cheng H, Sun W B and Wang Z Q. 2010. Research on methods of image registration. Image Technology, 22(4): 15-17, 46
梁勇, 程红, 孙文邦, 王志强. 2010. 图像配准方法研究. 影像技术, 22(4): 15-17, 46[DOI:10.3969/j.issn.1001-0270.2010.04.004]
Lin C H and Lucey S. 2017. Inverse compositional spatial transformer networks//Processing of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 2252-2260[ DOI: 10.1109/CVPR.2017.242 http://dx.doi.org/10.1109/CVPR.2017.242 ]
Liu H M, Wang H and Duan H F. 2009. A bidirectional matching SIFT algorithm. Ordnance Industry Automation, 28(6): 89-91
刘焕敏, 王华, 段慧芬. 2009. 一种改进的SIFT双向匹配算法. 兵工自动化, 28(6): 89-91[DOI:10.3969/j.issn.1006-1576.2009.06.033]
Liu R H. 2015. Research on SAR Images Matching Based on an Improved SIFT-Like Algorithm. Xi'an: Xidian University: 1-95
刘瑞红. 2015. 基于改进SIFT-Like算法的SAR图像特征匹配. 西安: 西安电子科技大学: 1-95
Lowe D G. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2): 91-110[DOI:10.1023/B:VISI.0000029664.99615.94]
Ma J Y, Jiang X J, Fan A X, Jiang J J and Yan J C. 2021. Image matching from handcrafted to deep features: a survey. International Journal of Computer Vision, 129(1): 23-79[DOI:10.1007/s11263-020-01359-2]
Ning J J and Kong L D. 2012. A parallax image algorithm based on two-way mutual matching. Computer Development and Applications, 25(1): 17-18, 72
宁静静, 孔令德. 2012. 一种基于双向互匹配的视差图算法. 电脑开发与应用, 25(1): 17-18, 72[DOI:10.3969/j.issn.1003-5850.2012.01.006]
Park J H, Nam W J and Lee S W. 2020. A two-stream symmetric network with bidirectional ensemble for aerial image matching. Remote Sensing, 12(3): #465[DOI:10.3390/rs12030465]
Quan D, Wang S, Ning M D, Xiong T and Jiao L C. 2016. Using deep neural networks for synthetic aperture radar image registration//2016 IEEE International Geoscience and Remote Sensing Symposium. Beijing, China: IEEE: 2799-2802[ DOI: 10.1109/IGARSS.2016.7729723 http://dx.doi.org/10.1109/IGARSS.2016.7729723 ]
Rocco I, Arandjelovic R and Sivic J. 2017. Convolutional neural network architecture for geometric matching//Proceesing of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 39-48[ DOI: 10.1109/CVPR.2017.12 http://dx.doi.org/10.1109/CVPR.2017.12 ]
Rublee E, Rabaud V, Konolige K and Bradski G. 2011. ORB: an efficient alternative to SIFT or SURF//Proceedings of International Conference on Computer Vision. Barcelona, Spain: IEEE: 2564-2571[ DOI: 10.1109/ICCV.2011.6126544 http://dx.doi.org/10.1109/ICCV.2011.6126544 ]
Seo P H, Lee J, Jung D, Han B and Cho M. 2018. Attentive semantic alignment with offset-aware correlation kernels//Proceesing of 2018 European Conference on Computer Vision. Munich, Germany: Springer: 367-383[ DOI: 10.1007/978-3-030-01225-0_22 http://dx.doi.org/10.1007/978-3-030-01225-0_22 ]
VeličkovićP, Cucurull G, Casanova A, Romero A, LiòP and Bengio Y. 2017. Graph attention networks[EB/OL]. [2020-09-12] . https://arxiv.org/pdf/1710.10903.pdf https://arxiv.org/pdf/1710.10903.pdf
Wang B S, Zhang J X, Lu L J, Huang G M and Zhao Z. 2015. A uniform SIFT-like algorithm for SAR image registration. IEEE Geoscience and Remote SensingLetters, 12(7): 1426-1430[DOI:10.1109/LGRS.2015.2406336]
Wang L F, Zhang C C, Qin P L, Lin S Z, Gao Y and Dou J L. 2020. Image registration method with residual dense relativistic average CGAN. Journal of Image and Graphics, 25(4): 745-758
王丽芳, 张程程, 秦品乐, 蔺素珍, 高媛, 窦杰亮. 2020. 残差密集相对平均CGAN的脑部图像配准. 中国图象图形学报, 25(4): 745-758[DOI:10.11834/jig.190116]
Wang S, Jiao L C, Fang S, Quan D, Wang R J, Liang X F, Hou B and Liu F H. 2018. Heterogeneous image matching method based on deep learning. China, 201810277816.7
王爽, 焦李成, 方帅, 权豆, 王若静, 梁雪峰, 侯彪, 刘飞航. 2018. 基于深度学习的异源图像匹配方法. 中国, 201810277816.7
Wang S, Quan D, Liang X F, Ning M D, Guo Y H and Jiao L C. 2018. A deep learning framework for remote sensing image registration. ISPRS Journal of Photogrammetry and Remote Sensing, 145: 148-164[DOI:10.1016/j.isprsjprs.2017.12.012]
Xu D L and Hu Z Z. 2019. Remote sensing image registration based on deep learning feature extraction. Spacecraft Recovery and Remote Sensing, 40(6): 107-118
许东丽, 胡忠正. 2019. 基于深度学习特征提取的遥感影像配准. 航天返回与遥感, 40(6): 107-118[DOI:10.3969/j.issn.1009-8518.2019.06.013]
Yang Z Q, Dan T T and Yang Y. 2018. Multi-temporal remote sensing image registration using deep convolutional features. IEEE Access, 6: 38544-38555[DOI:10.1109/ACCESS.2018.2853100]
Ye F M, Luo W, Su Y F, Zhao X Q, Xiao H and Min W D. 2019. Application of convolutional neural network feature to remote sensing image registration. Remote Sensing for Land and Resources, 31(2): 32-37
叶发茂, 罗威, 苏燕飞, 赵旭青, 肖慧, 闵卫东. 2019. 卷积神经网络特征在遥感图像配准中的应用. 国土资源遥感, 31(2): 32-37[DOI:10.6046/gtzyyg.2019.02.05]
Yu G S and Morel J M. 2011. ASIFT: an algorithm for fully affine invariant comparison. Image Processing on Line, 1: 11-38[DOI:10.5201/ipol.2011.my-asift]
Zhang F J, Wang Z Q and Wu D. 2016. Improved algorithm of image regestration based on SURF. Journal of Changchun University of Science and Technology (Natural Science Edition), 39(1): 112-115
张凤晶, 王志强, 吴迪. 2016. 基于SURF的图像配准改进算法. 长春理工大学学报(自然科学版), 39(1): 112-115[DOI:10.3969/j.issn.1672-9870.2016.01.025]
Zheng Y and Li G Y. 2011. Bilateral medical image registration based on regional and local information. Journal of Image and Graphics, 16(1): 90-96
郑莹, 李光耀. 2011. 区域和局部信息结合的双向医学图像配准. 中国图象图形学报, 16(1): 90-96[DOI:10.11834/jig.20110112]
相关作者
相关机构
京公网安备11010802024621
