浏览全部资源
扫码关注微信
纸质出版日期： 2020-12-16 ，
录用日期： 2020-02-26
移动端阅览
葛芸, 马琳, 储珺. 结合判别相关分析与特征融合的遥感图像检索[J]. 中国图象图形学报, 2020,25(12):2665-2676.
Yun Ge, Lin Ma, Jun Chu. Remote sensing image retrieval combining discriminant correlation analysis and feature fusion[J]. Journal of Image and Graphics, 2020,25(12):2665-2676.
目的
2
高分辨率遥感图像检索中，单一特征难以准确描述遥感图像的复杂信息。为了充分利用不同卷积神经网络（convolutional neural networks，CNN）的学习参数来提高遥感图像的特征表达，提出一种基于判别相关分析的方法融合不同CNN的高层特征。
方法
2
将高层特征作为特殊的卷积层特征处理，为了更好地保留图像的原始空间信息，在图像的原始输入尺寸下提取不同高层特征，再对高层特征进行最大池化来获得显著特征；计算高层特征的类间散布矩阵，结合判别相关分析来增强同类特征的联系，并突出不同类特征之间的差异，从而提高特征的判别力；选择串联与相加两种方法来对不同特征进行融合，用所得融合特征来检索高分辨率遥感图像。
结果
2
在UC-Merced、RSSCN7和WHU-RS19数据集上的实验表明，与单一高层特征相比，绝大多数融合特征的检索准确率和检索时间都得到有效改进。其中，在3个数据集上的平均精确率均值（mean average precision，mAP）分别提高了10.4% 14.1%、5.7% 9.9%和5.9% 17.6%。以检索能力接近的特征进行融合时，性能提升更明显。在UC-Merced数据集上，融合特征的平均归一化修改检索等级（average normalized modified retrieval rank，ANMRR）和mAP达到13.21%和84.06%，与几种较新的遥感图像检索方法相比有一定优势。
结论
2
本文提出的基于判别相关分析的特征融合方法有效结合了不同CNN高层特征的显著信息，在降低特征冗余性的同时，提升了特征的表达能力，从而提高了遥感图像的检索性能。
Objective
2
With the rapid development of remote sensing technology
numerous high-resolution remote sensing images have become available. As a result
the effective retrieval of remote sensing images has become a challenging research topic. Feature extraction is key to determining the retrieval performance of high-resolution remote sensing image retrieval tasks. Traditional feature extraction methods are mainly based on handcrafted features
whereas such shallow features are easily affected by artificial intervention. Convolutional neural networks (CNNs) can learn feature representations automatically
and thus are suitable to deal with high-resolution remote sensing images with complex content. However
the parameters of CNNs are difficult to train fully due to the small scale of currently available public remote sensing datasets. In this case
the transfer learning of CNNs has attracted much attention. CNNs pretrained on large-scale datasets have good generalization ability
and parameters can be transferred to small-scale data effectively. Therefore
extracting CNN features on the basis of transfer learning has become an effective method in the field of remote sensing image retrieval. Given the abundant and complex visual content of high-resolution remote sensing images
it is difficult to accurately express the content of remote sensing images using a single feature. Thus
feature fusion is a useful method to improve the feature representation of remote sensing images. To maximize the learning parameters of different CNNs to represent the content of remote sensing images
a method based on discriminant correlation analysis (DCA) is proposed to fuse the high-level features of different CNNs.
Method
2
First
CNN parameters from VGGM(visual geometry group medium)
VGG(visual geometry group)16
GoogLeNet
and ResNet50 are transferred for high-resolution remote sensing images
and the high-level features are adopted as special convolutional features. To preserve the original spatial information of the image
the high-level features are extracted under the original input image size
and the output form of three-dimensional tensor is retained. Then
max pooling is adopted on the high-level features to extract salient features. Second
DCA is adopted to enhance the feature representation. The DCA is the first to incorporate the class structure into the feature level fusion and has low computational complexity. To maximize the correlation of corresponding features across the two feature sets and in the same time decorrelates features that belong to different classes within each feature set
the between-class scatter matrices of the two sets of high-level features are calculated
and matrix diagonalization and singular value decomposition are adopted to transform the features. The transformed matrix contains the important eigenvectors of the between-class scatter matrix
and the dimension of the transformed matrix is reduced accordingly. Thus
the transformed feature vectors have strong discriminative power and low dimension. Lastly
two methods of concatenation and summation are selected to perform the fusion of transformed feature vectors
and the fused features are normalized via Gaussian normalization. The similarities between the query and dataset features are calculated using the Euclidean distance method
and the retrieval results are returned in accordance with the sort of similarities.
Result
2
Experiment results on the UC-Merced
RSSCN7
and WHU-RS19 datasets show that the retrieval accuracy and retrieval time of most fusion features are effectively improved in comparison with a single high-level feature; the mean average precision (mAP) of the fusion feature is improved by 10.4%14.1%
5.7%9.9%
and 5.9%17.6%
respectively. The retrieval results of the fused features using the concatenation method are better than that using the summation method. Multifeature fusion experiments show that the best result on the UC-Merced dataset is obtained from the fusion of four features
whereas the best results on the RSSCN7 and WHU-RS19 datasets are obtained from the fusion of three features. This finding indicates that a larger number of fused features does not translate into better performance; selecting the appropriate features is crucial for feature fusion. Especially
when the different features have good representation and similar retrieval capabilities
the fusion of these features can achieve good retrieval performance. Compared with other state-of-the-art approaches
the average normalized modified retrieval rank(ANMRR) and mAP of the proposed fused feature on the UC-Merced dataset reach 0.132 1 and 84.06%
respectively. Experimental results demonstrate that our method outperforms state-of-the-art approaches.
Conclusion
2
The feature fusion method based on discriminant correlation analysis combines the salient information of different high-level features. This method reduces feature redundancy while improving feature discrimination. Features with equivalent retrieval capabilities can be fused by the proposed method well
thus effectively improving the retrieval performance of high-resolution remote sensing images.
遥感图像检索卷积神经网络高层特征融合判别相关分析最大池化
remote sensing image retrievalconvolutional neural network (CNN)high-level feature fusiondiscriminant correlation analysis(DCA)max pooling
Chaib S, Liu H, Gu Y F and Yao H X. 2017. Deep feature fusion for VHR remote sensing scene classification. IEEE Transactions on Geoscience and Remote Sensing, 55(8):4775-4784[DOI:10.1109/TGRS.2017.2700322]
Chatfield K, Simonyan K, Vedaldi A and Zisserman A. 2014. Return of the devil in the details: delving deep into convolutional nets[EB/OL].[2019-12-23].https://arxiv.org/pdf/1405.3531.pdfhttps://arxiv.org/pdf/1405.3531.pdf
Du P J, Li E Z, Xia J S, Samat A and Bai X Y. 2019. Feature and model level fusion of pretrained CNN for remote sensing scene classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(8):2600-2611[DOI:10.1109/JSTARS.2018.2878037]
Farah J, Zhou J, Awrangjeb M and Gao Y S. 2018. Fusion of hyperspectral and LiDAR data using discriminant correlation analysis for land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(10):3905-3917[DOI:10.1109/JSTARS.2018.2868142]
Ge Y, Jiang S L, Xu Q Y, Jiang C L and Ye F M. 2018. Exploiting representations from pre-trained convolutional neural networks for high-resolution remote sensing image retrieval. Multimedia Tools and Applications, 77(13):17489-17515[DOI:10.1007/s11042-017-5314-5]
Haghighat M, Abdel-Mottaleb M and Alhalabi W. 2016. Discriminant correlation analysis:real-time feature level fusion for multimodal biometric recognition. IEEE Transactions on Information Forensics and Security, 11(9):1984-1996[DOI:10.1109/TIFS.2016.2569061]
He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 770-778[DOI: 10.1109/CVPR.2016.90http://dx.doi.org/10.1109/CVPR.2016.90]
Hu F, Tong X Y, Xia G S and Zhang L P. 2017. Delving into deep representations for remote sensing image retrieval//Proceedings of the 13th International Conference on Signal Processing. Chengdu, China: IEEE: 198-203[DOI: 10.1109/ICSP.2016.7877823http://dx.doi.org/10.1109/ICSP.2016.7877823]
Hu F, Xia G S, Hu J W and Zhang L P. 2015. Transferring deep convolutional neural networks for the scene classification of high-resolution remote sensing imagery. Remote Sensing, 7(11):14680-14707[DOI:10.3390/rs71114680]
Husain S S and Bober M. 2019. REMAP:multi-layer entropy-guided pooling of dense CNN features for image retrieval. IEEE Transactions on Image Processing, 28(10):5201-5213[DOI:10.1109/TIP.2019.2917234]
Imbriaco R, Sebastian C, Bondarev E and de With P H N. 2019. Aggregated deep local features for remote sensing image retrieval. Remote Sensing, 11(5):#493[DOI:10.3390/rs11050493]
Klaric M N, Scott G J and Shyu C R. 2012. Multi-index multi-object content-based retrieval. IEEE Transactions on Geoscience and Remote Sensing, 50(10):4036-4049[DOI:10.1109/TGRS.2012.2187353]
Li G, Li L L, Zhu H, Liu X and Jiao L C. 2019a. Adaptive multiscale deep fusion residual network for remote sensing image classification. IEEE Transactions on Geoscience and Remote Sensing, 57(11):8506-8521[DOI:10.1109/TGRS.2019.2921342]
Li S W, Purushotham S, Chen C, Ren Y Z and Kuo C C J. 2017. Measuring and predicting tag importance for image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12):2423-2436[DOI:10.1109/TPAMI.2017.2651818]
Li Y Y, Wang Q, Liang X X and Jiao L C. 2019b. A novel deep feature fusion network for remote sensing scene classification//Proceedings of 2019 IEEE International Geoscience and Remote Sensing Symposium. Yokohama, Japan: IEEE: 5484-5487[DOI: 10.1109/IGARSS.2019.8898900http://dx.doi.org/10.1109/IGARSS.2019.8898900]
Liu R J, Zhang H X and Kong W J. 2015. Image retrieval method based on feature-associated fusion. Journal of University of Jinan (Science and Technology), 29(5):327-332
刘润杰, 张化祥, 孔文杰. 2015.基于特征关联融合的图像检索方法.济南大学学报(自然科学版), 29(5):327-332[DOI:10.13349/j.cnki.jdxbn.2015.05.002]
Lu L Z, Liu R Y and Liu N. 2004. Remote sensing image retrieval using color and texture fused features. Journal of Image and Graphics, 9(3):328-333
陆丽珍, 刘仁义, 刘南. 2004.一种融合颜色和纹理特征的遥感图像检索方法.中国图象图形学报, 9(3):328-333[DOI:10.11834/jig.20040361]
Peng Y F, Song X N, Wu H and Zi L L. 2019. Remote sensing image retrieval combined with deep learning and relevance feedback. Journal of Image and Graphics, 24(3):420-434
彭晏飞, 宋晓男, 武宏, 訾玲玲. 2019.结合深度学习与相关反馈的遥感图像检索.中国图象图形学报, 24(3):420-434[DOI:10.11834/jig.180384]
Radenović F, Tolias G and Chum O. 2019. Fine-tuning CNN image retrieval with no human annotation. IEEE Transactions on Pattern Analysis and MachineIntelligence, 41(7):1655-1668[DOI:10.1109/TPAMI.2018.2846566]
Shelhamer E, Long J and Darrell T. 2017. Fully convolutional networks for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(4):640-651[DOI:10.1109/TPAMI.2016.2572683]
Simonyan K and Zisserman A. 2015. Very deep convolutional networks for large-scale image recognition[EB/OL].[2019-12-23].https://arxiv.org/pdf/1409.1556v6.pdfhttps://arxiv.org/pdf/1409.1556v6.pdf
Sun Q S, Zeng S G, Liu Y, Heng P A and Xia D S. 2005. A new method of feature fusion and its application in image recognition. Pattern Recognition, 38(12):2437-2448[DOI:10.1016/j.patcog.2004.12.013]
Szegedy C, Liu W, Jia Y Q, Sermanet R, Reed S, Anguelov D, Erhan D, Vanhoucke V and Rabinovich A. 2015. Going deeper with convolutions[EB/OL].[2019-12-23].https://arxiv.org/pdf/1409.4842.pdfhttps://arxiv.org/pdf/1409.4842.pdf
Tong X Y, Xia G S, Hu F, Zhong Y F, Datcu M and Zhang L P. 2017. Exploiting deep features for remote sensing image retrieval: a systematic investigation[EB/OL].[2019-12-23].https://arxiv.org/pdf/1707.07321.pdfhttps://arxiv.org/pdf/1707.07321.pdf
Vedaldi A and Lenc K. 2015. MatConvNet: convolutional neural networks for MATLAB//Proceedings of the 23rd ACM International Conference on Multimedia. Brisbane, Australia: ACM: 689-692[DOI: 10.1145/2733373.2807412http://dx.doi.org/10.1145/2733373.2807412]
Wang G L, Fan B, Xiang S M and Pan C H. 2017. Aggregating rich hierarchical features for scene classification in remote sensing imagery. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 10(9):4104-4115[DOI:10.1109/JSTARS.2017.2705419]
Wang Y B, Zhang L Q, Tong X H, Zhang L, Zhang Z X, Liu H, Xing X Y and Mathiopoulos P T. 2016. A three-layered graph-based learning approach for remote sensing image retrieval. IEEE Transactions on Geoscience and Remote Sensing, 54(10):6020-6034[DOI:10.1109/TGRS.2016.2579648]
Yang K, Li C M, Zhou W X, Cheng Q M and Ren Y C. 2019. Remote sensing image retrieval based on multi-layer feature integration of convolution neural networks. Science of Surveying and Mapping, 44(7):9-15, 34
杨珂, 李从敏, 周维勋, 程起敏, 任应超. 2019.卷积神经网络多层特征联合的遥感图像检索.测绘科学, 44(7):9-15, 34[DOI:10.16251/j.cnki.1009-2307.2019.07.002]
Yang Y and Newsam S. 2013. Geographic image retrieval using local invariant features. IEEE Transactions on Geoscience and Remote Sensing, 51(2):818-832[DOI:10.1109/tgrs.2012.2205158]
Ye F M, Xiao H, Zhao X Q, Dong M, Luo W and Min W D. 2018. Remote sensing image retrieval using convolutional neural network features and weighted distance. IEEE Geoscience and Remote Sensing Letters, 15(10):1535-1539[DOI:10.1109/LGRS.2018.2847303]
Zhou W X, Deng X Q and Shao Z F. 2018. Region convolutional features for multi-label remote sensing image retrieval[EB/OL].[2019-12-23].https://arxiv.org/pdf/1807.08634.pdfhttps://arxiv.org/pdf/1807.08634.pdf
Zhou W X, Newsam S, Li C M and Shao Z F. 2017. Learning low dimensional convolutional neural networks for high-resolution remote sensing image retrieval. Remote Sensing, 9(5):#489[DOI:10.3390/rs9050489]
Zou Q, Ni L H, Zhang T and Wang Q. 2015. Deep learning based feature selection for remote sensing scene classification. IEEE Geoscience and Remote Sensing Letters, 12(11):2321-2325[DOI:10.1109/LGRS.2015.2475299]
相关作者
相关机构
京公网安备11010802024621