浏览全部资源
扫码关注微信
纸质出版日期： 2023-09-16 ，
移动端阅览
张磊， 陈文， 王岳环. 2023. 用于遥感舰船细粒度检测与识别的关键子区域融合网络. 中国图象图形学报， 28(09):2940-2955
Zhang Lei， Chen Wen， Wang Yuehuan. 2023. Key sub-region feature fusion network for fine-grained ship detection and recognition in remote sensing images. Journal of Image and Graphics， 28(09):2940-2955
目的
2
遥感图像中的舰船目标细粒度检测与识别在港口海域监视以及情报搜集等应用中有很高的实际应用价值，但遥感图像中不同种类的舰船目标整体颜色、形状与纹理特征相近，分辨力不足，导致舰船细粒度识别困难。针对该问题，提出了一种端到端的基于关键子区域特征的舰船细粒度检测与识别方法。
方法
2
为了获得更适于目标细粒度识别的特征，提出多层次特征融合识别网络，按照整体、局部子区域两个层次从检测网络得到的候选目标区域中提取特征。然后结合候选目标中所有子区域的信息计算每个子区域的判别性显著度，对含有判别性组件的关键子区域进行挖掘。最后基于判别性显著度将子区域特征与整体特征进行自适应融合，形成表征能力更强的特征，对舰船目标进行细粒度识别。整个检测与识别网络采用端到端一体化设计，所有候选目标特征提取过程只需要经过一次骨干网络的计算，提高了计算效率。
结果
2
在公开的带有细粒度类别标签的HRSC2016（high resolution ship collection）数据集L3任务上，本文方法平均准确率为77.3%，相较于不采用多层次特征融合识别网络提升了6.3%；在自建的包含45类舰船目标的FGSAID（fine-grained ships in aerial images dataset）数据集上，本文方法平均准确率为71.5%。
结论
2
本文方法有效挖掘并融合了含有判别性组件的子区域的特征，解决了目标整体特征分辨力不足导致的细粒度目标识别困难问题，相较于现有的遥感图像舰船目标检测与识别算法准确性有明显提升。
Objective
2
The ocean has great economic and military value. The development of human society increases the impact of ocean activities on the development of a country. The sea is an important carrier of marine activities. Thus， the recognition and monitoring of ship targets in key sea areas through remote sensing images are crucial to the national defense and development of the economy. Fine-grained ship detection and recognition in high-resolution remote sensing images refer to the identification of specific types of ships based on ship detection. A precise and detailed classification is valuable in practical application fields， such as sea surveillance and intelligence gathering. Instead of coarse-grained classification categories， such as warcraft and merchant ships， specific ship types， such as Arleigh Burke-class destroyer， Nimitz-class aircraft carrier， container， and car carrier， are necessary. However， the overall color， shape， and texture of different types of ship targets are similar. The structures of ships belong to different types， but their uses are similar. Moreover， the coating color of military ships is monotonous. These characteristics complicate the classification of these targets. The existing ship detectors are designed to focus on locating targets. The design of the classification branch of these detectors is relatively simple. They only use the features of whole targets for classification， significantly decreasing the performance in the fine-grained labeled datasets. The existing ship classification methods， which mainly classify targets on the pre-cropped image patches， are separated from the detection process. This approach is unsatisfactory for practical applications for two reasons： 1） the whole backbone of these methods based on neural networks must be executed on every proposal to extract features. The remote sensing images of the harbor usually include several ships； thus， the computation cost increases sharply. 2） The detection and classification networks are optimized separately， and the parameters of both networks are optimized to the best. The whole process cannot obtain the optimal solution because the locations of proposals obtained by detection methods vary with the pre-cropped image patches. utilize prior knowledge of ships and propose the key sub-region feature fusion network （KSFFN）， which fuses features of sub-regions that are discriminative to the whole feature and combines detection and fine-grained recognition into one framework.
Method
2
KSFFN uses ResNet-50 as the backbone network to extract features and construct a proposal locating network by combining Faster R-CNN with region of interest（ROI） Transformer for obtaining proposal locations. Then， all of the proposals are ranked according to the probability of targets. The proposals with low probability are filtered. Then， the multi-level feature fusion recognition network （MLFFRN） is proposed to extract features from the proposals generated by the proposal locating network and to classify the proposals. First， the proposals are separated into several subregions along the axis and the overall features and sub-region features are extracted from different levels of the feature pyramid. Then， the self-supervision mechanism in the navigator-teacher-scrutinizer network （NTS-Net） finds the key subregion that may contain important parts contributing to fine-grained recognition. Due to the limitation of image quality and characteristics of the target， not all targets have a very discriminating subregion. Moreover， the self-supervision mechanism in NTS-Net cannot reflect this subregion. Therefore， the information from all subregions in the proposal is utilized to calculate the discriminant significance of the subregion， which reflects the influence of the subregion on target recognition. Based on the discriminant significance， the weight of the sub-region is calculated， and the key sub-region features are fused with the overall features according to the weight. The combined feature is used to obtain the final classification result， thereby improving the accuracy of fine-grained recognition of ship targets.
Result
2
Public high resolution ship collection 2016 （HRSC2016） dataset L3 task and self-built fine-grained ships in aerial images dataset （FGSAID） are used to evaluate the model. HRSC2016 dataset contains 1 061 images with 2 886 ships divided into 19 types. FGSAID dataset contains 1 690 images with 5 410 ships divided into 45 types. The average precision （AP） is used as an evaluation metric， and the intersection over union is set as 0.5 to determine whether the prediction box matches the ground truth. On the HRSC2016 dataset L3 task， the proposed method achieves an AP of 77.3%， and MLFFRN can improve the AP by 6.3%. On the FGSAID dataset， our method achieves an AP of 71.5%. A series of ablation experiments is conducted on the HRSC2016 dataset L3 task to show the effectiveness of different parts of the proposed method. In addition， the proposed method is compared with the state-of-the-art deep-learning-based ship detection framework on two datasets. The experiment results show that our model outperforms all other methods on both datasets. Compared with single-shot alignment network （S
2
ANet） network， the proposed method increases the AP by 7.8% and 8.9% on HRSC2016 and FGSAID， respectively. In particular， the AP of the proposed method increases by 16.7%， 11.1%， and 1.1% for the aircraft carrier/amphibious assault ships， other warships， and merchant ships， respectively， in the FGSAID dataset.
Conclusion
2
In this study， the end-to-end fine-grained ship detection and recognition network KSFFN is proposed. It extracts the overall features and sub-region features of the proposals and fuses them according to the discriminant significance. The proposed method combines detection and fine-grained recognition into one framework， thereby improving the processing speed greatly while performing excellently. Thus， KSFFN has great application value. The proposed method has a more powerful classification framework and can achieve more accurate results than the existing detection method. The experiment results show that our method outperforms several state-of-the-art deep-learning-based ship detection frameworks， thereby proving the effectiveness of KSFFN.
遥感图像舰船检测细粒度识别关键子区域判别性显著度
remote sensing imagesship detectionfine-grained recognitionkey sub-regiondiscriminant significance
Burges C， Shaked T， Renshaw E， Lazier A， Deeds M， Hamilton N and Hullender G. 2005. Learning to rank using gradient descent//Proceedings of the 22nd International Conference on Machine Learning. Bonn， Germany： ACM： 89-96 ［DOI： 10.1145/1102351.1102363http://dx.doi.org/10.1145/1102351.1102363］
Cui Y， Jia M L， Lin T Y， Song Y and Belongie S. 2019. Class-balanced loss based on effective number of samples//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Long Beach， USA： IEEE： 9260-9269 ［DOI： 10.1109/CVPR.2019.00949http://dx.doi.org/10.1109/CVPR.2019.00949］
Ding J， Xue N， Long Y， Xia G S and Lu Q K. 2019. Learning RoI Transformer for oriented object detection in aerial images//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Long Beach， USA： IEEE： 2844-2853 ［DOI： 10.1109/CVPR.2019.00296http://dx.doi.org/10.1109/CVPR.2019.00296］
Feng Y C， Diao W H， Sun X， Yan M L and Gao X. 2019. Towards automated ship detection and category recognition from high-resolution aerial images. Remote Sensing， 11（16）： #1901 ［DOI： 10.3390/rs11161901http://dx.doi.org/10.3390/rs11161901］
Gallego A J， Pertusa A and Gil P. 2018. Automatic ship classification from optical aerial images with convolutional neural networks. Remote Sensing， 10（4）： #511 ［DOI： 10.3390/rs10040511http://dx.doi.org/10.3390/rs10040511］
Guo H Y， Yang X， Wang N N， Song B and Gao X B. 2020. A rotational libra R-CNN method for ship detection. IEEE Transactions on Geoscience and Remote Sensing， 58（8）： 5772-5781 ［DOI： 10.1109/TGRS.2020.2969979http://dx.doi.org/10.1109/TGRS.2020.2969979］
Han J M， Ding J， Li J and Xia G S. 2021. Align deep features for oriented object detection ［EB/OL］. ［2022-06-01］. https://arxiv.org/pdf/2008.09397.pdfhttps://arxiv.org/pdf/2008.09397.pdf
He K M， Gkioxari G， Dollr P and Girshick R. 2017. Mask R-CNN//Proceedings of 2017 IEEE International Conference on Computer Vision （ICCV）. Venice， Italy： IEEE： 2980-2988 ［DOI： 10.1109/ICCV.2017.322http://dx.doi.org/10.1109/ICCV.2017.322］
He K M， Zhang X Y， Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Las Vegas， USA： IEEE： 770-778 ［DOI： 10.1109/CVPR.2016.90http://dx.doi.org/10.1109/CVPR.2016.90］
Huang J， Sivakumar V， Mnatsakanyan M and Pang G. 2018. Improving rotated text detection with rotation region proposal networks ［EB/OL］. ［2022-06-01］. https://arxiv.org/pdf/1811.07031.pdfhttps://arxiv.org/pdf/1811.07031.pdf
Li B， Xie X Y， Wei X X and Tang W T. 2021. Ship detection and classification from optical remote sensing images： a survey. Chinese Journal of Aeronautics， 34（3）： 145-163 ［DOI： 10.1016/j.cja.2020.09.022http://dx.doi.org/10.1016/j.cja.2020.09.022］
Li W J， Zhao H P and Shang S N. 2017. Onboard ship saliency detection algorithm based on multi-scale fractal dimension. Journal of Image and Graphics， 22（10）： 1447-1454
李文娟， 赵和平， 尚叔楠. 2017. 多尺度分形维的星载舰船显著性检测. 中国图象图形学报， 22（10）： 1447-1454 ［DOI： 10.11834/jig.160529http://dx.doi.org/10.11834/jig.160529］
Lin T Y， Dollr P， Girshick R， He K M， Hariharan B and Belongie S. 2017. Feature pyramid networks for object detection//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Honolulu， USA： IEEE： 936-944 ［DOI： 10.1109/CVPR.2017.106http://dx.doi.org/10.1109/CVPR.2017.106］
Liu G， Zhang Y S， Zheng X W， Sun X， Fu K and Wang H Q. 2014. A new method on inshore ship detection in high-resolution satellite images using shape and context information. IEEE Geoscience and Remote Sensing Letters， 11（3）： 617-621 ［DOI： 10.1109/LGRS.2013.2272492http://dx.doi.org/10.1109/LGRS.2013.2272492］
Liu W C， Ma L and Chen H. 2018. Arbitrary-oriented ship detection framework in optical remote-sensing images. IEEE Geoscience and Remote Sensing Letters， 15（6）： 937-941 ［DOI： 10.1109/LGRS.2018.2813094http://dx.doi.org/10.1109/LGRS.2018.2813094］
Liu Z K， Hu J G， Weng L B and Yang Y P. 2017a. Rotated region based CNN for ship detection//Proceedings of 2017 IEEE International Conference on Image Processing （ICIP）. Beijing， China： IEEE： 900-904 ［DOI： 10.1109/ICIP.2017.8296411http://dx.doi.org/10.1109/ICIP.2017.8296411］
Liu Z K， Yuan L， Weng L B and Yang Y P. 2017b. A high resolution optical satellite image dataset for ship recognition and some new baselines//Proceedings of the 6th International Conference on Pattern Recognition Applications and Methods （ICPRAM）. Porto， Portugal： SciTePress： 324-331 ［DOI： 10.5220/0006120603240331http://dx.doi.org/10.5220/0006120603240331］
Qi S X， Ma J， Lin J， Li Y S and Tian J W. 2015. Unsupervised ship detection based on saliency and S-HOG descriptor from optical satellite images. IEEE Geoscience and Remote Sensing Letters， 12（7）： 1451-1455 ［DOI： 10.1109/LGRS.2015.2408 355http://dx.doi.org/10.1109/LGRS.2015.2408355］
Redmon J， Divvala S， Girshick R and Farhadi A. 2016. You only look once： unified， real-time object detection//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Las Vegas， USA： IEEE： 779-788 ［DOI： 10.1109/CVPR.2016.91http://dx.doi.org/10.1109/CVPR.2016.91］
Ren S Q， He K M， Girshick R and Sun J. 2017. Faster R-CNN： towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence， 39（6）： 1137-1149 ［DOI： 10.1109/TPAMI.2016.2577031http://dx.doi.org/10.1109/TPAMI.2016.2577031］
Shi Z W， Yu X R， Jiang Z G and Li B. 2014. Ship detection in high-resolution optical imagery based on anomaly detector and local shape feature. IEEE Transactions on Geoscience and Remote Sensing， 52（8）： 4511-4523 ［DOI： 10.1109/TGRS.2013.2282355http://dx.doi.org/10.1109/TGRS.2013.2282355］
Tian Z， Shen C H， Chen H and He T. 2019. FCOS： fully convolutional one-stage object detection//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision （ICCV）. Seoul， Korea （South）： IEEE： 9626-9635 ［DOI： 10.1109/ICCV.2019.00972http://dx.doi.org/10.1109/ICCV.2019.00972］
Xia Y， Wan S H， Jin P Q and Yue L H. 2014. A novel sea-land segmentation algorithm based on local binary patterns for ship detection. International Journal of Signal Processing， Image Processing and Pattern Recognition， 7（3）： 237-246 ［DOI： 10.14257/ijsip.2014.7.3.19http://dx.doi.org/10.14257/ijsip.2014.7.3.19］
Xie X Y， Xu Q Z and Hu L. 2016. Fast ship detection from optical satellite images based on ship distribution probability analysis//Proceedings of the 4th International Workshop on Earth Observation and Remote Sensing Applications （EORSA）. Guangzhou， China： IEEE： 97-101 ［DOI： 10.1109/EORSA.2016.7552774http://dx.doi.org/10.1109/EORSA.2016.7552774］
Xu J， Sun X， Zhang D B and Fu K. 2014. Automatic detection of inshore ships in high-resolution remote sensing images using robust invariant generalized hough transform. IEEE Geoscience and Remote Sensing Letters， 11（12）： 2070-2074 ［DOI： 10.1109/LGRS.2014.2319082http://dx.doi.org/10.1109/LGRS.2014.2319082］
Xu Y C， Fu M T， Wang Q M， Wang Y K， Chen K， Xia G S and Bai X. 2021. Gliding vertex on the horizontal bounding box for multi-oriented object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence， 43（4）： 1452-1459 ［DOI： 10.1109/TPAMI.2020.2974745http://dx.doi.org/10.1109/TPAMI.2020.2974745］
Yang X， Sun H， Fu K， Yang J R， Sun X， Yan M L and Guo Z. 2018a. Automatic ship detection in remote sensing images from google earth of complex scenes based on multiscale rotation dense feature pyramid networks. Remote Sensing， 10（1）： #132 ［DOI： 10.3390/rs10010132http://dx.doi.org/10.3390/rs10010132］
Yang X， Sun H， Sun X， Yan M L， Guo Z and Fu K. 2018b. Position detection and direction prediction for arbitrary-oriented ships via multitask rotation region convolutional neural network. IEEE Access， 6： 50839-50849 ［DOI： 10.1109/ACCESS.2018.2869884http://dx.doi.org/10.1109/ACCESS.2018.2869884］
Yang X， Yan J C， Feng Z M and He T. 2021. R3Det： refined single-stage detector with feature refinement for rotating object//Proceedings of the AAAI Conference on Artificial Intelligence. San Francisco， USA： AAAI： 35（4）： 3163-3171 ［DOI： 10.1609/aaai.v35i4.16426http://dx.doi.org/10.1609/aaai.v35i4.16426］
Yang Z， Luo T G， Wang D， Hu Z Q， Gao J and Wang L W. 2018c. Learning to navigate for fine-grained classification//Proceedings of the 15th European Conference on Computer Vision. Munich， Germany： Springer： 438-454 ［DOI： 10.1007/978-3-030-01264-9_ 26http://dx.doi.org/10.1007/978-3-030-01264-9_26］
Yu D H， Zhang B M， Guo H T， Zhao C and Xu J F. 2018. Joint salient feature and convolutional neural network for ship detection in remote sensing images. Journal of Image and Graphics， 23（12）： 1947-1958
余东行， 张保明， 郭海涛， 赵传， 徐俊峰. 2018. 联合显著性特征与卷积神经网络的遥感影像舰船检测. 中国图象图形学报， 23（12）： 1947-1958 ［DOI： 10.11834/jig.180323http://dx.doi.org/10.11834/jig.180323］
Zhang X H， Lv Y F， Yao L B， Xiong W and Fu C L. 2020. A new benchmark and an attribute-guided multilevel feature representation network for fine-grained ship classification in optical remote sensing images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing， 13： 1271-1285 ［DOI： 10.1109/JSTARS.2020.2981686http://dx.doi.org/10.1109/JSTARS.2020.2981686］
相关作者
相关机构
京公网安备11010802024621