浏览全部资源
扫码关注微信
Published： 16 August 2023 ，
移动端阅览
李新叶， 陈丁. 2023. 融合改进ASPP和极化自注意力的自底向上全景分割. 中国图象图形学报， 28(08):2410-2419
Li Xinye， Chen Ding. 2023. The improved atrous spatial pyramid pooling and polarized self-attention based bottom-up panoptic segmentation. Journal of Image and Graphics， 28(08):2410-2419
目的
2
针对ASPP（atrous spatial pyramid pooling）在空洞率变大时空洞（atrous）卷积效果会变差的情况，以及图像分类经典模型ResNet（residual neural network）并不能有效地适用于细粒度图像分割任务的问题，提出一种基于改进ASPP和极化自注意力的自底向上全景分割方法。
方法
2
重新设计ASPP模块，将小空洞率卷积的输出与原始输入进行拼接（concat），将得到的结果作为新的输入传递给大空洞率卷积，然后将不同空洞率卷积的输出结果拼接，并将得到的结果与ASPP中的其他模块进行最后拼接，从而改善ASPP中因空洞率变大导致的空洞卷积效果变差的问题，达到既获得足够感受野的同时又能编码多尺度信息的目的；在主干网络的输出后引入改进的极化自注意力模块，实现对图像像素级的自我注意强化，使其得到的特征能直接适用于细粒度像素分割任务。
结果
2
本文在Cityscapes数据集的验证集上进行测试，与复现的基线网络Panoptic-DeepLab（58.26%）相比，改进ASPP模块后分割精度PQ（panoptic quality）（58.61%）提高了0.35%，运行时间从103 ms增加到124 ms，运行速度没有明显变化；通过进一步引入极化自注意力，PQ指标（58.86%）提高了0.25%，运行时间增加到187 ms；通过对该注意力模块进一步改进，PQ指标（59.36%）在58.86%基础上又提高了0.50%，运行时间增加到192 ms，速度略有下降，但实时性仍好于大多数方法。
结论
2
本文采用改进ASPP和极化自注意力模块，能够更有效地提取适合细粒度像素分割的特征，且在保证足够感受野的同时能编码多尺度信息，从而提升全景分割性能。
Objective
2
Panoptic segmentation can be as a challenging task in computer vision and image segmentation nowadays. It is focused on all objects-related segmentation in an image relevant to such categories of foreground “thing” and background “stuff”. Panoptic segmentation can optimize semantic segmentation and instance segmentation to a certain extent in relevance to such domain of vision applications like autonomous driving， simultaneous localization and mapping （SLAM）， multi-object tracking and segmentation （MOTS）. Most of panoptic segmentation methods can be used to follow the top-down path and the principle of detection before segmentation. Such method is based on instance segmentation or object detection， and a semantic branch is added to rich semantic segmentation. The segmentation performance of these models is feasible， but it needs a complex post-processing stage to deal with branches-between and within conflicts， which can make the inference be slower. Another category of these methods can follow the idea of bottom-up， for which semantic segmentation can be regarded as the basis， and the image can be recognized as a whole at the pixel level. It can be used to optimize tedious post-processing. Recently， a bottom-up panoptic segmentation （Panoptic-DeepLab） is used to divide the panoptic segmentation task into two branches. Each branch has a specific decoder network and segmentation head network. The semantic segmentation head outputs the semantic segmentation results. The same structure-related two instance heads can be used to predict the center instance and offset simutaneously. It can get better segmentation accuracy and speed. However， the atrous spatial pyramid pooling （ASPP） module is still used in the decoder network to increase the receptive field. For ASPP， to obtain a large enough receptive field， it needs sufficient dilation rate. When the dilation rate is larger， the effect of atrous convolution is worse. On the other hand， residual neural network （ResNet） is used as a shared encoder， which may be sub-optimal for fine-grained image segmentation. To resolve the problems mentioned above， we develop a new panoptic segmentation model for better segmentation performance.
Method
2
A bottom-up panoptic segmentation method is developed in terms of improved ASPP and polarized self-attention. First， for ASPP， we redesigned it， called improved atrous spatial pyramid pooling （IASPP）. Specifically， 1） dilation rate of rate1-related output of 3 × 3 convolutions is concatenated with the original input， and it is input into 3 × 3 convolution with the dilation rate2； 2） dilation rate1 and rate2-related output of 3 × 3 convolutions is concatenated with the original input， and it is input into 3 × 3 convolution with the dilation rate of rate3. Then， different dilation rates-related output of convolution is concatenated as well. Finally， the results are obtained and concatenated with other ASPP-related modules. Through a series of atrous convolutions and feature concatenations， final output of the IASPP can obtain a larger receptive field without ASPP-related kernel degradation. Furthermore， the IASPP are not used to increase the size of the model significantly， and the speed of the model is not increased dramatically as well. In addition， polarization self-attention （PSA） can be used to enhance the feature extraction ability of the shared backbone further. After the fourth layer of ResNet-50 is concerned about， improved polarization self-attention （IPSA） module is introduced to extract pixel-level features. This process can enhance the ability of ResNet to extract cost-efficient pixel-level information. The output features can be used preserve pixel-level information， and it can be applied to typical fine-grained image segmentation tasks to estimate the highly nonlinear pixel-wise semantics straightfoward.
Result
2
The method is tested on the cityscapes dataset. The cityscapes dataset is composed of 19 categories， including 11 background and 8 foreground contexts. It consists of images samples of 2 975 training， 500 validation， and 1 525 test contexts. Each image has a size of 1 024 × 2 048 pixels approximately. The training set can be used to train the network and the validation set is used to test the network. Compared to the baseline， experimental results demonstrate that the proposed model’s panoptic quality （PQ） is improved from 58.26% to 58.61%， and the runtime is optimized from 103 ms to 124 ms when the improved atrous spatial pyramid pooling （IASPP） module is melted into. Additionally， after the addition of the polarized self-attention （PSA）， the PQ of the model is improved from 58.61% to 58.86% at the cost of the runtime from 124 ms to 187 ms. After improving the polarized self-attention （IPSA）， the PQ of the model is improved from 58.86% to 59.36% while the runtime is reached to 192 ms. We carried out visual experiments， including the visualization of the image， performance comparison of different categories， and comparison with other related methods further.
Conclusion
2
To optimize the bottom-up panoptic segmentation method， a panoptic segmentation method is developed based on improved ASPP （IASPP） and polarized self-attention （IPSA）. This redesigned ASPP method can resolve the problem of atrous convolution failure effectively derived of the increase of dilation rate in ASPP， and the introduction of IPSA can improve the ability of ResNet-50 to extract pixel-level fine-grained features， and rich pixel-level feature information can be preserved in the process of feature extraction to estimate the highly nonlinear pixel-wise semantics. To improve the comprehensive performance of panoptic segmentation， it cannot only achieve better segmentation accuracy， but also maintain a better speed further.
全景分割语义分割实例分割极化自注意力ASPP
panoptic segmentationsemantic segmentationinstance segmentationpolarized self-attentionatrous spatial pyramid pooling （ASPP）
Chen L C， Papandreou G， Schroff F and Adam H. 2017. Rethinking atrous convolution for semantic image segmentation ［EB/OL］. ［2022-03-10］. https://arxiv.org/pdf/1706.05587.pdfhttps://arxiv.org/pdf/1706.05587.pdf
Chen L C， Zhu Y K， Papandreou G， Schroff F and Adam H. 2018. Encoder-decoder with atrous separable convolution for semantic image segmentation//Proceedings of the 15th European Conference on Computer Vision. Munich， Germany： Springer： 833-851 ［DOI： 10.1007/978-3-030-01234-2_49http://dx.doi.org/10.1007/978-3-030-01234-2_49］
Chen X， Wang J R and Hebert M. 2020a. PanoNet： real-time panoptic segmentation through position-sensitive feature embedding ［EB/OL］. ［2022-03-10］. https://arxiv.org/pdf/2008.00192.pdfhttps://arxiv.org/pdf/2008.00192.pdf
Chen Y F， Lin G C， Li S Y， Bourahla O， Wu Y M， Wang F F， Feng J Y， Xu M L and Li X. 2020b. BANet： bidirectional aggregation network with occlusion handling for panoptic segmentation//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Seattle， USA： IEEE： 3792-3801 ［DOI： 10.1109/CVPR42600.2020.00385http://dx.doi.org/10.1109/CVPR42600.2020.00385］
Cheng B W， Collins M D， Zhu Y K， Liu T， Huang T S， Adam H and Chen L C. 2020. Panoptic-DeepLab： a simple， strong， and fast baseline for bottom-up panoptic segmentation//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Seattle， USA： IEEE： 12472-12482 ［DOI： 10.1109/CVPR42600.2020.01249http://dx.doi.org/10.1109/CVPR42600.2020.01249］
Cordts M， Omran M， Ramos S， Rehfeld T， Enzweiler M， Benenson R， Franke U， Roth S and Schiele B. 2016. The cityscapes dataset for semantic urban scene understanding//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Las Vegas， USA： IEEE： 3213-3223 ［DOI： 10.1109/CVPR.2016.350http://dx.doi.org/10.1109/CVPR.2016.350］
Geus D D， Meletis P and Dubbelman G. 2020. Fast panoptic segmentation network. IEEE Robotics and Automation Letters， 5（2）： 1742-1749 ［DOI： 10.1109/LRA.2020.2969919http://dx.doi.org/10.1109/LRA.2020.2969919］
He K M， Zhang X Y， Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Las Vegas， USA： IEEE： 770-778 ［DOI： 10.1109/CVPR.2016.90http://dx.doi.org/10.1109/CVPR.2016.90］
He K M， Gkioxari G， Dollr P and Girshick R. 2017. Mask R-CNN//Proceedings of 2017 IEEE International Conference on Computer Vision （ICCV）. Venice， Italy： IEEE： 2980-2988 ［DOI： 10.1109/ICCV.2017.322http://dx.doi.org/10.1109/ICCV.2017.322］
Hou R， Li J， Bhargava A， Raventos A， Guizilini V， Fang C， Lynch J and Gaidon A. 2020. Real-time panoptic segmentation from dense detections//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Seattle， USA： IEEE： 8520-8529 ［DOI： 10.1109/CVPR42600.2020.00855http://dx.doi.org/10.1109/CVPR42600.2020.00855］
Kirillov A， He K M， Girshick R， Rother C and Dollr P. 2019a. Panoptic segmentation//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Long Beach， USA： IEEE： 9396-9405 ［DOI： 10.1109/CVPR.2019.00963http://dx.doi.org/10.1109/CVPR.2019.00963］
Kirillov A， Girshick R， He K M and Dollr P. 2019b. Panoptic feature pyramid networks//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Long Beach， USA： IEEE： 6392-6401 ［DOI： 10.1109/CVPR.2019.00656http://dx.doi.org/10.1109/CVPR.2019.00656］
Lazarow J， Lee K， Shi K Y and Tu Z W. 2020. Learning instance occlusion for panoptic segmentation//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Seattle， USA： IEEE： 10717-10726 ［DOI： 10.1109/CVPR42600.2020.01073http://dx.doi.org/10.1109/CVPR42600.2020.01073］
Li X Y and Chen D. 2022. A survey on deep learning-based panoptic segmentation. Digital Signal Processing， 120： #103283 ［DOI： 10.1016/j.dsp.2021.103283http://dx.doi.org/10.1016/j.dsp.2021.103283］
Li Y W， Chen X Z， Zhu Z， Xie L X， Huang G， Du D L and Wang X G. 2019. Attention-guided unified network for panoptic segmentation//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Long Beach， USA： IEEE： 7019-7028 ［DOI： 10.1109/CVPR.2019.00719http://dx.doi.org/10.1109/CVPR.2019.00719］
Liu H J， Liu F Q， Fan X Y and Huang D. 2021. Polarized self-attention： towards high-quality pixel-wise regression ［EB/OL］. ［2022-03-10］. https://arxiv.org/pdf/2107.00782.pdfhttps://arxiv.org/pdf/2107.00782.pdf
Wang H C， Luo R T， Maire M and Shakhnarovich G. 2020. Pixel consensus voting for panoptic segmentation//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Seattle， USA： IEEE： 9461-9470 ［DOI： 10.1109/CVPR42600.2020.00948http://dx.doi.org/10.1109/CVPR42600.2020.00948］
Wang H Y， Zhu Y K， Adam H， Yuille A and Chen L C. 2021. MaX-DeepLab： end-to-end panoptic segmentation with mask transformers//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Nashville， USA： IEEE： 5459-5470 ［DOI： 10.1109/CVPR46437.2021.00542http://dx.doi.org/10.1109/CVPR46437.2021.00542］
Wang N， Hou Z Q， Pu L， Ma S G and Cheng H H. 2022. Real-time semantic segmentation analysis based on cavity separable convolution and attention mechanism. Journal of Image and Graphics， 27（4）： 1216-1225
王囡， 侯志强， 蒲磊， 马素刚， 程环环. 2022. 空洞可分离卷积和注意力机制的实时语义分割. 中国图象图形学报， 27（4）： 1216-1225 ［DOI： 10.11834/jig.200729http://dx.doi.org/10.11834/jig.200729］
Xiong Y W， Liao R J， Zhao H S， Hu R， Bai M， Yumer E and Urtasun R. 2019. UPSNet： a unified panoptic segmentation network//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Long Beach， USA： IEEE： 8810-8818 ［DOI： 10.1109/CVPR.2019.00902http://dx.doi.org/10.1109/CVPR.2019.00902］
Yang T J， Collins M D， Zhu Y K， Hwang J J， Liu T， Zhang X， Sze V， Papandreou G and Chen L C. 2019. DeeperLab： single-shot image parser ［EB/OL］. ［2022-03-10］. https://arxiv.org/pdf/1902.05093.pdfhttps://arxiv.org/pdf/1902.05093.pdf
Zhao H S， Qi X J， Shen X Y， Shi J P and Jia J Y. 2018. ICNet for real-time semantic segmentation on high-resolution images//Proceedings of the 15th European Conference on Computer Vision. Munich， Germany： Springer： 418-434 ［DOI： 10.1007/978-3-030-01219-9_25http://dx.doi.org/10.1007/978-3-030-01219-9_25］
相关文章
相关作者
相关机构
京公网安备11010802024621