Ancient mural classification with AlexNet model using feature fusion

Jianfang Cao; Hongyan Cui; Qi Zhang

doi:10.11834/jig.190221

Image Analysis and Recognition | Views : 0 下载量: 0 CSCD: 3

Export
Share
Collection
Album

Ancient mural classification with AlexNet model using feature fusion
Vol. 25, Issue 1, Pages: 92-101(2020)
Received：24 May 2019，

Revised：2019-8-21，

Accepted：28 August 2019，

Published：16 January 2020
DOI： 10.11834/jig.190221
稿件说明：

移动端阅览

DOI：

Jianfang Cao, Hongyan Cui, Qi Zhang. Ancient mural classification with AlexNet model using feature fusion[J]. Journal of Image and Graphics, 2020, 25(1): 92-101. DOI： 10.11834/jig.190221.

摘要

目的

针对古代壁画图像自身特征提取存在的主观单一性和客观不充分性等问题，以经典AlexNet网络模型为基础，提出了一种结合特征融合思想的卷积神经网络模型，用于古代壁画图像的自动分类。

方法

首先，由于大型壁画数据集较为缺乏，通过对壁画样本使用缩放、亮度变换、加噪和翻转等图像增强算法来扩大数据集，并提取壁画图像第1阶段的边缘等底层特征；其次，采用结构不同的双通道网络对提取的第1阶段特征进行第2阶段的深层抽象，得到两个通道的特征；最后，融合两个通道的特征，共同构建损失函数得到分类结果，从而提高模型的鲁棒性和特征表达能力。

结果

实验结果表明，在构造的壁画图像数据集上，该模型最终达到了85.39%的准确率。与AlexNet模型以及一些改进的卷积神经网络模型相比，各项评价指标均有大约5%的提高；与未进行预训练的经典模型相比，本文网络结构不易产生过拟合现象；与结合预训练的经典模型相比，准确率大致上有1%~5%的提升，从硬件条件、网络结构和内存消耗上来说代价更小。由此验证了本文模型对于壁画图像自动分类的合理性和有效性。

结论

本文提出的壁画分类模型，综合考虑网络宽度和深度的影响，能从多局部的角度提取壁画图像丰富的细节特征，具有一定的优势和使用价值，可进一步结合到与壁画图像分类的相关模型中。

Abstract

Objective

Chinese ancient murals

as a type of painting on the wall

have a long history of 4000 years and are an indispensable part of Chinese ancient paintings. With the increasing abundance of digital mural images

classifying mural resources is becoming increasingly urgent. The core of mural image classification is how to construct the feature description of an object. In addition to expressing the object adequately

this description should be able to distinguish among different types of objects. However

ancient mural images have certain pluralism and subjectivity due to artificial drawing. Considering the subjective singularity and objective insufficiency of traditional mural image feature extraction

we propose a convolutional neural network based on classical AlexNet network model and feature fusion idea for the automatic classification of ancient mural images.

Method

First

we define the optimizer as Adam with a learning rate of 0.001 through experiments. We then extract each convolution layer feature of AlexNet for classification. Through the comparison of running time and accuracy

we select the convolution layer to express mural features further. Second

we combine the idea of feature fusion and exchange the two convolution kernels to form channels 1 and 2. The convolution kernels of channel 1 are 11

and 3

and those of channel 2 are 11

and 5. The combination of this method constitutes a two-channel convolution feature extraction layer

which enables the model to utilize multilocal features fully. The overfitting phenomenon caused by numerous full-connection layers is considered. On the basis of the two-channel convolution feature extraction layer

we continue to compare the features of different full-connection layers and select further appropriate full-connection layer features to express mural images. Finally

a mural image classification model with a two-channel convolution layer and optimal full connection layer is presented. The proposed mural image classification model can be divided into three processes. 1) Mural image preprocessing. Given the lack of large mural datasets

we use image enhancement operations

such as zooming

brightness transformation

noise addition

and flipping

to enlarge the mural samples. An ancient mural image dataset

including Buddha

Bodhisattva

Buddhist disciples

secular figures

animals

plants

buildings

and auspicious cloud

is constructed. 2) Training stage of mural image classification model. The module has three stages. In the first stage

the model extracts the low-level features

such as the edge information of the trainset images. In the second stage

the two-channel network with different structures is used to abstract the features of the first stage. The features of the two channels are then obtained. In the last stage

the loss function training network model is constructed by fusing the features of the two channels. Feature fusion improves the robustness of the model and the capability of feature expression. 3) Training stage of mural image classification model. We use the network model with trained parameters to predict the classification results of test set samples. The classification accuracy

recall

and f1-score are obtained.

Result

Through the comparison of running time and accuracy

the comparative experimental results of different convolution layers show that in the AlexNet model

the third convolution layer is the most suitable network layer for this dataset. In addition

the accuracy rate will decrease if the number of layers is higher or lower than the number of layers in the paper. Similarly

the comparative experimental results of different full-connection layers show that the features of the three-layer full-connection layer are further stable and sufficient based on the two-channel convolution extraction layer. Therefore

a six-layer convolution neural network model

including a three-layer dual channel and three-layer full connection layer

is presented with five convolution layers in the two-channel model. The model achieves 85.39% accuracy on the constructed mural image dataset. Experimental results show that the accuracy of the model in most classes is the highest

and each evaluation index of the model is improved by approximately 5%

compared with the AlexNet model and several improved convolution neural network models. Compared with the classical model without pretraining

this model encounters increased difficulty in producing overfitting. Compared with the model with pretraining

the accuracy of the model is improved by approximately 1%~5%

and the cost is reduced in terms of hardware conditions

network structure

and memory consumption. These experimental data verify the validity of the model for the automatic classification of mural images.

Conclusion

Considering the influence of network width and depth

ancient mural classification model with AlexNet model using feature fusion can fully express the rich details of mural images. This model has certain advantages and application value and can be further integrated into the mural classification-related model. However

this method is a shallow convolution neural network based on AlexNet

which fails to mine the high-level features of mural images fully. As a result

some images with similar low-level features

such as color and texture

cannot be classified correctly. Moreover

the running time of mural classification in this model is measured by hour

which consumes considerable resources and is inefficient. Therefore

we will combine deep models to express the high-level features of mural images in future work. We will also improve the efficiency of model training to make mural classification further effective and fast.

关键词

Keywords

references

Almisreb A A, Jamil N and Din N M. 2018. Utilizing Alexnet deep transfer learning for ear recognition//Proceedings of the 2018 4th International Conference on Information Retrieval and Knowledge Management. Kota Kinabalu, Malaysia: IEEE: 1-5[ DOI:10.1109/infr km .2018.8464769 http://dx.doi.org/10.1109/infrkm.2018.8464769 ]

Alom M Z, Taha T M, Yakopcic C, Westberg S, Sidike P, Nasrin M S, van Esesn B C, Awwal A A S and Asari V K. 2018. The history began from AlexNet: a comprehensive survey on deep learning approaches[EB/OL]. 2018-03-03[2018-09-12] . https://arxiv.org/abs/1803.01164 https://arxiv.org/abs/1803.01164

Caldelli R, Becarelli R and Amerini I. 2017. Image origin classification based on social network provenance. IEEE Transactions on Information Forensics and Security, 12(6):1299-1308[DOI:10.1109/tifs.2017.2656842]

Chen C and Qi F. 2019. Review on development of convolutional neural network and its application in computer vision. Computer Science, 46(3):63-73

陈超, 齐峰. 2019.卷积神经网络的发展及其在计算机视觉领域中的应用综述.计算机科学, 46(3):63-73)[DOI:10.11896/j.issn.1002-137X.2019.03.008]

Hao Y B. 2017. Research and Implementation on Classification Algorthm with People of Ancient Chinese Murals Based on Style Characteristics. Tianjin:Tianjin University

郝亚博. 2017.基于风格特征的中国古代人物壁画分类算法研究与实现.天津:天津大学)

He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 770-778[ DOI:10.1109/cvpr.2016.90 http://dx.doi.org/10.1109/cvpr.2016.90 ]

Krizhevsky A, Sutskever I and Hinton G E. 2012. ImageNet classification with deep convolutional neural networks//Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe, Nevada: Curran Associates Inc.: 1097-1105

Liu L M. 2016. Research of Rapid Three-dimensional Reconstruction Based Digital of Ultra-high Resolution Mural. Hangzhou:Zhejiang University

刘礼铭. 2016.基于3维重建的超高分辨率壁画快速数字化技术研究.杭州:浙江大学)

Liu X J. 2014. Research on Feature Extraction and Evolution Patterns of Auspicious Cloud in the Dunhuang Grotto murals. Wuhan:Wuhan University of Technology

刘喜姣. 2014.敦煌壁画中祥云文化元素的特征提取及演化规律研究.武汉:武汉理工大学)

Shin H C, Roth H R, Gao M C, Lu L, Xu Z Y, Nogues I, Yao J H, Mollura D and Summers R M. 2016. Deep convolutional neural networks for computer-aided detection:CNN architectures, dataset characteristics and transfer learning. IEEE Transactions on Medical Imaging, 35(5):1285-1298[DOI:10.1109/TMI.2016.2528162]

Simonyan K and Zisserman A. 2014. Very deep convolutional networks for large-scale image recognition[EB/OL]. 2014-09-24[2015-04-10] . https://arxiv.org/pdf/1409.1556.pdf https://arxiv.org/pdf/1409.1556.pdf

Sun T. 2018. Undertaking the Responsibility for the Revival of Chinese Mural Art. Art Research, (5):17-26

孙韬. 2018.中央美术学院托起中国壁画复兴之重任.美术研究, (5):17-26)[DOI:10.13318/j.cnki.msyj.2018.05.005]

Szegedy C, Vanhoucke V, Ioffe S, Shlens J and Wojna Z. 2016. Rethinking the inception architecture for computer vision//Proceeding of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2818-2826[ DOI:10.1109/cvpr.2016.308 http://dx.doi.org/10.1109/cvpr.2016.308 ]

Tang D W, Lu D M, Xu D Q and Yang B. 2014. Clustered multiple instance learning for mural image classification. Journal of Image and Graphics, 19(5):708-715

唐大伟, 鲁东明, 许端清, 杨冰. 2014.壁画图像分类中的分组多实例学习方法.中国图象图形学报, 19(5):708-715)[DOI:10.11834/jig.20140508]

Tang D W, Lu D M, Yang B and Xu D Q. 2013. Similarity metrics between mural images with constraints of the overall structure of contours. Journal of Image and Graphics, 18(8):968-975

唐大伟, 鲁东明, 杨冰, 许端清. 2013.轮廓整体结构约束的壁画图像相似性度量.中国图象图形学报, 18(8):968-975)[DOI:10.11843/jig.20130811]

Wang Q and Lu D M. 2012. Composition analysis-based relevance ranking for ancient mural. Journal of Zhejiang University (Engineering Science), 46(3):392-401

王琦, 鲁东明. 2012.基于构图分析的古代壁画相关度评价方法.浙江大学学报(工学版), 46(3):392-401)[DOI:10.3785/j.issn.1008-973x.2012.03.003]

Wei B G, Pan Y H and Hua Z. 1999. An analogy-based virtual approach for color restoration of wall painting. Journal of Computer Research and Development, 36(11):1364-1368

魏宝刚, 潘云鹤, 华忠. 1999.基于类比的壁画色彩虚拟复原.计算机研究与发展, 36(11):1364-1368)

Wei W, Qian W. 2013. Review on the application of 3D laser scanning in conservation and archaeology. Sciences of Conservation and Archaeology, 25(1):96-107

魏薇, 潜伟. 2013. 3维激光扫描在文物考古中应用述评.文物保护与考古科学, 25(1):96-107)

Xiao L S, Yan Q and Deng S Y. 2017. Scene classification with improved AlexNet model//Proceedings of the 2017 12th International Conference on Intelligent Systems and Knowledge Engineering. Nanjing, China: IEEE: 1-6[ DOI:10.1109/iske.2017.8258820 http://dx.doi.org/10.1109/iske.2017.8258820 ]

Zeng P P and Li L S. 2019. Classification and recognition of common fruit images based on convolutional neural network. Machine Design and Research, 35(1):23-26, 34

曾平平, 李林升. 2019.基于卷积神经网络的水果图像分类识别研究.机械设计与研究, 35(1):23-26, 34)[DOI:10.13952/j.cnki.jofmdr.2019.0093]

Zhang J W and Wang Q W. 2018. Digital representation and reconstruction of architecture in Dunhuang frescoes. Journal of Dunhuang Studies, 3(3):187-196

张加万, 王巧雯. 2018.敦煌壁画中建筑的数字化再现与重构.敦煌学辑刊, 3(3):187-196)[DOI:10.3969/j.issn.1001-6252.2018.03.015]

Zhang L C, Ma R and Zhang Y X. 2018. Application of improved lenet-5 model in apple image recognition. Computer Engineering and Design, 39(11):3570-3575

张力超, 马蓉, 张垚鑫. 2018.改进的LeNet-5模型在苹果图像识别中的应用.计算机工程与设计, 39(11):3570-3575)[DOI:10.16208/j.issn1000-7024.2018.11.048]

Zhong Z Q, Yuan J and Tang X Y. 2018. Left-vs-right eye discrimination based on convolutional neural network. Journal of Computer Research and Development, 55(8):1667-1673

钟志权, 袁进, 唐晓颖. 2018.基于卷积神经网络的左右眼识别.计算机研究与发展, 55(8):1667-1673)[DOI:10.7544/issn1000-1239.2018.20180215]

Alert me when the article has been cited

提交

Contour perception based on multi-path convolution neural network

Cross-modal feature fusion and detail-enhanced RGB-D salient object detection

Two-stage vision transformer for fusing global and local features in distracted driving behavior recognition

MoLiNet： a local and global information interactive fusion network for improving multi-classification of pathological image artifacts

Image denoising via Swin Transformer V2 and feature fusion U-Net