Improved Criminisi algorithm based on structure tensor

Yuting He; Xianghong Tang; Yue Zhang; Rui Yang

doi:10.11834/jig.170650

Image Processing and Coding | Views : 0 下载量: 5 CSCD: 4

PDF
Export
Share
Collection
Album

Improved Criminisi algorithm based on structure tensor
Vol. 23, Issue 10, Pages: 1492-1507(2018)
Received：22 December 2017，

Revised：2018-4-7，

Published：16 October 2018
DOI： 10.11834/jig.170650
稿件说明：

移动端阅览

DOI：

Yuting He, Xianghong Tang, Yue Zhang, Rui Yang. Improved Criminisi algorithm based on structure tensor[J]. Journal of Image and Graphics, 2018, 23(10): 1492-1507. DOI： 10.11834/jig.170650.

摘要

目的

针对传统基于样本块的图像修复算法中仅利用图像的梯度信息和颜色信息来修复破损区域时，容易产生错误填充块的问题，本文在Criminisi算法的基础上，利用结构张量特性，提出了一种改进的基于结构张量的彩色图像修复算法。

方法

首先利用结构张量的特征值定义新的数据项，以确保图像的结构信息能够更加准确地传播；然后利用该数据项构成新的优先权函数，使得图像的填充顺序更加精准；最后利用结构张量的平均相干性来自适应选择样本块大小，以克服结构不连续和错误延伸的缺点；同时在匹配准则中，利用结构张量特征值来增加约束条件，以减少错误匹配率。

结果

实验结果表明，改进算法的修复效果较理想，在主观视觉上有明显的提升，其修复结果的峰值信噪比（PSNR）和结构相似度（SSIM）都有所提高；与传统Criminisi算法相比，其峰值信噪比提高了1~3 dB。

结论

本文算法利用结构张量的特性实现了对不同结构特征的彩色破损图像的修复，对复杂的线性结构和纹理区域都有较理想的修复，有效地保持了图像边缘结构的平滑性，而且对大物体的移除和文字去除也有较好的修复效果。

Abstract

Objective

The rapid development of multimedia information technology has led to images that have become the main carrier of information in people's lives.People communicate information through various means

such as voice

images

text

and video.Consequently

digital image inpainting technology has gradually attracted increasing attention

and its application fields are extensive.Digital image inpainting refers to the process of repairing or rebuilding missing information in damaged images by using a specific image inpainting algorithm

such that the observer cannot easily detect that the image has been repaired or damaged.Image inpainting technology has been used in many research areas

such as the restoration of old photos

the removal of image text

and the preservation of cultural relics.The traditional exemplar-based image inpainting algorithm only uses gradient information and the color information of the image to repair damaged areas

which can easily generate incorrect filling patches.In addition

the definition of the priority function is unreasonable

and thus causes the wrong filling order during inpainting

which affects the overall restoration effect.To solve these problems

an improved color image inpainting algorithm that is based on structure tensor is presented in this paper.

Method

Structure tensor is often used to analyze the local geometry of an image

which not only contains the intensity information of the local region

but also the main directions of the neighborhood gradient of a pixel and the degree of coherence of these directions.Its two eigenvalues can distinguish the edge

texture

and flat areas of an image.First

the proposed algorithm uses the structure tensor to define the data items to ensure that the structure information of the image can be transmitted accurately

and then uses the data items to form a new priority function for a more precise filling order.Second

different sizes of sample patches can be used to search for the best matching patch because an image has different structural features in different regions.Therefore

the size of the sample patch is adaptively selected according to the average coherence of the structure tensor.In other words

when the average coherence of the patch to be repaired is large

this patch is at the edge of the image and a small sample patch should be used; when the average coherence is small

it is in the flat region of the image and a large sample patch should be used.In this manner

when repairing complex damaged images

the continuity of the edge structure can be maintained; the flat area of the image can be effectively repaired.Finally

the traditional inpainting algorithm only uses the color information of the image to find the best matching patch

which renders the matching patch sub optimal.In this study

the eigenvalues of the structure tensor are added to the matching criteria to reduce the false matching rate.

Result

Experimental results show that the improved algorithm is more effective in subjective vision than the other related algorithms.Moreover

the improved algorithm can achieve good results for different types of damaged images and effectively maintain the smoothness of the edge structure of the image.Compared with the traditional Criminisi algorithm

the power signal-to-noise ratio of the result has improved by approximately 1~3 dB and enhanced structure similarity.In addition

the proposed algorithm has a higher running time than other algorithms because in the inpainting process

the proposed algorithm uses the adaptive sample patch size to search for the best matching patch

and when analyzing the local structural features of the image

it needs to calculate the coherence factor of the pixels.These steps consequently increase the running time and reduce the efficiency of image inpainting.

Conclusion

When the traditional algorithm repairs the strong damaged area of the edge

the structural integrity and the good visual effect of the image are difficult to balance.In this study

we use the structure tensor of the color image to analyze the structure and texture area of the image.This paper discusses a color image inpainting algorithm based on the structure tensor.The proposed algorithm first uses the eigenvalue of the structure tensor instead of the isophote line in the traditional algorithm to improve the data items

which can spread the structure information of the image more accurately.Then

the average coherence factor of the structure tensor is used to analyze the texture and structural features of the image to repair the different image structural features.Finally

the matching rate used to select the best matching patch is improved with the addition of the constraint of the structural tensor to the traditional matching criteria.The proposed algorithm can obtain a better visual effect for damaged images with different structural features.The proposed algorithm can also effectively maintain the structural integrity of the image

and the complex texture area does not exhibit a wrong filled patch.Moreover

the large object and text removals by the proposed algorithm also have a good restoration effect.Compared with related Criminisi algorithms

the proposed algorithm has a better repair effect on complex linear structure and texture regions and effectively improves the overall quality of image restoration.

关键词

Keywords

references

Bertalmio M, Sapiro G, Caselles V, et al.Image inpainting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques.New York, USA: ACM, 2000: 417-424.[ DOI: 10.1145/344779.344972 http://dx.doi.org/10.1145/344779.344972 ]

Shen J H, Chan T F.Mathematical models for local nontexture inpaintings[J].SIAM Journal on Applied Mathematics, 2002, 62(3):1019-1043.[DOI:10.1137/S0036139900368844]

Shen J H, Kang S H, Chan T F.Euler's elastica and curvature-based inpainting[J].SIAM Journal on Applied Mathematics, 2002, 63(2):564-592.[DOI:10.1137/S0036139901390088]

Tsai A, Yezzi A, Willsky A S.Curve evolution implementation of the Mumford-Shah functional for image segmentation, denoising, interpolation, and magnification[J].IEEE Transactions on Image Processing, 2001, 10(8):1169-1186.[DOI:10.1109/83.935033]

Criminisi A, Perez P, Toyama K.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on Image Processing, 2004, 13(9):1200-1212.[DOI:10.1109/TIP.2004.833105]

Ren S, Tang X H, Kang J L.Improved Criminisi algorithm with the texture and edge features[J].Journal of Image and Graphics, 2012, 17(9):1086-1091.

任澍, 唐向宏, 康佳伦.利用纹理和边缘特征的Criminisi改进算法[J].中国图象图形学报, 2012, 17(9):1085-1091. [DOI:10.11834/jig.20120906]

Zhou Y T, Li L, XiaK W.Research on weighted priority of exemplar-based image inpainting[J].Journal of Electronics (China), 2012, 29(1-2):166-170.[DOI:10.1007/s11767-012-0843-6]

Xu Z B, Sun J.Image inpainting by patch propagation using patch sparsity[J].IEEE Transactions on Image Processing, 2010, 19(5):1153-1165.[DOI:10.1109/TIP.2010.2042098]

Hou Y T, Peng J Y, Han D C.Research of improved Criminisi image resoration algorithm[J].Computer Engineering and Applications, 2015, 51(11):135-138, 144.

侯玉婷, 彭进业, 韩东辰.改进的Criminisi图像修复算法研究[J].计算机工程与应用, 2015, 51(11):135-138, 144. [DOI:10.3778/j.issn.1002-8331.1306-0276]

Choi J H, Hahm C H.An exemplar-based image inpainting method with search region prior[C]//Proceedings of the 2nd Global Conference on Consumer Electronics.Tokyo: IEEE, 2013: 68-71.[ DOI:10.1109/GCCE.2013.6664927 http://dx.doi.org/10.1109/GCCE.2013.6664927 ]

Patel K R, Jain L.A novel approach to exemplar based image inpainting[C]//Proceedings of 2016 International Conference on Communication and Signal Processing.Melmaruvathur, India: IEEE, 2016: 0810-0814.[ DOI: 10.1109/ICCSP.2016.7754257 http://dx.doi.org/10.1109/ICCSP.2016.7754257 ]

Ren S, Tang X H, Kang J L.An image inpainting algorithm combined with texture and edge features[J].Journal of Computer-Aided Design&Computer Graphics, 2013, 25(11):1682-1693.

任澍, 唐向宏, 康佳伦.纹理和边缘特征相结合的图像修复算法[J].计算机辅助设计与图形学学报, 2013, 25(11):1682-1693.

Liu Y, Liu C J, Zou H L, et al.A novel exemplar-based image inpainting algorithm[C]//Proceedings of International Conference on Intelligent Networking and Collaborative Systems.Taipei, Taiwan, China: IEEE, 2015: 86-90.[ DOI: 10.1109/INCoS.2015.15 http://dx.doi.org/10.1109/INCoS.2015.15 ]

Siadati S Z, Yaghmaee F, Mahdavi P.A new exemplar-based image inpainting algorithm using image structure tensors[C]//Proceeding of the 24th Iranian Conference on Electrical Engineering.Shiraz, Iran: IEEE, 2016: 995-1001.[ DOI: 10.1109/IranianCEE.2016.7585666 http://dx.doi.org/10.1109/IranianCEE.2016.7585666 ]

Wang J, Lu K, Pan D R, et al.Robust object removal with an exemplar-based image inpainting approach[J].Neurocomputing, 2014, 123:150-155.[DOI:10.1016/j.Neucom.2013.06.022]

Akl A, Saad E, Yaacoub C.Structure-based image inpainting[C]//Proceedings of 2017 International Conference on Image Processing Theory, TOOLS and Applications.Oulu, Finland: IEEE, 2017.[ DOI: 10.1109/IPTA.2016.7820976 http://dx.doi.org/10.1109/IPTA.2016.7820976 ]

Akl A, Yaacoub C, Donias M, et al.Texture synthesis using the structure tensor[J].IEEE Transactions on Image Processing, 2015, 24(11):4082-4095.[DOI:10.1109/TIP.2015.2458701]

Meur O L, Gautier J, Guillemot C.Examplar-based inpainting based on local geometry[C]//Proceedings of the 18th IEEE International Conference on Image Processing.Brussels, Belgium: IEEE, 2011: 3401-3404.[ DOI: 10.1109/ICIP.2011.6116441 http://dx.doi.org/10.1109/ICIP.2011.6116441 ]

Wang Z, Bovik A C, Sheikh H R, et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing, 2004, 13(4):600-612.[DOI:10.1109/TIP.2003.819861]

Alert me when the article has been cited

提交

Sparsity image inpainting algorithm based on similar patch group

Infrared and visible image fusion based on multi-level guided network

A Novel Approach on Edge Preserving Anisotropic Diffusion

Application of Coherence-enhancing Diffusion in the Image Inpainting

Texture segmentation based on the use of the structure tensor and the wasserstein distance