Sparsity image inpainting algorithm based on similar patch group

Xingxin Lou; Xianghong Tang; Yue Zhang

doi:10.11834/jig.180567

Image Processing and Coding | Views : 0 下载量: 229 CSCD: 1

PDF
Export
Share
Collection
Album

Sparsity image inpainting algorithm based on similar patch group
Vol. 24, Issue 7, Pages: 1055-1066(2019)
Received：21 September 2018，

Revised：2018-12-7，

Published：16 July 2019
DOI： 10.11834/jig.180567
稿件说明：

移动端阅览

DOI：

Xingxin Lou, Xianghong Tang, Yue Zhang. Sparsity image inpainting algorithm based on similar patch group[J]. Journal of Image and Graphics, 2019, 24(7): 1055-1066. DOI： 10.11834/jig.180567.

摘要

目的

传统多样本块稀疏表示的图像修复过程中，容易因为匹配样本块的错误，重构出不正确的填充块，致使在边缘部分产生不连贯的现象，然而单样本匹配填充又容易产生明显的伪影，不能保证与周围结构保持连续，以致产生明显的修复痕迹，为此，本文提出一种改进的基于相似匹配块组的稀疏表示方法。

方法

首先，采用颜色信息与余弦距离结合的方法定义图像块匹配准则，在目标邻域范围内获得结构变化趋势更相似的匹配块组；然后，在稀疏重构过程中，同时考虑已知信息和估计的未知信息，利用相似块与目标块的匹配程度，对稀疏系数增加不同的权重，以此来增强筛选匹配块的能力，减少纹理模糊现象；最后，根据结构稀疏度自适应地在各结构复杂度不同的区域确定样本块尺寸，减少图像修复过程中的错误传播现象。

结果

实验结果表明，本文算法改善了在边缘修复过程中产生断裂或者纹理延伸的现象，不仅在主观视觉有明显的提升，其修复结果的峰值信噪比（PSNR）相较于其他相关稀疏表示修复算法提高了0.53 dB。

结论

本文算法实现了对不同结构特征的彩色破损图像的修复，在结构边缘处有理想的修复效果，并且对各种形状的破损也具有良好的修复效果。

Abstract

Objective

In the process of traditional patch-group-based image inpainting

synthetizing the incorrect filling patch is easy

thus causing incoherence at the edge because of incorrect matching patches. However

a traditional exemplar-based image inpainting algorithm only uses a single patch to fill a damaged area

making it prone to generate strong artifacts. It could give rise to strong inpainting trace

which affects a person's visual sense

because of its uncertain consistency with the surrounding structure. To solve these problems

an improved sparsity image inpainting algorithm based on a similar patch group is proposed in this study.

Method

First

searching for similar patch groups only uses color differences

thus making it easy to match incorrect patches whose structural trend is obviously different from the target patch. The cosine distance is an effective tool for measuring the change in direction between vectors

and it can measure the proximity in the vector direction. Thus

the patch matching criterion is defined by the combination of color information and cosine distance in this study. In this manner

matching patches whose structural trend is similar to the target patch can be obtained

and the error matching rate can be reduced when matching similar patches. Second

in the process of sparse reconstruction

to enhance the capability to filter matched patches and decrease texture blur

we estimate the unknown pixels according to the similar patch group obtained in the first step. Considering that different matching patches have different filling effects on the filling area and to make the whole sparse representation model capable of filtering matching patches

we utilize known and estimated unknown information to calculate the matching degree between every similar patch and the target patch to add different weights on the sparse coefficients. Finally

repairing the detailed structure of the area with a fixed patch size is difficult because an image always has rich and varied information. Structural sparsity is used to reflect structural complexity; thus

the patch size is adaptively decided by structural sparsity in this study. Different from the method that uses a fixed threshold to determine the patch size

the proposed method iteratively calculates structural sparsity and decreases the patch size until the structural complexity of the patch is reduced to the average value

which helps reduce error propagation during image inpainting.

Result

This study shows several representative color images that are damaged artificially into different shapes for inpainting in

Fig. 5

—

. In addition

it compares the repair results for the same image damaged with different kinds of shapes and provides a detailed view of the result in

Fig. 9

. Experimental results show that the proposed algorithm is more effective in subjective vision than other related algorithms and improves the phenomenon of fracture or texture extension during the process of edge structure repair. Other experimental data are given in

Tables 1

. The results show a significant improvement in subjective vision

and the peak signal-to-noise ratio of the result is improved to approximately 0.53 dB. However

the running time is increased because the method needs to calculate the patch structure sparsity iteratively and adjust the adaptive patch size dynamically.

Conclusion

This study proposes a weighted sparsity image inpainting method based on a similar patch group. The method combines the features of color information and cosine distance so that trends similar to the target patch can be obtained while searching for similar patch groups. Thus

the method is precise during patch propagation. Moreover

different sparse weights are added according to the degree of similarity; thus

the structural information is preserved as much as possible during the entire process of inpainting

and texture blur is reduced. The combination of the two strategies can improve the efficiency of the image inpainting algorithm. The proposed inpainting algorithm achieves the inpainting of damaged color images with different structural features and shapes. We perform a comparison to prove that the proposed method is more efficient than existing color image inpainting methods. Results show that the improved method in this study has an ideal repair effect on the strong edge structure and multi-structure area

thus effectively improving the image restoration quality. Nevertheless

this method only utilizes the information within a certain range of the image. Therefore

several shortcomings

such as the unsatisfactory repairing result in the case of inconsistent edge structure and the inability to maintain edge curvature

still exist. Thus

our method is unsuitable for all kinds of images. Future research will focus on the recovery of the edge curve and irregular texture breakage.

关键词

Keywords

references

Bertalmio M, Sapiro G, Caselles V, et al. Image inpainting[C]//Proceedings of the 27th Annual Conference on Computer Graphics. New York, USA: ACM, 2000: 417-424.[ DOI: 10.1145/344779.344972 http://dx.doi.org/10.1145/344779.344972 ]

Criminisi A, Pérez P, Toyama K. Region filling and object removal by exemplar-based image inpainting[J]. IEEE Transactions on Image Processing, 2004, 13(9):1200-1212.[DOI:10.1109/TIP.2004.833105]

Aharon M, Elad M, Bruckstein A. K-SVD:an algorithm for designing overcomplete dictionaries for sparse representation[J]. IEEE Transactions on Signal Processing, 2006, 54(11):4311-4322.[DOI:10.1109/TSP.2006.881199]

Filipovic' M, Kopriva I, Cichocki A. Inpainting color images in learned dictionary[C]//Proceedings of the 20th European Signal Processing Conference. Bucharest: IEEE, 2012: 66-70.

Kang J L, Tang X H, Zhang D, et al. Image inpainting by characteristic classification learning and patch sparsity propagation[J]. Journal of Computer-Aided Design&Computer Graphics, 2015, 27(5):864-872.

康佳伦, 唐向宏, 张东, 等.特征分类学习的结构稀疏传播图像修复方法[J].计算机辅助设计与图形学学报, 2015, 27(5):864-872. [DOI:10.3969/j.issn.1003-9775.2015.05.012]

Shen B, Hu W, Zhang Y M, et al. Image inpainting via sparse representation[C]//Proceedings of 2009 IEEE International Conference on Acoustics, Speech, and Signal Processing. Taipei, China: IEEE, 2009: 697-700.[ DOI: 10.1109/ICASSP.2009.4959679 http://dx.doi.org/10.1109/ICASSP.2009.4959679 ]

Xu Z B, Sun J. Image inpainting by patch propagation using patch sparsity[J]. IEEE Transactions on Image Processing, 2010, 19(5):1153-1165.[DOI:10.1109/TIP.2010.2042098]

Shi J G, Qi C. Sparse modeling based image inpainting with local similarity constraint[C]//Proceedings of 2013 IEEE International Conference on Image Processing.Melbourne: IEEE, 2013: 1371-1375.[ DOI: 10.1109/ICIP.2013.6738282 http://dx.doi.org/10.1109/ICIP.2013.6738282 ]

Li Z D, He H J, Yin Z K, et al. A sparsity image inpainting algorithm combining color with gradient information[J]. Journal of Computer Research and Development, 2014, 51(9):2081-2093.

李志丹, 和红杰, 尹忠科, 等.结合颜色和梯度信息的稀疏图像修复算法[J].计算机研究与发展, 2014, 51(9):2081-2093. [DOI:10.7544/issn1000-1239.2014.20130071]

Ghorai M, Mandal S, Chanda B. Patch sparsity based image inpainting using local patch statistics and steering kernel descriptor[C]//Proceedings of the 23rd International Conference on Pattern Recognition. Cancun: IEEE, 2017: 781-786.[ DOI: 10.1109/ICPR.2016.7899730 http://dx.doi.org/10.1109/ICPR.2016.7899730 ]

Rubinstein R, Bruckstein A M, Elad M. Dictionaries for sparse representation modeling[J]. Proceedings of 2010 IEEE, 2010, 98(6):1045-1057.[DOI:10.1109/JPROC.2010.2040551]

Donoho D L, Huo X. Uncertainty principles and ideal atomic decomposition[J]. IEEE Transactions on Information Theory, 2001, 47(7):2845-2862.[DOI:10.1109/18.959265]

Lee H, Battle A, Raina R, et al. Efficient sparse coding algorithms[C]//Proceedings of teh 20th Annual Conference on Neural Information Processing Systems. Vancouver, Canada, 2006: 801-808.[ DOI: http://dx.doi.org/ http://dx.doi.org/http://dx.doi.org/ ]

Wang Z, Bovik A C, Sheikh H R, et al. Image quality assessment:from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4):600-612.[DOI:10.1109/TIP.2003.819861]

Alert me when the article has been cited

提交

Improved Criminisi algorithm based on structure tensor

Sparse constraint and spatial-temporal regularized correlation filter for UAV tracking

Brain tumor segmentation based on prior sparse shapes

Image completion method with adaptive patch size