发布时间: 2019-05-16 摘要点击次数: 全文下载次数: DOI: 10.11834/jig.180509 2019 | Volume 24 | Number 5 图像处理和编码

 收稿日期: 2018-09-12; 修回日期: 2018-10-31 基金项目: 国家自然科学基金项目（61462065，61661036） 第一作者简介: 张桂梅, 1970年生, 女, 教授, 主要研究方向为计算机视觉、图像处理和模式识别。E-mail:guimei.zh@163.com;李艳兵, 男, 南昌航空大学机械工程硕士研究生, 主要研究方向为计算机视觉、图像处理和模式识别。E-mail:472944577@qq.com. 中图法分类号: TP391.41 文献标识码: A 文章编号: 1006-8961(2019)05-0700-14

# 关键词

Image inpainting of fractional TV model combined with texture structure
Zhang Guimei, Li Yanbing
Key Laboratory of Image Processing and Pattern Recognition, Nanchang Hangkong University, Nanchang 330063, China
Supported by: National Natural Science Foundation of China (61462065, 61661036)

# Abstract

Objective As a fundamental issue in the field of image processing, image inpainting has been widely studied in the past two decades. The goal of image inpainting is to reconstruct the original high-quality image from its corrupted observation. Notably, prior knowledge of images is important in image inpainting. Thus, designing an effective regularization to represent image priors is a critical task for image inpainting. The TV(total variation) model usually exploits local structures and high effectiveness to preserve image edges; thus, it has been found to be widely used in image inpainting. However, the regular term of the TV model is a first-order differential, which usually loses image details and tends to suffer from over-smooth effects owing to the piecewise constant assumption. Fortunately, fractional differential is capable of enhancing low-and intermediate-frequency signals and amplifying high-frequency signals moderately; thus, it is introduced into the TV model. However, the existing fractional TV model is limited with regard to its preservation of the information of images with texture and edge details. Furthermore, it does not fully use prior information such as known edges and textures. Method To address these problems, we propose a new fractional-order TV model that introduces texture structure information into a fractional TV model for image inpainting. A minimum value is used in the TV model to calculate the image gradient when solving the fractional model. Thus, the improved model is robust because it overcomes the problem of the model being non-differentiable at zero point. In this way, the weak texture information is effectively preserved. The improved model determines the texture direction of the region to be restored on the basis of the priors of the known region of the image and fully uses the texture information of the image to improve the accuracy of image inpainting. Result The Barbara and Lena images are selected as test images. The Barbara image presents a large weak texture area. By contrast, the Lena image includes few texture regions and a highly smooth area. Therefore, these two images are used for the experiment. To improve efficiency, we intercept the texture part of the original image and conduct many experiments by using differently sized templates and different orders of fractional differential. Then, the optimal parameters for different images, such as template size and order, can be obtained. The optimal parameters for the Barbara and Lena images are as follows. For the Barbara image, the optimal order is 0.1, and the optimal template size is 3×3 pixels; for the Lena image, the optimal order is 0.9, and the optimal template size is 5×5 pixels. The algorithm is compared with three algorithms with better restoration effects. Mean square error (MSE) and peak signal-to-noise ratio (PSNR) are introduced to evaluate the performance of the different methods. Experimental results indicate that the proposed algorithm achieves improved inpainting result. Unlike that in the TV model, the PSNR values after the restoration of the Barbara, Lena, and Rock images in the proposed method increase by 5.94%, 8.07%, and 3.85%, respectively; and the MSE values decrease by 48.66%, 65.89%, and 35%, respectively. Relative to the fractional TV model, the proposed method achieves PSNR values for the Barbara image, Lena image, and Rock image that increase by 4.17%, 8.59%, and 1.81%, respectively; its MSE values decrease by 37.90%, 68.00%, and 18.68%, respectively. Conclusion The relationship between inpainting effect, template order, and template size is are demonstrated in experiments, thereby providing the basis for selecting optimal parameters. Although the optimal parameters of different types of images are different, the optimal inpainting order is generally between 0 and 1 because the smooth part of the image corresponds to the low-frequency part of the signal. The texture details of the image correspond to the intermediate-frequency part of the signal. Meanwhile, the TV algorithm is not ideal for the weak texture region. To enhance the gradient information of the region, we must improve the low-and intermediate-frequency parts. Therefore, choosing the order between 0 and 1 is recommended. Furthermore, although the optimal order varies with the type of the image, a weak texture region usually results in a small order. Theoretical analysis and experimental results show that the proposed model can effectively improve the accuracy of image restoration relative to the original TV model and fractional order TV model. The proposed model is suitable for inpainting images with weak texture and edge information. This model is an important extension of the TV model.

# Key words

image inpainting; fractional differential; weak texture; TV model; edge detail

# 0 引言

TV模型由于能去除图像中的噪声，近年来学者们将TV模型应用到图像修复中。如Chan等人[9]提出了将TV模型应用到图像修复中，该方法是基于偏微分方程的图像修复算法，但是其收敛速度较慢，且容易产生过度平滑。针对基于偏微分方程修复算法运行速度较慢的问题，Telea[10]提出了一种新的基于水平集的快速图像修复方法，该算法解决了修复算法运行速度较慢的问题，但是基于水平集的方法易造成区域模糊并且对边缘信息保持不佳。针对该问题，李开宇等人[11]提出一种改进方案，在权函数的设计中，引入连续强度来保持边缘信息，并利用等照度线的方向计算出两个像素点的位置关系。当修复单个点时，引入置信因子来加权插值点。该算法在保证运行效率的同时提高了修复精度，但是该方法并没有解决纹理过度平滑问题。文献[8]提出了分数阶与TV模型相结合的修复方法，相对原始的TV模型有了较大提升，能够非线性地保留图像平滑区域的纹理信息，有效地解决纹理过度平滑问题，对图像细节信息具有较好的修复效果。但是该方法对具有弱导数性质的纹理细节等信息的保持仍不够理想，并且在模型最小化过程中导致了计算上的困难，其主要原因是正则项和数据项在零点处不可微。

# 1.1 分数阶微分

G-L是比较经典的分数阶定义，它的实质是由整数阶推导出来的，根据整数阶微分的定义，在区间$t\in [a, b](a<b, a, b\in \bf{R})$内存在函数$f\left( t \right)$连续可微，可得该连续函数的1阶微分定义为

 ${f^\prime }(t) = \mathop {\lim }\limits_{h \to 0} \frac{{f(t + h) - f(h)}}{h}$ (1)

 $\begin{array}{*{20}{l}} {{f^{\prime \prime }}(t) = \mathop {\lim }\limits_{h \to 0} \frac{{{f^\prime }(t + h) - {f^\prime }(h)}}{h} = }\\ {\mathop {\lim }\limits_{h \to 0} \frac{{f(t + 2h) - 2f(t + h) + f(h)}}{{{h^2}}}} \end{array}$ (2)

 $\sum\limits_{\begin{array}{*{20}{c}} {X \in \boldsymbol{B}}\\ {x \in \boldsymbol{B'}} \end{array}} {\frac{1}{{\left| {\nabla {\boldsymbol{u}_x}} \right|}}} \left( {{\boldsymbol{u}_o} - {\boldsymbol{u}_X}} \right) + {\lambda _e}(O)\left( {{\boldsymbol{u}_O} - \boldsymbol{u}_O^0} \right) = 0$ (8)

 ${\mathit{\boldsymbol{u}}_O} = \frac{{\sum\limits_{\begin{array}{*{20}{c}} {X \in \mathit{\boldsymbol{B}}}\\ {x \in \mathit{\boldsymbol{B'}}} \end{array}} {\frac{{{\mathit{\boldsymbol{u}}_X}}}{{\sqrt {\left| {\nabla {\mathit{\boldsymbol{u}}_x}} \right|} }} + {\lambda _e}(O)\mathit{\boldsymbol{u}}_O^0} }}{{\sum\limits_{\begin{array}{*{20}{c}} {X \in \mathit{\boldsymbol{B}}}\\ {x \in \mathit{\boldsymbol{B'}}} \end{array}} {\frac{1}{{\sqrt {\left| {\nabla {\mathit{\boldsymbol{u}}_x}} \right|} }} + {\lambda _e}(O)} }}$ (9)

# 2.1 分数阶TV模型

 ${J_\lambda }(\mathit{\boldsymbol{u}}) = \int\limits_{\mathit{\boldsymbol{E}} \cup \mathit{\boldsymbol{D}}} {\left| {{\nabla ^\alpha }\mathit{\boldsymbol{u}}} \right|{\rm{d}}x{\rm{d}}y} + \frac{\lambda }{2}\int_\mathit{\boldsymbol{E}} {{{\left| {\mathit{\boldsymbol{u}} - {\mathit{\boldsymbol{u}}^0}} \right|}^2}} {\rm{d}}x{\rm{d}}y$ (10)

 $-\operatorname{div}\left(\frac{\nabla^{\alpha} \boldsymbol{u}}{\left|\nabla^{\alpha} \boldsymbol{u}\right|}\right)+\lambda_{e}\left(\boldsymbol{u}-\boldsymbol{u}^{0}\right)=0$ (11)

 ${\mathit{\boldsymbol{u}}_O} = \frac{{\sum\limits_{\begin{array}{*{20}{c}} {X \in \mathit{\boldsymbol{B}}}\\ {x \in \mathit{\boldsymbol{B'}}} \end{array}} {\frac{{{\mathit{\boldsymbol{u}}_X}}}{{\sqrt {\left| {{\nabla ^\alpha }{\mathit{\boldsymbol{u}}_x}} \right|} }} + {\lambda _e}(O)\mathit{\boldsymbol{u}}_O^0} }}{{\sum\limits_{\begin{array}{*{20}{c}} {X \in \mathit{\boldsymbol{B}}}\\ {x \in \mathit{\boldsymbol{B'}}} \end{array}} {\frac{1}{{\sqrt {\left| {{\nabla ^a}{\mathit{\boldsymbol{u}}_x}} \right|} }} + {\lambda _e}(O)} }}$ (12)

# 2.2 改进的分数阶TV模型

 $-\operatorname{div}\left(\frac{\nabla^{\alpha} \boldsymbol{u}}{\left|\nabla^{\alpha} \boldsymbol{u}\right|+p}\right)+\lambda_{e}\left(\boldsymbol{u}-\boldsymbol{u}^{0}\right)=0$ (13)

 ${\mathit{\boldsymbol{u}}_O} = \frac{{\sum\limits_{\begin{array}{*{20}{c}} {X \in \mathit{\boldsymbol{B}}}\\ {x \in \mathit{\boldsymbol{B'}}} \end{array}} {\frac{{{\mathit{\boldsymbol{u}}_X}}}{{\sqrt {\left| {{\nabla ^\alpha }{\mathit{\boldsymbol{u}}_x}} \right| + {p^2}} }} + {\lambda _e}(O)\mathit{\boldsymbol{u}}_O^0} }}{{\sum\limits_{\begin{array}{*{20}{c}} {X \in \mathit{\boldsymbol{B}}}\\ {x \in \mathit{\boldsymbol{B'}}} \end{array}} {\frac{1}{{\sqrt {\left| {{\nabla ^a}{\mathit{\boldsymbol{u}}_x}} \right| + {p^2}} }} + {\lambda _e}(O)} }}$ (14)

# 2.4 实验参数的设置

 $\begin{array}{*{20}{c}} {{D^\alpha }f(x)\mathop \Leftrightarrow \limits^{{\rm{FT}}} {{(DF)}^\alpha }(u) = {{(vu)}^\alpha }F(u) = {\hat h ^\alpha }(u)F(u)}\\ {{{\hat h}^\alpha }(u) = {{\hat b}^\alpha }(u){{\rm{e}}^{v{\theta ^\alpha }(u)}}} \end{array}$ (17)

# 2.5 算法步骤

1) 根据$\mathit{\boldsymbol{u}} - {\mathit{\boldsymbol{u}}^0}$计算差值图像，确定图像的待修复区域；

2) 根据

 $\mathit{\boldsymbol{\rho }} = \mathop {\arg \min }\limits_{\left( {{c_x},{c_y}} \right) \in \mathit{\boldsymbol{C}}} \frac{1}{{|\mathit{\boldsymbol{R}}|}}\sum\limits_{\begin{array}{*{20}{c}} {(u,v) \in \mathit{\boldsymbol{R}}}\\ {\left( {{u^\prime },{v^\prime }} \right) \in \mathit{\boldsymbol{S}}} \end{array}} {\left| {I(u,v) - I\left( {{u^\prime },{v^\prime }} \right)} \right|}$

3) 为了增加模型的稳定性，根据式(7)，在能量泛函的梯度下降方程引入极小值$p$，如

 $- {\mathop{\rm div}\nolimits} \left( {\frac{{{\nabla ^\alpha }\mathit{\boldsymbol{u}}}}{{\sqrt {{{\left| {{\nabla ^\alpha }\mathit{\boldsymbol{u}}} \right|}^2} + {p^2}} }}} \right)$

4) 根据式(13)最终化简得到

 ${\mathit{\boldsymbol{u}}_O} = \frac{{\sum\limits_{\begin{array}{*{20}{c}} {X \in \mathit{\boldsymbol{B}}}\\ {x \in \mathit{\boldsymbol{B'}}} \end{array}} {\frac{{{\mathit{\boldsymbol{u}}_X}}}{{\sqrt {\left| {{\nabla ^\alpha }{\mathit{\boldsymbol{u}}_x}} \right| + {p^2}} }} + {\lambda _e}(O)\mathit{\boldsymbol{u}}_O^0} }}{{\sum\limits_{\begin{array}{*{20}{c}} {X \in \mathit{\boldsymbol{B}}}\\ {x \in \mathit{\boldsymbol{B'}}} \end{array}} {\frac{1}{{\sqrt {\left| {{\nabla ^a}{\mathit{\boldsymbol{u}}_x}} \right| + {p^2}} }} + {\lambda _e}(O)} }}$

5) 根据经验取$p = \frac{p}{5}$进行迭代，经过多次循环迭代得到最佳的$p$值，输出${{\boldsymbol{u}}_{O}}$

# 3 实验结果与分析

 $f_{\text{MSE}}=\frac{1}{m \times n} \sum\limits_{i, j}\left(I_{1}(i, j)-I(i, j)\right)^{2}$ (18)

 ${f_{{\rm{PSWR}}}} = 10\lg \frac{{\mathop {\max }\limits_{1 \le i \le n,1 \le n} {{\left| {{I_1}(i,j)} \right|}^2}}}{{\frac{1}{{m \times n}}\sum\limits_{i = 1}^m {\sum\limits_{j = 1}^n {{{\left[ {{I_1}(i,j) - I(i,j)} \right]}^2}} } }}$ (19)

# 3.1 模板大小和分数阶阶次对修复效果的影响实验

Table 1 Gray-scale mean square error comparison (Barbara image)

 k值 阶次 α=0.1 α=0.3 α=0.5 α=0.7 α=0.9 α=1.1 α=1.3 α=1.5 α=1.7 α=1.9 1 0.445 2 0.446 0 0.446 2 0.445 8 0.447 2 0.447 4 0.447 8 0.447 4 0.447 0 0.448 5 2 0.445 4 0.445 8 0.446 2 0.446 4 0.447 2 0.447 4 0.447 6 0.447 4 0.448 3 0.449 1 3 0.445 4 0.445 8 0.446 0 0.446 6 0.447 2 0.447 4 0.447 6 0.447 6 0.448 9 0.449 1 注：加粗字体表示最优结果。

Table 2 Peak signal to noise ratio comparison (Barbara image)

 /dB k值 阶次 α=0.1 α=0.3 α=0.5 α=0.7 α=0.9 α=1.1 α=1.3 α=1.5 α=1.7 α=1.9 1 51.645 51.637 51.635 51.639 51.626 51.624 51.620 51.624 51.628 51.614 2 51.643 51.639 51.635 51.633 51.626 51.624 51.622 51.624 51.616 51.608 3 51.643 51.639 51.637 51.631 51.626 51.624 51.622 51.622 51.610 51.608 注：加粗字体表示最优结果。

Table 3 Gray-scale mean square error comperison (Lena image)

 k值 阶次 α=0.1 α=0.3 α=0.5 α=0.7 α=0.9 α=1.1 α=1.3 α=1.5 α=1.7 α=1.9 1 0.044 5 0.044 5 0.044 1 0.042 0 0.037 2 0.037 2 0.038 8 0.039 0 0.037 6 0.037 8 2 0.044 5 0.044 5 0.043 9 0.040 4 0.036 7 0.037 2 0.037 4 0.037 4 0.036 9 0.037 6 3 0.044 5 0.044 5 0.043 7 0.040 2 0.036 9 0.036 9 0.037 2 0.036 9 0.036 9 0.037 6 注：加粗字体表示最优结果。

Table 4 Peak signal to noise ratio comparison (Lena image)

 /dB k值 阶次 α=0.1 α=0.3 α=0.5 α=0.7 α=0.9 α=1.1 α=1.3 α=1.5 α=1.7 α=1.9 1 61.647 61.647 61.687 61.893 62.431 62.431 62.244 62.222 62.384 62.360 2 61.647 61.647 61.708 62.065 62.479 62.431 62.407 62.407 62.455 62.384 3 61.647 61.647 61.728 62.087 62.455 62.455 62.431 62.455 62.455 62.384 注：加粗字体表示最优结果。

# 3.2 修复性能验证

Table 5 Gray-scale mean square error comparison

 图像 修复前 TV(文献[9]) 文献[2]算法 文献[8]算法 本文算法 Barbara(α=0.1, k=1) 4.008 2 0.867 1 0.591 1 0.716 9 0.445 2 Lena(α=0.9, k=2) 2.601 6 0.107 6 0.108 4 0.114 7 0.036 7 岩石(α=0.1, k=1) 8.635 0 0.890 0 1.366 2 0.711 4 0.578 5 注：加粗字体表示每种图片修复的最优结果。

Table 6 Peak signal to noise ratio comparison

 /dB 图像 修复前 TV(文献[9]) 文献[2]算法 文献[8]算法 本文算法 Barbara(α=0.1, k=1) 42.101 48.750 50.414 49.576 51.645 Lena(α=0.9, k=2) 43.979 57.814 57.781 57.535 62.479 岩石(α=0.1, k=1) 38.768 48.637 46.776 49.610 50.508 注：加粗字体表示每种图片修复的最优结果。

# 参考文献

• [1] Bertalmio M, Sapiro G, Caselles V, et al. Image inpainting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. New York, USA: ACM, 2000: 417-424.[DOI:10.1145/344779.344972]
• [2] Criminisi A, Perez P, Toyama K. Object removal by exemplar-based inpainting[C]//Proceedings of 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Madison, WI, USA: IEEE, 2003: 721-728.[DOI:10.1109/CVPR.2003.1211538]
• [3] Zhang Y. When is missing data recoverable?[R]. Houston, TX: Rice University, 2006.
• [4] Zhang G M, Sun X X, Liu J X. Fractional total variation denoising model based on adaptive projection algorithm[J]. Pattern Recognition and Artificial Intelligence, 2016, 29(11): 1009–1018. [张桂梅, 孙晓旭, 刘建新. 基于自适应投影算法的分数阶全变分去噪模型[J]. 模式识别与人工智能, 2016, 29(11): 1009–1018. ] [DOI:10.16451/j.cnki.issn1003-6059.201611006]
• [5] Yang Z Z, Zhou J L, Yan X Y, et al. Image enhancement based on fractional differentials[J]. Journal of Computer-Aided Design & Computer Graphics, 2008, 20(3): 343–348. [杨柱中, 周激流, 晏祥玉, 等. 基于分数阶微分的图像增强[J]. 计算机辅助设计与图形学学报, 2008, 20(3): 343–348. ]
• [6] Zhang G M, Sun X X, Liu J X, et al. Research on TV-L1 optical flow model for image registration based on fractional-order differentiation[J]. Acta Automatica Sinica, 2017, 43(12): 2213–2224. [张桂梅, 孙晓旭, 刘建新, 等. 基于分数阶微分的TV-L1光流模型的图像配准方法研究[J]. 自动化学报, 2017, 43(12): 2213–2224. ] [DOI:10.16383/j.aas.2017.c160367]
• [7] Zhang G M, Sun X X, Chen B B, et al. Edge detection algorithm combining fractional order derivative and Canny operator[J]. Journal of Image and Graphics, 2016, 21(8): 1028–1038. [张桂梅, 孙晓旭, 陈彬彬, 等. 结合分数阶微分和Canny算子的边缘检测[J]. 中国图象图形学报, 2016, 21(8): 1028–1038. ] [DOI:10.11834/jig.20160807]
• [8] Zhang Y, Pu Y F, Hu J R, et al. A class of fractional-order variational image inpainting models[J]. Applied Mathematics & Information Sciences, 2012, 6(2): 299–306.
• [9] Chan T F, Shen J H. Mathematical models for local nontexture inpaintings[J]. SIAM Journal on Applied Mathematics, 2002, 62(3): 1019–1043. [DOI:10.1137/S0036139900368844]
• [10] Telea A. An image inpainting technique based on the fast marching method[J]. Journal of Graphics Tools, 2004, 9(1): 23–34. [DOI:10.1080/10867651.2004.10487596]
• [11] Li K Y, Sun Y G. Fast image inpainting algorithm introducing continuous strength and confidence factor[J]. Journal of Image and Graphics, 2012, 17(4): 465–470. [李开宇, 孙玉刚. 引入连续性强度和置信度因子的快速图像修复[J]. 中国图象图形学报, 2012, 17(4): 465–470. ] [DOI:10.11834/jig.20120403]
• [12] Cafagna D. Fractional calculus:a mathematical tool from the past for present engineers[J]. IEEE Industrial Electronics Magazine, 2007, 1(2): 35–40. [DOI:10.1109/MIE.2007.901479]
• [13] Zhu Y G, Zhang G M. Fractional total variation denoising algorithm based on an adaptive residual image[J]. Journal of Image and Graphics, 2017, 22(12): 1677–1689. [祝严刚, 张桂梅. 自适应残差图像的分数阶全变分去噪算法[J]. 中国图象图形学报, 2017, 22(12): 1677–1689. ] [DOI:10.11834/jig.170198]
• [14] Shao X X, Guo S X, Wang L. Image mosaic algorithm based on extended phase correlation of edge[J]. Journal of Jilin University, 2010, 28(01).
• [15] Zhang C Y, Zhao Y, Chen H X. Registration of depth and video data based on edge detection[J]. Journal of Jilin University(Information Science Edition), 2011, 58(3): 587–593. [DOI:10.1007/s10971-011-2431-x]