发布时间: 2018-10-30 摘要点击次数: 全文下载次数: DOI: 10.11834/jig.180059 2018 | Volume 23 | Number 11 图像处理和编码

1. 广东第二师范学院计算机科学系, 广州 510303;
2. 中山大学电子与通信工程学院, 广州 510006
 收稿日期: 2018-03-07; 修回日期: 2018-06-15 基金项目: 国家自然科学基金项目（61473322，61772140）；广东省自然科学基金-博士启动项目（2016A030310335） 第一作者简介: 朱雄泳, 1976年生, 男, 讲师, 博士, 主要研究方向为数字图像处理、视频信号处理、机器视觉。E-mail:zhuxiongyong@gdei.edu.cn;陆许明, 男, 博士, 讲师, 主要研究方向为数字图像处理、视频信号处理、无线通信、集成电路设计。E-mail:luxuming@gdei.edu.cn;李智文, 男, 硕士, 主要研究方向为数字图像处理、数据挖掘、机器学习。E-mail:imagelee@foxmail.com;吴炆芳, 女, 硕士研究生, 主要研究方向为数字图像处理。E-mail:Wuwf_828@163.com;陈强, 男, 教授, 主要研究方向为数据系统、基于内容的视觉信息检索与理解。E-mail:cq_c@gdei.edu.cn. 中图法分类号: TP301.6 文献标识码: A 文章编号: 1006-8961(2018)11-1652-14

# 关键词

High dynamic range image fusion with low rank matrix recovery
Zhu Xiongyong1,2, Lu Xuming1, Li Zhiwen2, Wu Wenfang2, Tan Hongzhou2, Chen Qiang1
1. Department of Computer Science, Guangdong University of Education, Guangzhou 510303, China;
2. School of Electronic and Information Engineering, Sun Yat-sen University, Guangzhou 510006, China
Supported by: National Natural Science Foundation of China (61473322, 61772140); The PhD Start-up Fund of Natural Science Foundation of Guangdong Province, China (2016A030310335)

# Abstract

Objective Most traditional methods used to merge sequential multi-exposure low dynamic range (LDR) images into a high dynamic range (HDR) image are sensitive to certain problems, such as noise and object motion, and must address large-scale data, which hinder the application and further development of HDR image acquisition technology. Low-rank matrix recovery can extract an aligned low-rank image with linear correlation from a sparse noise-corrupted data matrix. A new method that exploits the abovementioned feature based on the low-rank matrix recovery is proposed to merge sequential multi-exposure LDR images into an HDR image and improve the anti-noise and de-artifact performances in capturing HDR images. Method First, the sequential multi-exposure LDR images are inputted and mapped to the linear luminance space by a calibrated camera response function (CRF). Second, a partial sum of singular values (PSSV) is used as an optimization objective function to build a low-rank matrix mathematical model for HDR image fusion method, which is used to merge the captured sequential multi-exposure LDR images. With the help of the proposed method, the data matrix is decomposed into low-rank and sparse matrices through the exact augmented Lagrange multiplier method, where the PSSV is the objective function. This algorithm is optimized given the motivation for an alternating direction multiplier method. An adaptive penalty factor is set to address different singular values. If a singular value tends to 0, then the algorithm will update the low-rank and sparse matrices with a new partial singular value thresholding (PSVT); otherwise, the algorithm will update the low-rank and sparse matrices with the classical PSVT. Moreover, the augmented Lagrange multiplier and penalty factor are updated simultaneously. The algorithm will terminate when the optimal solution concentrates within the space of the maximum singular value as much as possible after a finite number of iteration steps. Thus, a low-rank matrix with the light information of an entire scene, where the noises and artifacts are eliminated, is obtained. This obtained low-rank matrix is also the final merged HDR image from the captured sequential multi-exposure LDR images. Result The convergence and anti-noise performance are first evaluated. The proposed method and two other comparison methods are applied to the randomly generated data matrices with a size of 10 000 ×50 pixels and rank from 1 to 4. Simultaneously, a sparse noise is added to each data matrix with a ratio from 0.1 to 0.4. The comparison methods refer to robust principal component analysis (RPCA) and the PSSV. Simulation results indicate that the proposed method has better convergence and anti-noise performance than the two other comparison methods. The experimental results of various matrices with different ranks and sparse noise ratios show that the proposed method achieves low normalized mean square and solution errors. Furthermore, the proposed algorithm guarantees that the rank of the result is sufficiently lower than the original matrix. Thus, the singular value of the main information will not be considerably attenuated. This finding indicates that the new method can obtain low-rank results even when the reconstruction error is low. The performance of HDR image fusion is evaluated by analyzing the values of peak signal-to-noise ratio (PSNR) and structural similarity index metric based on perceptually uniform mapping. The experiments run with the classical sequential multi-exposure LDR images, such as memorial church and arch, to acquire the HDR images. The experimental results show that the expectation is achieved. The proposed method can eliminate the artifacts in dynamic scenes with sparse noise and improve the quality of the fused HDR images compared with the recovering high dynamic range radiance maps from photographs (RHDRRMP), RPCA, and PSSV algorithms. The RHDRRMP method cannot suppress the sparse noise and artifacts and produces poor brightness and contrast. The RPCA method cannot suppress artifacts well, and missing details and even inaccurate results have emerged. The PSSV method can obtain better results but fewer details than the proposed method. The index metrics of the PSNR and SSIM of the results obtained through the proposed method from the objective indicators are higher than those of the comparison algorithms. For the memorial church sequence without noise, the PSNR and SSIM of the RPCA method are 28.117 dB and 0.935, respectively; those of the PSSV method are 30.557 dB and 0.959, correspondingly; and those of our method are 32.550 dB and 0.968, respectively. The PSNR and SSIM of the RPCA method are 28.115 dB and 0.935, correspondingly; those of the PSSV method are 30.579 dB and 0.959, respectively; and those of the proposed method are 32.562 dB and 0.967, correspondingly. The proposed algorithm can recover the low-rank matrix to obtain the HDR image, even with few images in the multi-exposure image sequence. In this situation, the RPCA method cannot obtain the optimal solution to the low-rank matrix. The PSSV method only ensures that the variance of the singular value vectors in the data, rather than the low-rank data, is not the largest and cannot guarantee that the low-rank data have the maximum variance on the singular value vector. Overall, the results show that the proposed algorithm has better robustness than the traditional fusion methods. Conclusion In this study, a new method based on low-rank matrix recovery optimization theory is proposed. The proposed method can merge sequential multi-exposure LDR images into an HDR image. With the help of the proposed method, the HDR image can be obtained with a low reconstruction error in the case of few datasets, and the interference of the noise and artifacts can be removed in a dynamic scene. Thus, the proposed method has better robustness than the traditional experimental methods. The demand for high-quality images can be satisfied by improving HDR images. However, the proposed method depends on the CRF, that is, an accurate CRF indicates an improved quality of the result of image fusion. The proposed method also requires the aligned sequential multi-exposure LDR images to further eliminate the serious problems of image displacement or high-speed moving objects in a scene. Otherwise, the ghost and blur phenomena will affect the fused HDR image.

# Key words

image fusion; high dynamic range image; low-rank matrix recovery; de-ghosting; Lagrange multiplier

# 0 引言

Goshtasby[4]对多曝光图像分块并基于信息熵融合图像最优的曝光图像块，利用高斯函数将最优块进行融合，以消除块与块之间存在的边界不连续现象。Mertens等人[5]对输入图像序列进行拉普拉斯金字塔分解并根据对比度、饱和度与曝光良好率求得融合图像的权值，将图像拉普拉斯金字塔与权重高斯金字塔进行融合，由融合后的金字塔重构HDR图像。Shen等人[6]提出一种基于曝光质量测量以及最小可察觉失真获取多曝光图像融合权重的方法，同时利用快速的拉普拉斯金字塔加速分解图像纹理层与细节层，分别对各层给予对应权重进行融合，融合后的图像具有保持细节与色彩的效果。Bruce[7]提出了一种基于局部熵获取像素融合权重的方法，并把图像转换到对数域进行处理，通过设置权值的大小突出局部细节的对比度。由于直接融合多曝光图像像素的方法除了能压缩动态范围，还能作为HDR图像重现方法。付争方等人[8]将多曝光图像直接融合结合Sigmoid函数拟合，直接从获得的LDR图像中提取每个像素位置的亮度序列，利用最小二乘法来拟合适应视觉的S曲线，从而建立起亮度信息的数学模型，给出最佳成像亮度的判决方式，快速有效地合成HDR图像。

# 1 多曝光HDR图像融合的低秩模型

 $\begin{array}{*{20}{c}} {\left| {{{\left\| \mathit{\boldsymbol{L}} \right\|}_ * } - {{\left\| {{P_N}\left( \mathit{\boldsymbol{L}} \right)} \right\|}_ * }} \right| = }\\ {\left| {\sum\limits_{i = 1}^{rank\left( L \right)} {{\sigma _i}\left( \mathit{\boldsymbol{L}} \right)} - \sum\limits_{i = 1}^N {{\sigma _i}\left( \mathit{\boldsymbol{L}} \right)} } \right| = }\\ {\sum\limits_{i = N + 1}^{rank\left( L \right)} {{\sigma _i}\left( \mathit{\boldsymbol{L}} \right)} = {{\left\| \mathit{\boldsymbol{L}} \right\|}_{p = N}}} \end{array}$ (8)

 $\begin{array}{*{20}{c}} {\mathop {\min }\limits_{\mathit{\boldsymbol{L}},\mathit{\boldsymbol{E}}} {{\left\| \mathit{\boldsymbol{L}} \right\|}_{p = 1}} + \lambda {{\left\| \mathit{\boldsymbol{E}} \right\|}_1}}\\ {{\rm{s}}.\;{\rm{t}}.\;\;\mathit{\boldsymbol{D}} = \mathit{\boldsymbol{L}} + \mathit{\boldsymbol{E}}} \end{array}$ (9)

# 2.1 EALM求解低秩矩阵

 $\min f\left( \mathit{\boldsymbol{X}} \right){\rm{s}}.\;{\rm{t}}.\;\;h\left( \mathit{\boldsymbol{X}} \right) = 0$ (10)

 $\begin{array}{*{20}{c}} {L\left( {\mathit{\boldsymbol{X}},\mathit{\boldsymbol{Y}},\mu } \right) = f\left( \mathit{\boldsymbol{X}} \right) + < \mathit{\boldsymbol{Y}},h\left( \mathit{\boldsymbol{X}} \right) > + }\\ {\frac{\mu }{2}\left\| {h\left( \mathit{\boldsymbol{X}} \right)} \right\|_{\rm{F}}^2} \end{array}$ (11)

 $\begin{array}{l} {\mathit{\boldsymbol{X}}_{k + 1}} = \arg \mathop {\min }\limits_\mathit{\boldsymbol{X}} L\left( {\mathit{\boldsymbol{X}},{\mathit{\boldsymbol{Y}}_k},{\mu _k}} \right)\\ {\mathit{\boldsymbol{Y}}_{k + 1}} = {\mathit{\boldsymbol{Y}}_k} + {\mu _k}h\left( {{\mathit{\boldsymbol{X}}_{k + 1}}} \right)\\ {\mu _{k + 1}} = \rho {\mu _k} \end{array}$ (12)

 $\begin{array}{l} \mathit{\boldsymbol{X}} = \left( {\mathit{\boldsymbol{L}},\mathit{\boldsymbol{E}}} \right)\\ f\left( \mathit{\boldsymbol{X}} \right) = {\left\| \mathit{\boldsymbol{L}} \right\|_{p = N}} + \lambda {\left\| \mathit{\boldsymbol{E}} \right\|_1}\\ h\left( \mathit{\boldsymbol{X}} \right) = \mathit{\boldsymbol{D}} - \mathit{\boldsymbol{L}} - \mathit{\boldsymbol{E}} \end{array}$ (13)

 $\begin{array}{*{20}{c}} {L\left( {\mathit{\boldsymbol{L}},\mathit{\boldsymbol{E}},\mathit{\boldsymbol{Y}},\mu } \right) = {{\left\| \mathit{\boldsymbol{L}} \right\|}_{p = N}} + \lambda {{\left\| \mathit{\boldsymbol{E}} \right\|}_1} + }\\ { < \mathit{\boldsymbol{Y}},\mathit{\boldsymbol{D}} - \mathit{\boldsymbol{L}} - \mathit{\boldsymbol{E}} > + }\\ {\frac{\mu }{2}\left\| {\mathit{\boldsymbol{D}} - \mathit{\boldsymbol{L}} - \mathit{\boldsymbol{E}}} \right\|_{\rm{F}}^2} \end{array}$ (14)

 ${{S'}_\tau } = \left\{ {\begin{array}{*{20}{c}} \begin{array}{l} \sigma - \tau {S_{a,b}}\left( \sigma \right)\\ 0 \end{array}&\begin{array}{l} \sigma > \tau + \tau {S_{a,b}}\left( \sigma \right)\\ \sigma \le \tau + \tau {S_{a,b}}\left( \sigma \right) \end{array} \end{array}} \right.$ (20)

 $\begin{array}{*{20}{c}} {{{\mathit{\boldsymbol{F'}}}_{N,\mu _k^{ - 1}}}\left[ \mathit{\boldsymbol{Y}} \right] = \mathit{\boldsymbol{U}}\left( {{\mathit{\boldsymbol{S}}_{{Y_1}}} + {{\mathit{\boldsymbol{H'}}}_\tau }\left[ {{\mathit{\boldsymbol{S}}_{{Y_2}}}} \right]} \right){\mathit{\boldsymbol{V}}^{\rm{T}}} = }\\ {\arg \mathop {\min }\limits_\mathit{\boldsymbol{X}} \frac{1}{2}\left\| {\mathit{\boldsymbol{X}} - \mathit{\boldsymbol{Y}}} \right\|_{\rm{F}}^2 + \tau {{\left\| \mathit{\boldsymbol{X}} \right\|}_{p = N}}} \end{array}$ (21)

${\mathit{\boldsymbol{S}}_{{Y_2}}} = {\rm{diag}}\left( {{\sigma _1}, \cdots, {\sigma _{N + 1}}, 0, \cdots, {\sigma _l}} \right)$

 $\begin{array}{*{20}{c}} {{\mathit{\boldsymbol{L}}_{k + 1}} = \arg \mathop {\min }\limits_\mathit{\boldsymbol{L}} L\left( {\mathit{\boldsymbol{L}},{\mathit{\boldsymbol{E}}_k},{\mathit{\boldsymbol{Y}}_k},{\mu _k}} \right)}\\ { = \arg \mathop {\min }\limits_\mathit{\boldsymbol{L}} \mu _k^{ - 1}{{\left\| \mathit{\boldsymbol{L}} \right\|}_{p = N}} + }\\ {\frac{1}{2}\left\| {\mathit{\boldsymbol{L}} - \left( {\mathit{\boldsymbol{D}} - {\mathit{\boldsymbol{E}}_k} + \mu _k^{ - 1}{\mathit{\boldsymbol{Y}}_k}} \right)} \right\|_{\rm{F}}^2 = }\\ {{{\mathit{\boldsymbol{F'}}}_{N,\mu _k^{ - 1}}}\left[ {\mathit{\boldsymbol{D}} - {\mathit{\boldsymbol{E}}_k} + \mu _k^{ - 1}{\mathit{\boldsymbol{Y}}_k}} \right]} \end{array}$ (22)

Table 1 The singular values of the restored low-rank matrices

 矩阵尺寸/像素 噪声比例 ${\sigma _A}$ 方法 $rank\left( \mathit{\boldsymbol{L}} \right)$ ${\sigma _1}\left( \mathit{\boldsymbol{L}} \right)$ ${\sigma _2}\left( \mathit{\boldsymbol{L}} \right)$ ${\sigma _3}\left( \mathit{\boldsymbol{L}} \right)$ ${\sigma _4}\left( \mathit{\boldsymbol{L}} \right)$ 10 000×5 0.2 403.97 RPCA 3 130.59 6.51 0.01 0.00 PSSV 1 371.80 0.00 0.00 0.00 OURS 1 370.11 0.00 0.00 0.00 0.4 248.72 RPCA 2 65.63 2.95 0.00 0.00 PSSV 3 213.56 2.89 0.04 0.00 OURS 1 216.65 0.00 0.00 0.00 0.6 202.11 RPCA 2 30.27 0.63 0.00 0.00 PSSV 3 162.35 11.61 0.17 0.00 OURS 1 159.91 0.00 0.00 0.00 0.8 112.80 RPCA 2 19.70 0.05 0.00 0.00 PSSV 3 103.61 20.34 3.05 0.00 OURS 1 103.25 0.00 0.00 0.00 10 000×50 0.2 804.61 RPCA 1 787.31 0.00 0.00 0.00 PSSV 1 804.40 0.00 0.00 0.00 OURS 1 804.46 0.00 0.00 0.00 0.4 735.16 RPCA 6 592.87 16.93 4.55 0.83 PSSV 1 727.78 0.00 0.00 0.00 OURS 1 730.48 0.00 0.00 0.00 0.6 741.74 RPCA 7 431.27 25.79 4.16 1.37 PSSV 2 677.85 2.31 0.00 0.00 OURS 1 687.65 0.00 0.00 0.00 0.8 630.26 RPCA 11 258.11 17.97 3.67 1.10 PSSV 9 471.15 25.58 3.34 1.30 OURS 2 490.32 0.93 0.00 0.00

PSSV与本文算法在较少数据的情况下恢复的低秩结果都要比RPCA的结果好，非常适用于多曝光图像融合数据集并不多的情况，而本文提出的算法对比于PSSV能够在较低重构误差的前提下得到更加低秩的结果。

# 3.3 多曝光HDR图像融合的仿真

 $R = \frac{1}{N}\sum\limits_{i = 1}^N {\frac{{{\mathit{\boldsymbol{L}}_i}}}{{\Delta {t_i}}}}$ (23)

# 参考文献

• [1] Debevec P E, Malik J. Recovering high dynamic range radiance maps from photographs[C]//Proceedings of ACM SIGGRAPH 2008 Classes. Los Angeles, California: ACM, 2008: #31.
• [2] Mitsunaga T, Nayar S K. Radiometric self calibration[C]//Proceedings of 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Fort Collins, CO, USA: IEEE, 1999: 374-380.[DOI:10.1109/CVPR.1999.786966]
• [3] Fang H M, Yi B S, Gan L C, et al. A fast calibration method of camera response function for high dynamic range image[J]. Acta Photonica Sinica, 2013, 42(6): 737–741. [方华猛, 易本顺, 甘良才, 等. 高动态范围图像合成中相机响应函数的快速标定[J]. 光子学报, 2013, 42(6): 737–741. ] [DOI:10.3788/gzxb20134206.0737]
• [4] Goshtasby A A. Fusion of multi-exposure images[J]. Image and Vision Computing, 2005, 23(6): 611–618. [DOI:10.1016/j.imavis.2005.02.004]
• [5] Mertens T, Kautz J, Van Reeth F. Exposure fusion:A simple and practical alternative to high dynamic range photography[J]. Computer Graphics Forum, 2009, 28(1): 161–171.
• [6] Shen J B, Zhao Y, Yan S C, et al. Exposure fusion using boosting Laplacian pyramid[J]. IEEE Transactions on Cybernetics, 2014, 44(9): 1579–1590. [DOI:10.1109/TCYB.2013.2290435]
• [7] Bruce N D B. ExpoBlend:information preserving exposure blending based on normalized log-domain entropy[J]. Computers & Graphics, 2014, 39: 12–23. [DOI:10.1016/j.cag.2013.10.001]
• [8] Fu Z F, Zhu H, Xue S, et al. Direct fusion algorithm for multi-exposed images based on sigmoid function fitting[J]. Chinese Journal of Scientific Instrument, 2015, 36(10): 2321–2329. [付争方, 朱虹, 薛杉, 等. 基于Sigmoid函数拟合的多曝光图像直接融合算法[J]. 仪器仪表学报, 2015, 36(10): 2321–2329. ] [DOI:10.3969/j.issn.0254-3087.2015.10.021]
• [9] Peng Y G, Ganesh A, Wright J, et al. RASL:robust alignment by sparse and low-rank decomposition for linearly correlated images[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(11): 2233–2246. [DOI:10.1109/TPAMI.2011.282]
• [10] Shi J R, Zheng X Y, Wei Z T, et al. Survey on algorithms of low-rank matrix recovery[J]. Application Research of Computers, 2013, 30(6): 1601–1605. [史加荣, 郑秀云, 魏宗田, 等. 低秩矩阵恢复算法综述[J]. 计算机应用研究, 2013, 30(6): 1601–1605. ] [DOI:10.3969/j.issn.1001-3695.2013.06.001]
• [11] Donoho D L. Compressed sensing[J]. IEEE Transactions on Information Theory, 2006, 52(4): 1289–1306. [DOI:10.1109/TIT.2006.871582]
• [12] Candès E J, Wakin M B. An introduction to compressive sampling[J]. IEEE Signal Processing Magazine, 2008, 25(2): 21–30. [DOI:10.1109/MSP.2007.914731]
• [13] Wright J, Ma Y, Mairal J, et al. Sparse representation for computer vision and pattern recognition[J]. Proceedings of the IEEE, 2010, 98(6): 1031–1044. [DOI:10.1109/JPROC.2010.2044470]
• [14] Peng Y G, Suo J L, Dai Q H, et al. From compressed Sensing to low-rank matrix recovery:theory and applications[J]. Acta Automatica Sinica, 2013, 39(7): 981–994. [彭义刚, 索津莉, 戴琼海, 等. 从压缩传感到低秩矩阵恢复:理论与应用[J]. 自动化学报, 2013, 39(7): 981–994. ] [DOI:10.3724/SP.J.1004.2013.00981]
• [15] Candès E J, Li X D, Ma Y, et al. Robust principal component analysis?[J]. Journal of the ACM, 2011, 58(3): #11. [DOI:10.1145/1970392.1970395]
• [16] Oh T H, Kim H, Tai Y W, et al. Partial sum minimization of singular values in RPCA for low-level vision[C]//Proceedings of the IEEE International Conference on Computer Vision. Sydney, NSW, Australia: IEEE, 2013: 145-152.[DOI:10.1109/ICCV.2013.25]
• [17] Oh T H, Tai Y W, Bazin J C, et al. Partial sum minimization of singular values in robust PCA:algorithm and applications[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(4): 744–758. [DOI:10.1109/TPAMI.2015.2465956]
• [18] Zhang D B, Hu Y, Ye J P, et al. Matrix completion by truncated nuclear norm regularization[C]//Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, RI, USA: IEEE, 2012: 2192-2199.[DOI:10.1109/CVPR.2012.6247927]
• [19] Wright J, Ganesh A, Rao S, et al. Robust principal component analysis: exact recovery of corrupted low-rank matrices via convex optimization[R]. Illinois: Coordinated Science Laboratory, University of Illinois at Urbana-Champaign, 2009: 2080-2088.
• [20] Lin Z C, Ganesh A, Wright J, et al. Fast convex optimization algorithms for exact recovery of a corrupted low-rank matrix[R]. Illinois: Coordinated Science Laboratory, University of Illinois at Urbana-Champaign, 2009: 61.
• [21] Lin Z C, Chen M M, Ma Y. The augmented Lagrange multiplier method for exact recovery of corrupted low-rank matrices[J]. arXiv preprint arXiv:1009.5055, 2010.
• [22] Bhardwaj A, Raman S. Robust PCA-Based Solution to Image Composition Using Augmented Lagrange Multiplier (ALM)[M]. Berlin, Heidelberg: Springer-Verlag, 2015.
• [23] Ayd ın T O, Mantiuk R, Seidel H P. Extending quality metrics to full luminance range images[C]//Proceedings of Human Vision and Electronic Imaging XⅢ. San Jose, California, United States: SPIE, 2008, 6806: 68060B.[DOI:10.1117/12.765095]
• [24] Gallo O, Gelfandz N, Chen W C, et al. Artifact-free high dynamic range imaging[C]//Proceedings of 2009 IEEE International Conference on Computational Photography. San Francisco, CA, USA: IEEE, 2009: 1-7.[DOI:10.1109/ICCPHOT.2009.5559003]
• [25] Karaduzovic-Hadziabdic K, Telalovic J H, Mantiuk R. Expert comparison of deghosting algorithms for multi-exposure high dynamic range imaging[C]//Proceedings of the 29th Spring Conference on Computer Graphics. Smolenice, Slovakia: ACM, 2013: 21-28.[DOI:10.1145/2508244.2508247]