High dynamic range image fusion with low rank matrix recovery
Zhu Xiongyong,Lu Xuming,Li Zhiwen,Wu Wenfang,Tan Hongzhou,Chen Qiang(Department;of Computer Science, Guangdong University of Education,, Guangzhou, 510303;School of Electronic and Information Engineering, Sun Yat-sen University,, Guangzhou, 510006)
Objective Most traditional methods for merging the sequential multi-exposure low dynamic range (LDR) images into a high dynamic range (HDR) image are sensitive to some problems, such as noise and object motion, and have to deal with large scale data, which hinders the application and also the further development of HDR image acquisition technology. Low-rank matrix recovery can extract the aligned low-rank image with linear correlation from sparse noise corrupted data matrix. Taking advantage of this feature, a new method based on low-rank matrix recovery is proposed to merge sequential multi-exposure LDR images into an HDR image, and improve the anti-noise and de-artifact performance for HDR image acquirement.Method First of all, the input sequential multi-exposure LDR images are mapped to the linear luminance space by a calibrated camera response function. Then the partial sum of singular values is taken as the optimization objective function, to build a low-rank matrix mathematical model for the HDR image fusion method, which is used for merging the captured sequential multi-exposure LDR images. With the help of this proposed model, the data matrix is decomposed into low-rank matrix and sparse matrix respectively by the exact augmented Lagrange multiplier method, where the partial sum of singular values is the objective function. This algorithm is further optimized since the idea of alternating direction multipliers method inspires us: an adaptive penalty factor is set to deal with different singular values: if the singular value tends to 0, the algorithm will update the low-rank matrix and sparse matrix with new partial singular value thresholding (PSVT); otherwise it will update the low-rank matrix and sparse matrix with the classical partial singular value thresholding. Meanwhile, the augmented Lagrange multiplier and the penalty factor are also updated simultaneously. After a finite number of iteration steps, the algorithm will terminate with the optimal solution concentrates within the space of the maximum singular value as more as possible. Hence, a low-rank matrix with the light information of the whole scene is obtained, where the noises and the artifacts are eliminated. This obtained low-rank matrix is also the final merged HDR image from the captured sequential multi-exposure LDR images.Result The convergence and anti-noise performance is first evaluated. The proposed method and other two compared methods are applied to the random generated data matrices with size of 10000×50, and with different ranks from 1 to 4. At the same time, a sparse noise is also added into each data matrix with a ratio from 0.1 to 0.4. The compared methods are the algorithm of the robust principal component analysis (RPCA) and the algorithm of the partial sum of singular values (PSSV). Simulation results demonstrate that the proposed method has better convergence performance and the anti-noise performance. In the results of the experiments on different matrices with different ranks and sparse noise ratios, the proposed method obtains lower normalized mean square error (MSE) and solving error as well. Simultaneously, compared with the original matrix, the proposed algorithm guarantees that the rank of the result is low enough, and the singular value of the main information will not be attenuated enormously. It means that the new method can get lower rank results even when the reconstruction error is low. The performance of HDR image fusion are evaluated by the values of the peak signal to noise ratio (PSNR) and the structural similarity index metric (SSIM) based on the perceptually uniform (PU) mapping. The experiments run with the classical sequential multi-exposure LDR images, such as Memorial Church and Arch, to acquiring the HDR images. The experimental results show that the expectation is achieved. The proposed method can eliminate the artifact in dynamic scenes with spare noise and improve the quality of the fused HDR images, compared with the algorithm of 1 ref., RPCA and PSSV. The method of 1 ref. cannot suppress the sparse noise and the artifact, and has poor brightness and contrast. The RPCA method can’t suppress artifacts well, and missing details, even the wrong results emerged. The PSSV method can obtain a better result but gets fewer details than the proposed method. From the objective indicators, the index metrics of PSNR and SSIM of the results by our method are higher than the rest of the comparison algorithms: for the Memorial Church sequence without noise, the PSNR and SSIM of RPCA method are 28.117 and 0.935. Those of the PSSV method are 30.557 and 0.959, while those of ours are 32.550 and 0.968. After adding uniformly distributed noise to the sequence, the two results of RPCA method are 28.115 and 0.935, respectively. The two results of the PSSV method are 30.579 and 0.959, while these of ours are 32.562 and 0.967. Even if fewer images in the multi-exposure image sequence, the proposed algorithm also can recover the low rank matrix to obtain the HDR image. At this situation, the RPCA method cannot get the optimal solution of the low rank; and the PSSV method only ensures that the variance of the singular value vectors in the data, rather than the low rank data, is not the largest, and cannot guarantee that the low rank data have the maximum variance on the singular value vector. All in all, the results show that the proposed algorithm has better robustness than traditional fusion methods.Conclusion In this paper, a new method based on the low-rank matrix recovery optimization theory is proposed. It can merge the sequential multi-exposure LDR images into an HDR image. With the help of this method, not only can the HDR image be obtained with a lower reconstruction error in the case of a small amount of data sets, but also the interference can be removed from the noise and artifact in a dynamic scene. Hence the proposed method has a better robustness compared with the experimental traditional methods. With the above improvement of HDR images, the demand for high-quality images would be met. However, the prosed method depends on the camera response function (CRF), that is, the more accurate the CRF is, the better quality the image fusion will result in. In addition, the proposed method also need the aligned sequential multi-exposure LDR images to further eliminate the problems of image displacement seriously or the high speed moving objects in a scene. Otherwise, it will cause the ghost and blur phenomenon arise in the fused HDR image.