浏览全部资源
扫码关注微信
Received：18 March 2019，
Revised：2019-6-21，
Accepted：28 June 2019，
Published：16 December 2019
移动端阅览
目的
2
颜色恒常性通常指人类在任意光源条件下正确感知物体颜色的自适应能力，是实现识别、分割、3维视觉等高层任务的重要前提。对图像进行光源颜色估计是实现颜色恒常性计算的主要途径之一，现有光源颜色估计方法往往因局部场景的歧义颜色导致估计误差较大。为此，提出一种基于深度残差学习的光源颜色估计方法。
方法
2
将输入图像均匀分块，根据局部图像块的光源颜色估计整幅图像的全局光源颜色。算法包括光源颜色估计和图像块选择两个残差网络：光源颜色估计网络通过较深的网络层次和残差结构提高光源颜色估计的准确性；图像块选择网络按照光源颜色估计误差对图像块进行分类，根据分类结果去除图像中误差较大的图像块，进一步提高全局光源颜色估计精度。此外，对输入图像进行对数色度预处理，可以降低图像亮度对光源颜色估计的影响，提高计算效率。
结果
2
在NUS-8和重处理的ColorChecker数据集上的实验结果表明，本文方法的估计精度和稳健性较好；此外，在相同条件下，对数色度图像比原始图像的估计误差低10% 15%，图像块选择网络能够进一步使光源颜色估计网络的误差降低约5%。
结论
2
在两组单光源数据集上的实验表明，本文方法的总体设计合理有效，算法精度和稳健性好，可应用于需要进行色彩校正的图像处理和计算机视觉等领域。
Objective
2
Color constancy refers to the human ability that allows the brain to recognize an object as having a consistent color under varying illuminants. Color constancy has become an important prerequisite of high-level tasks
such as recognition
segmentation
and 3D vision. In the computer vision community
the goal of computational color constancy is to remove illuminant color casts and obtain accurate color representations for images. Therefore
illuminant estimation is an important means to achieve computational color constancy
which is a difficult and underdetermined problem because the observed image color is influenced by unknown factors
such as scene illuminants and object reflections. Illuminant estimation methods can be categorized into two classes:statistics-based (or static) and learning-based methods. Statistics-based methods estimate the illuminant based on the statistical properties (e.g.
reflectance distributions) of the image. Learning-based methods learn a model from training images then estimate the illuminant using the model. Convolutional neural networks (CNNs) are very powerful methods of estimating illuminants
and many competitive results have been obtained with CNN-based methods. We propose a CNN-based illuminant estimation algorithm in this study. We use deep residual learning to improve network accuracy and a patch-selecting network to overcome the color ambiguity issue of local patches.
Method
2
We uniformly sample local patches from the image
estimate the local illuminant of each patch individually
and generate a global illuminant estimation of the entire image by combining the local illuminants. We use a 64×64 patch size in the patch sampling to guarantee the estimation accuracy of the local illuminant and provide sufficient training inputs without data augmentation. The proposed approach includes two residual networks
namely
illuminant estimation net (IEN) and patch selection net (PSN). IEN estimates the local illuminant of image patches. To improve the estimation accuracy of IEN
we increase the feature extraction hierarchy by adding network depth and use the residual structure to ensure gradient propagation and facilitate the training of the deep network. IEN is based on the residual structure
which consists of many stacked 3×3 and 1×1 convolutional layers
batch normalization layers
and rectified linear unit layers. The remaining part is composed of one global average pooling layer and one full connection layer. We use Euclidean loss and stochastic gradient descent (SGD) to optimize IEN. PSN shares a similar architecture with IEN
except that PSN has an additional Softmax layer that serves as the classifier at the end of the network. PSN is proposed to classify image patches according to their illuminant estimation errors. We use cross entropy loss and SGD to optimize PSN. According to the results of PSN
patches with a large estimation error are removed from the entire image
thus improving the performance of global illuminant estimation. Additionally
we preprocess the input image by using the log-chrominance algorithm
which converts a three-channel RGB image into a two-channel log-chrominance image; this reduces the influence of image luminance and improves the computational efficiency by decreasing the amount of data by 1/3.
Result
2
We implement the proposed IEN and PSN on the Caffe library. To evaluate the performance of our approach
we use two standard single-illuminant datasets
namely
the NUS-8 dataset and the reprocessed ColorChecker dataset. Both datasets include indoor and outdoor images
and a Macbeth ColorChecker is placed in each image to calculate the ground truth illuminant. The NUS-8 dataset contains 1 736 images captured from 8 different cameras
and the reprocessed ColorChecker dataset consists of 568 images from 2 cameras. Following the configurations of previous studies
we report the following metrics:the mean
the median
the tri-mean
and the mean of the lowest 25% and the highest 25% of angular errors. We also report the additional metric of the 95th percentile for the reprocessed ColorChecker dataset. We divide the NUS-8 dataset into eight subsets
apply three-fold cross-validation on the eight subsets individually
and report the geometric mean of the proposed metrics for all eight subsets. We directly apply three-fold cross-validation on the reprocessed ColorChecker dataset. Experimental results show that the proposed approach is competitive with state-of-the-art methods. For the NUS-8 dataset
the proposed IEN achieves the best results among all compared methods
and the proposed PSN can further increase the precision of the IEN results. For the reprocessed ColorChecker dataset
our results are comparable with those of other advanced methods. In addition
we conduct ablation studies to evaluate the model components of the proposed approach. We compare the proposed IEN with several shallower CNNs. Experimental results show that deep residual learning is effective in improving illuminant estimation accuracy. Moreover
compared with the estimated illuminant on the original image
log-chrominance preprocessing can reduce the illuminant estimation error by 10% to 15%. The proposed PSN can further decrease the global illuminant estimation error by 5% compared with the method that uses IEN alone. Finally
we evaluate the time cost of our method on a PC with an Intel i5 2.7 GHz CPU
16GB of memory
and an NVIDIA GeForce GTX 1080Ti GPU. Our code takes less than 1.4 s to estimate a 2 K image
which has a typical resolution of 2 048×1 080 pixels.
Conclusion
2
Experiments on the two single-illuminant datasets show that the proposed approach
which includes log-chrominance preprocessing
deep residual learning-based network structure
and patch selection for global illuminant estimation
is reasonable and effective. The proposed approach has high precision and robustness and can be widely used in image processing and computer vision systems that require color calibrations.
Barron J T. 2015. Convolutional color constancy//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE, 379-387[ DOI: 10.1109/ICCV.2015.51 http://dx.doi.org/10.1109/ICCV.2015.51 ]
Bi D Y, Ku T, Zha Y F, Zhang L C and Yang Y. 2016. Scale-adaptive object tracking based on color names histogram. Journal of Electronics&Information Technology, 38(5):1099-1106
毕笃彦, 库涛, 查宇飞, 张立朝, 杨源. 2016.基于颜色属性直方图的尺度目标跟踪算法研究.电子与信息学报, 38(5):1099-1106[DOI:10.11999/JEIT150921]
Bianco S, Cusano C and Schettini R. 2015. Color constancy using CNNs//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Boston, MA, USA: IEEE, 81-89[ DOI: 10.1109/CVPRW.2015.7301275 http://dx.doi.org/10.1109/CVPRW.2015.7301275 ]
Buchsbaum G. 1980. A spatial processor model for object colour perception. Journal of the Franklin Institute, 310(1):1-26.
Chakrabarti A, Hirakawa K and Zickler T. 2012. Color constancy with spatio-spectral statistics. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(8):1509-1519[DOI:10.1109/TPAMI.2011.252]
Cheng D L, Prasad D K and Brown M S. 2014. Illuminant estimation for color constancy:why spatial-domain methods work and the role of thecolor distribution. Journal of the Optical Society of America A, 31(5):1049-1058[DOI:10.1364/JOSAA.31.001049]
Cheng D L, Price B, Cohen S and Brown M. 2015. Effective learning-based illuminant estimation using simple features//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA: IEEE, 1000-1008[ DOI: 10.1109/CVPR.2015.7298702 http://dx.doi.org/10.1109/CVPR.2015.7298702 ]
Cheng Y, Jiao L B, Tong Y, Li Z, Hu Y and Cao X. 2017. Directional illumination estimation sets and multilevel matching metric for illumination-robust face recognition. IEEE Access, 5:25835-25845[DOI:10.1109/ACCESS.2017.2766128]
Cui S, Zhang J and Gao J. 2018. Illumination estimation based on exemplar learning in logarithm domain. Acta Optica Sinica, 38(2):#0233001
崔帅, 张骏, 高隽. 2018.对数域中基于实例学习的光照估计.光学学报, 38(2):#0233001[DOI:10.3788/aos201838.0233001]
Deng J, Dong W, Socher R, Li L J and Li F F. 2009. ImageNet: a large-scale hierarchical image database//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL, USA: IEEE, 248-255[ DOI: 10.1109/CVPR.2009.5206848 http://dx.doi.org/10.1109/CVPR.2009.5206848 ]
Duan Z G, Li Y, Wang E D, Tian, J D and Tang Y D. 2016. Road and navigation line detection algorithm from shadow image based on the illumination invariant image. Acta Optica Sinica, 36(12):#1215004
段志刚, 李勇, 王恩德, 田建东, 唐延东. 2016.基于光照不变图像的阴影图像道路及导航线提取算法.光学学报, 36(12):#1215004
Finlayson G D, Drew M S and Funt B V. 1994. Color constancy:generalized diagonal transforms suffice. Journal of the Optical Society of America A, 11(11):3011-3019[DOI:10.1364/JOSAA.11.003011]
Finlayson G D and Trezzi E. 2004. Shades of gray and colour constancy//Proceedings of the 12th Color Imaging Conference: Color Science and Engineering Systems, Technologies, and Applications. Scottsdale, AZ, USA: IS&T, 37-41.
Gao S B, Yang K F, Li C Y and Li Y. 2013. A color constancy model with double-opponency mechanisms//Proceedings of the 2013 IEEE International Conference on Computer Vision. Sydney, NSW, Australia: IEEE, 929-936[ DOI: 10.1109/ICCV.2013.119 http://dx.doi.org/10.1109/ICCV.2013.119 ]
Gao S B, Han W W, Yang K F and Li C. 2014. Efficient color constancy with local surface reflectance statistics//Proceedings of the 13th European Conference on Computer Vision. Zurich, Switzerland: Springer, 158-173[ DOI: 10.1007/978-3-319-10605-2_11 http://dx.doi.org/10.1007/978-3-319-10605-2_11 ]
Gao S B, Yang K F, Li C Y and Li Y. 2015. Color constancy using double-opponency. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(10):1973-1985[DOI:10.1109/TPAMI.2015.2396053]
Gao S B, Ren Y Z, Zhang M and Li Y. 2019. Combining bottom-up and top-down visual mechanisms for color constancy under varying illumination. IEEE Transactions on Image Processing, 28(9):4387-4400[DOI:10.1109/TIP.2019.2908783]
Gehler P V, Rother C, Blake A, Minka T and Sharp T. 2008. Bayesian color constancy revisited//Proceedings of 2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, AK, USA: IEEE, 1-8[ DOI: 10.1109/CVPR.2008.4587765 http://dx.doi.org/10.1109/CVPR.2008.4587765 ]
Gijsenij A, Gevers T and Van de Weijer J. 2010. Generalized gamut mapping using image derivative structures for color constancy. International Journal of Computer Vision, 86(2-3):127-139[DOI:10.1007/s11263-008-0171-3]
Gijsenij A and Gevers T. 2011. Color constancy using natural image statistics and scene semantics. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(4):687-698[DOI:10.1109/TPAMI.2010.93]
He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE, 770-778[ DOI: 10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]
Hu Y M, Wang B Y and Lin S. 2017. FC 4 : Fully convolutional color constancy with confidence-weighted pooling//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA: IEEE, 330-339[ DOI: 10.1109/CVPR.2017.43 http://dx.doi.org/10.1109/CVPR.2017.43 ].
Hu Y M, Wang B Y and Lin S. 2017. Code and resources for "FC4: Fully convolutional color constancy with confidence-weighted pooling" (CVPR 2017)[EB/OL]. (2018-11-20)[2019-02-17] . DOI: https://github.com/yuanming-hu/fc4 https://github.com/yuanming-hu/fc4
Huang D M, Wang Y, Song W, Wang Z H and Du Y L. 2018. Underwater image enhancement method using adaptive histogram stretching in different color models. Journal of Image and Graphics, 23(5):640-651
黄冬梅, 王龑, 宋巍, 王振华, 杜艳玲. 2018.不同颜色模型下自适应直方图拉伸的水下图像增强.中国图象图形学报, 23(5):640-651[DOI:10.11834/jig.170610]
Jia Y Q, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S and Darrell T. 2014. Caffe: convolutional architecture for fast feature embedding//Proceedings of the 22nd ACM International Conference on Multimedia. Orlando, FL, USA: ACM, 675-678[ DOI: 10.1145/2647868.2654889 http://dx.doi.org/10.1145/2647868.2654889 ]
Joze H R V, Drew M S, Finlayson G D and Rey P. 2012. The role of bright pixels in illumination estimation//Proceedings of the 20th Color and Imaging Conference. Los Angeles, CA, USA: IS&T, 493-496
Joze H R V and Drew M S. 2014. Exemplar-based color constancy and multiple illumination. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(5):860-873[DOI:10.1109/TPAMI.2013.169]
Krizhevsky A, Sutskever I and Hinton G E. 2012. ImageNet classification with deep convolutional neural networks//Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe, Nevada: ACM, 1097-1105
Land E H. 1977. The retinex theory of color vision. Scientific America, 237(6):108-129[DOI:10.1038/scientificamerican1277-108]
Li B, Xiong W H, Hu W M and Peng H. 2013. Illumination estimation based on bilayer sparse coding//Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland, OR, USA: IEEE, 1423-1429[ DOI: 10.1109/CVPR.2013.187 http://dx.doi.org/10.1109/CVPR.2013.187 ]
Li B, Xiong W H, Hu W M, Funt B and Xing J. 2016. Multi-cue illumination estimation via a tree-structured group joint sparse representation. International Journal of Computer Vision, 117(1):21-47[DOI:10.1007/s11263-015-0844-7]
Lou Z Y, Gevers T, Hu N H and Lucassen M. 2015. Color constancy by deep learning//Proceedings of the British Machine Vision Conference. Swansea, UK: BMVA, 76.1-76.12[ DOI: 10.5244/C.29.76 http://dx.doi.org/10.5244/C.29.76 ]
Oh S W and Kim S J. 2017. Approaching the computational color constancy as a classification problem through deep learning. Pattern Recognition, 61:405-416[DOI:10.1016/j.patcog.2016.08.013]
Qian Y L, Chen K, Kämäräinen J K, Nikkanen J and Matas J. 2016. Deep structured-output regression learning for computational color constancy//Proceedings of the 23rd International Conference on Pattern Recognition. Cancun, Mexico: IEEE, 1899-1904[ DOI: 10.1109/ICPR.2016.7899914 http://dx.doi.org/10.1109/ICPR.2016.7899914 ]
Shi L L and Funt B. 2014. Re-processed version of the Gehler color constancy dataset of 568 images[EB/OL]. (2014-05-07)[2019-02-17] . DOI: http://www.cs.sfu.ca/colour/data http://www.cs.sfu.ca/colour/data .
Shi W, Loy C C and Tang X O. 2016. Deep specialized network for illuminant estimation//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, Netherlands: Springer, 371-387[ DOI: 10.1007/978-3-319-46493-0_23 http://dx.doi.org/10.1007/978-3-319-46493-0_23 ]
Simonyan K and Zisserman A. 2015. Very deep convolutional networks for large-scale image recognition[EB/OL][2019-03-01] . DOI: https://arxiv.org/pdf/1409.1556.pdf https://arxiv.org/pdf/1409.1556.pdf .
Sutskever I, Martens J, Dahl G and Hinton G. 2013. On the importance of initialization and momentum in deep learning//Proceedings of the 30th International Conference on International Conference on Machine Learning. Atlanta, GA, USA: ACM, Ⅲ-1139-Ⅲ-1147.
Szegedy C, Liu W, Jia Y Q, Sermanet P, Reed S and Anguelov D. 2015. Going deeper with convolutions//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA: IEEE, 1-9[ DOI: 10.1109/CVPR.2015.7298594 http://dx.doi.org/10.1109/CVPR.2015.7298594 ]
Van de Weijer J, Gevers T and Gijsenij A. 2007. Edge-based color constancy. IEEE Transactions on Image Processing, 16(9):2207-2214[DOI:10.1109/TIP.2007.901808]
Wu K W, Yang X Z and Xie Z. 2016. Regional-oriented non-uniform illumination estimation. Acta Optica Sinica, 36(2):#0233001
吴克伟, 杨学志, 谢昭. 2016.面向区域的非均匀光照估计方法.光学学报, 36(2):#0233001[DOI:10.3788/aos201636.0233001]
Xiong W H and Funt B. 2006. Estimating illumination chromaticity via support vector regression. Journal of Imaging Science and Technology, 50(4):341-348[DOI:10.2352/J.ImagingSci.Technol.(2006)50:4(341)]
Yang K F, Gao S B and Li Y J. 2015. Efficient illuminant estimation for color constancy using grey pixels//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA: IEEE[ DOI: 10.1109/CVPR.2015.7298838 http://dx.doi.org/10.1109/CVPR.2015.7298838 ]
Zhang X S, Gao S B, Li R X, Du X, Li C and Li Y. 2016. A retinal mechanism inspired color constancy model. IEEE Transactions on Image Processing, 25(3):1219-1232[DOI:10.1109/TIP.2016.2516953]
相关作者
相关机构
京公网安备11010802024621
