浏览全部资源
扫码关注微信
纸质出版日期： 2022-04-16 ，
录用日期： 2021-01-05
移动端阅览
雷清桦, 杨婷, 程鹏. 注意力机制的曲面沉浸式投影系统补偿[J]. 中国图象图形学报, 2022,27(4):1238-1250.
Qinghua Lei, Ting Yang, Peng Cheng. An attention mechanism based inter-reflection compensation network for immersive projection system[J]. Journal of Image and Graphics, 2022,27(4):1238-1250.
目的
2
沉浸式投影系统已广泛运用于虚拟现实系统之中，然而沉浸式投影系统中的互反射现象严重影响着虚拟现实系统的落地使用。沉浸式投影系统的互反射是指由于投影机光线和屏幕反射光线相互叠加造成的亮度冗余现象，严重影响了投影系统的成像质量和人眼的视觉感受。为此，本文提出一种新的基于互反射通道(inter-reflection channel，IRC)先验和注意力机制的神经网络。
方法
2
IRC先验基于这样一个事实，即大多数受到互反射影响的投影图像都包含一些亮度较高的区域。高亮度区域往往受互反射影响更为严重，而低亮度区域受互反射影响程度较低。根据这一规律，采用IRC先验作为注意力图的监督样本，获取补偿图像的亮度区域信息。同时，为了对投影图像不同区域按影响程度进行差异化补偿，提出一种新的由两个相同子网络构成的补偿网络结构Pair-Net。
结果
2
实验对比了4种现有方法，Pair-Net在ROI(region of interesting)指标分析上取得了明显优势，在人眼感受上有显著的效果提升。
结论
2
本文提出的基于注意力机制的网络模型能够针对不同区域进行差异化补偿，很大程度上消除了互反射影响，提升了沉浸式投影系统的成像质量。
Objective
2
Immersive projection system is focused on for the aspects of virtual reality and augmented reality system nowadays. In the context of immersive projection system
the inner-reflection issue is essential to the projection images quality and the fidelity of reality scenes. Inter-reflection refers to brightness redundancy problems derived of overlapping of projector light and screen reflection light in immersive projection system
which severely affects the imaging quality of the projection system. Meanwhile
it is a challenging issue to eliminate optics-based inner-reflection due to the complexity of light transmission in immersive environment.
Method
2
A new and simple image prior like inner-reflection channel (IRC) prior and a new attention guide neural network like Pair-Net generate the high-quality inner-reflection compensated projection image in immersive projection system. The IRC prior is a kind of statistic of projection image in immersive projection system. The scenario of most inner-reflection effected projection images are composed of some high intensity pixels. Those high intensity local patches are affected through inner-reflection
which can be used as an attention map to train our compensation net
IRC prior based Pair-Net
a new compensation network
learns the complex reflection and compensation function of immersive projection environment.
Result
2
Our experiment demonstrated the improvement in region of interesting (ROI) analysis indicators and human visual perception compared the four existing methods. Pair-Net is capable to learn the complex inner-reflection information and pay attention to the high inner-reflection region. The result of Pair-Net is qualified to the end-to-end projection compensation methods qualitatively and quantitatively.
Conclusion
2
Our method illustrates its qualitative and quantitative effectiveness based on significant margin. Immersive projection system have been widely using in those large-scare virtual-reality scene. But
inner-reflection almost exits in all immersive projection system which can heavily decrease the quality of projection image and the fidelity of reality scenes. These challenges often create bottlenecks for generalization of projector system and block the implementation of virtual reality projects. Inner-reflection compensation aims to compensate the projector input image to enhance the projection images quality and lower the effect of inner-reflection. The typical compensation system consists of in-situ projector-camera (pro-cam) pair and a curved screen. The geometric modeling sorts the light transmission and reflection function out. First
light transmission and reflection function in immersive projection environment need to invert a potential large-scale matrix. Next
it is hard for traditional inner-reflection compensation solution to produce high visually quality result due to the mathematical error are inevitable. Finally
current solutions compensate the whole images more and ignore multi-regions based single image intensity issue. A new convolutional neural network (CNN) is prior to photometric compensation domain based photometric compensation algorithm. We facilitated IRC prior and a Pair-Net for inner-reflection compensation. Pair-Net intends to the different patches of image in multiple light intensity immersive projection scenario. The adopted attention mechanisms for different intensity region compensation and use IRC prior to get the attention map. We design Pair-Net as composed of two sub-nets for paying different attention to the higher intensity and lower intensity region in single image. Two auto-encoder sub-net encourages rich multi-level interaction between the camera captured projection image and the ground truth image
and thus capturing the reflection information of the projection screen. Then
the IRC prior yields two sub-net to pay different attention to variance intensity region in immersive projection scenario summary
we first harness an attention guide inner-reflection compensation Pair-Net model in immersive projection system. In addition
the IRC prior is generated the attention map initially.
沉浸式投影系统互反射补偿深度学习注意力机制虚拟现实
immersive projection systeminter-reflection compensationdeep learningattention mechanismvirtual reality
Ashdown M, Okabe T, Sato I and Sato Y. 2006. Robust content-dependent photometric projector compensation//Proceedings of 2006 Conference on Computer Vision and Pattern Recognition Workshop. New York, USA: IEEE: 4-6 [DOI: 10.1109/CVPRW.2006.172http://dx.doi.org/10.1109/CVPRW.2006.172]
Bimber O, Grundhöfer A, Zeidler T, Danch D and Kapakos P. 2006. Compensating indirect scattering for immersive and semi-immersive projection displays//Proceedings of 2006 IEEE Virtual Reality Conference. Alexandria, USA: IEEE: 151-158 [DOI: 10.1109/VR.2006.34http://dx.doi.org/10.1109/VR.2006.34]
Chen L C, Yang Y, Wang J, Wei X and Yuille A L. 2016. Attention to scale: scale-aware semantic image segmentation//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 3640-3649 [DOI: 10.1109/CVPR.2016.396http://dx.doi.org/10.1109/CVPR.2016.396]
Grundhöfer A. 2013. Practical non-linear photometric projector compensation//Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Portland, USA: IEEE: 924-929 [DOI: 10.1109/CVPRW.2013.136http://dx.doi.org/10.1109/CVPRW.2013.136]
Grundhöfer A and Iwai D. 2015. Robust, error-tolerant photometric projector compensation. IEEE Transactions on Image Processing, 24(12): 5086-5099 [DOI: 10.1109/TIP.2015.2478388]
Habe H, Saeki N and Matsuyama T. 2007. Inter-reflection compensation for immersive projection display//Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis, USA: IEEE: 1-2 [DOI: 10.1109/CVPR.2007.383473http://dx.doi.org/10.1109/CVPR.2007.383473]
Huang B Y and Ling H B. 2019a. CompenNet++: end-to-end full projector compensation//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul, Korea (South): IEEE: 7164-7173 [DOI: 10.1109/ICCV.2019.00726http://dx.doi.org/10.1109/ICCV.2019.00726]
HuangB Y and Ling H B. 2019b. End-to-end projector photometric compensation//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 6803-6812 [DOI: 10.1109/CVPR.2019.00697http://dx.doi.org/10.1109/CVPR.2019.00697]
Kuen J, Wang Z H and Wang G. 2016. Recurrent attentional networks for saliency detection//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 3668-3677 [DOI: 10.1109/CVPR.2016.399http://dx.doi.org/10.1109/CVPR.2016.399]
Li J N, Wei Y C, Liang X D, Dong J, Xu T F, Feng J S and Yan S C. 2017. Attentive contexts for object detection. IEEE Transactions on Multimedia, 19(5): 944-954 [DOI: 10.1109/TMM.2016.2642789]
Li Y Q, Yuan Q S and Lu D M. 2013. Perceptual radiometric compensation for inter-reflection in immersive projection environment//Proceedings of the 19th ACM Symposium on Virtual Reality Software and Technology. Singapore, Singapore: ACM: 201-208 [DOI: 10.1145/2503713.2503720http://dx.doi.org/10.1145/2503713.2503720]
Ma S L, Ma B K and Song M. 2012. Signal attenuation time series prediction method based on the chaos algorithm//Proceedings of 2012 International Symposium on Antennas, Propagation and EM Theory. Xi'an, China: IEEE: 802-806 [DOI: 10.1109/ISAPE.2012.6408893http://dx.doi.org/10.1109/ISAPE.2012.6408893]
Mejjati Y A, Richardt C, Tompkin J, Cosker D and Kim K I. 2018. Unsupervised attention-guided image-to-image translation//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Montréal, Canada: ACM: 3697-3707
Ng R, Ramamoorthi R and Hanrahan P. 2003. All-frequency shadows using non-linear wavelet lighting approximation//ACM SIGGRAPH 2003 Papers. San Diego, USA: ACM: 376-381 [DOI: 10.1145/1201775.882280http://dx.doi.org/10.1145/1201775.882280]
Ng T T, Pahwa R S, Bai J M, Tan K H and Ramamoorthi R. 2012. From the rendering equation to stratified light transport inversion. International Journal of Computer Vision, 96(2): 235-251 [DOI: 10.1007/s11263-011-0467-6]
Ronneberger O, Fischer P and Brox T. 2015. U-Net: Convolutional networks for biomedical image segmentation//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich, Germany: Springer: 234-241 [DOI: 10.1007/978-3-319-24574-4_28http://dx.doi.org/10.1007/978-3-319-24574-4_28]
Takeda S, Iwai D and Sato K. 2016. Inter-reflection compensation of immersive projection display by spatio-temporal screen reflectance modulation. IEEE Transactions on Visualization and Computer Graphics, 22(4): 1424-1431 [DOI: 10.1109/TVCG.2016.2518136]
Wang J P, Dong Y, Tong X, Lin Z C and Guo B N. 2009. Kernel Nyström method for light transport. ACM Transactions on Graphics, 28(3): #29 [DOI: 10.1145/1531326.1531335]
Xu H R, Guo J M, Liu Q and Ye L L. 2012. Fast image dehazing using improved dark channel prior//Proceedings of 2012 IEEE International Conference on Information Science and Technology. Wuhan, China: IEEE: 663-667 [DOI: 10.1109/ICIST.2012.6221729http://dx.doi.org/10.1109/ICIST.2012.6221729]
Zhao H, Gallo O, Frosio I and Kautz J. 2017. Loss functions for image restoration with neural networks. IEEE Transactions on Computational Imaging, 3(1): 47-57 [DOI: 10.1109/tci.2016.2644865]
Zhu J Y, Park T and Isola P. 2010. Unpaired image-to-image translation using cycle-consistent adversarial networks//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE: 183-202 [DOI: 10.1109/ICCV.2017.244http://dx.doi.org/10.1109/ICCV.2017.244]
Zou W H, Xu H S, Han B and Park D. 2008. A novel methodology for photometric compensation of projection display on patterned screen. Chinese Optics Letters, 6(7): 499-501
相关作者
相关机构
京公网安备11010802024621