An attention mechanism based inter-reflection compensation network for immersive projection system

Qinghua Lei; Ting Yang; Peng Cheng

doi:10.11834/jig.200608

Virtual Reality and Augmented Reality | Views : 0 下载量: 1992 CSCD: 0

PDF
Export
Share
Collection
Album

An attention mechanism based inter-reflection compensation network for immersive projection system
Vol. 27, Issue 4, Pages: 1238-1250(2022)
Received：22 October 2020，

Revised：2020-12-29，

Accepted：05 January 2021，

Published：16 April 2022
DOI： 10.11834/jig.200608
稿件说明：

移动端阅览

DOI：

Qinghua Lei, Ting Yang, Peng Cheng. An attention mechanism based inter-reflection compensation network for immersive projection system[J]. Journal of Image and Graphics, 2022, 27(4): 1238-1250. DOI： 10.11834/jig.200608.

摘要

目的

沉浸式投影系统已广泛运用于虚拟现实系统之中，然而沉浸式投影系统中的互反射现象严重影响着虚拟现实系统的落地使用。沉浸式投影系统的互反射是指由于投影机光线和屏幕反射光线相互叠加造成的亮度冗余现象，严重影响了投影系统的成像质量和人眼的视觉感受。为此，本文提出一种新的基于互反射通道(inter-reflection channel，IRC)先验和注意力机制的神经网络。

方法

IRC先验基于这样一个事实，即大多数受到互反射影响的投影图像都包含一些亮度较高的区域。高亮度区域往往受互反射影响更为严重，而低亮度区域受互反射影响程度较低。根据这一规律，采用IRC先验作为注意力图的监督样本，获取补偿图像的亮度区域信息。同时，为了对投影图像不同区域按影响程度进行差异化补偿，提出一种新的由两个相同子网络构成的补偿网络结构Pair-Net。

结果

实验对比了4种现有方法，Pair-Net在ROI(region of interesting)指标分析上取得了明显优势，在人眼感受上有显著的效果提升。

结论

本文提出的基于注意力机制的网络模型能够针对不同区域进行差异化补偿，很大程度上消除了互反射影响，提升了沉浸式投影系统的成像质量。

Abstract

Objective

Immersive projection system is focused on for the aspects of virtual reality and augmented reality system nowadays. In the context of immersive projection system

the inner-reflection issue is essential to the projection images quality and the fidelity of reality scenes. Inter-reflection refers to brightness redundancy problems derived of overlapping of projector light and screen reflection light in immersive projection system

which severely affects the imaging quality of the projection system. Meanwhile

it is a challenging issue to eliminate optics-based inner-reflection due to the complexity of light transmission in immersive environment.

Method

A new and simple image prior like inner-reflection channel (IRC) prior and a new attention guide neural network like Pair-Net generate the high-quality inner-reflection compensated projection image in immersive projection system. The IRC prior is a kind of statistic of projection image in immersive projection system. The scenario of most inner-reflection effected projection images are composed of some high intensity pixels. Those high intensity local patches are affected through inner-reflection

which can be used as an attention map to train our compensation net

IRC prior based Pair-Net

a new compensation network

learns the complex reflection and compensation function of immersive projection environment.

Result

Our experiment demonstrated the improvement in region of interesting (ROI) analysis indicators and human visual perception compared the four existing methods. Pair-Net is capable to learn the complex inner-reflection information and pay attention to the high inner-reflection region. The result of Pair-Net is qualified to the end-to-end projection compensation methods qualitatively and quantitatively.

Conclusion

Our method illustrates its qualitative and quantitative effectiveness based on significant margin. Immersive projection system have been widely using in those large-scare virtual-reality scene. But

inner-reflection almost exits in all immersive projection system which can heavily decrease the quality of projection image and the fidelity of reality scenes. These challenges often create bottlenecks for generalization of projector system and block the implementation of virtual reality projects. Inner-reflection compensation aims to compensate the projector input image to enhance the projection images quality and lower the effect of inner-reflection. The typical compensation system consists of in-situ projector-camera (pro-cam) pair and a curved screen. The geometric modeling sorts the light transmission and reflection function out. First

light transmission and reflection function in immersive projection environment need to invert a potential large-scale matrix. Next

it is hard for traditional inner-reflection compensation solution to produce high visually quality result due to the mathematical error are inevitable. Finally

current solutions compensate the whole images more and ignore multi-regions based single image intensity issue. A new convolutional neural network (CNN) is prior to photometric compensation domain based photometric compensation algorithm. We facilitated IRC prior and a Pair-Net for inner-reflection compensation. Pair-Net intends to the different patches of image in multiple light intensity immersive projection scenario. The adopted attention mechanisms for different intensity region compensation and use IRC prior to get the attention map. We design Pair-Net as composed of two sub-nets for paying different attention to the higher intensity and lower intensity region in single image. Two auto-encoder sub-net encourages rich multi-level interaction between the camera captured projection image and the ground truth image

and thus capturing the reflection information of the projection screen. Then

the IRC prior yields two sub-net to pay different attention to variance intensity region in immersive projection scenario summary

we first harness an attention guide inner-reflection compensation Pair-Net model in immersive projection system. In addition

the IRC prior is generated the attention map initially.

关键词

Keywords

references

Ashdown M, Okabe T, Sato I and Sato Y. 2006. Robust content-dependent photometric projector compensation//Proceedings of 2006 Conference on Computer Vision and Pattern Recognition Workshop. New York, USA: IEEE: 4-6 [ DOI: 10.1109/CVPRW.2006.172 http://dx.doi.org/10.1109/CVPRW.2006.172 ]

Bimber O, Grundhöfer A, Zeidler T, Danch D and Kapakos P. 2006. Compensating indirect scattering for immersive and semi-immersive projection displays//Proceedings of 2006 IEEE Virtual Reality Conference. Alexandria, USA: IEEE: 151-158 [ DOI: 10.1109/VR.2006.34 http://dx.doi.org/10.1109/VR.2006.34 ]

Chen L C, Yang Y, Wang J, Wei X and Yuille A L. 2016. Attention to scale: scale-aware semantic image segmentation//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE: 3640-3649 [ DOI: 10.1109/CVPR.2016.396 http://dx.doi.org/10.1109/CVPR.2016.396 ]

Grundhöfer A. 2013. Practical non-linear photometric projector compensation//Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Portland, USA: IEEE: 924-929 [ DOI: 10.1109/CVPRW.2013.136 http://dx.doi.org/10.1109/CVPRW.2013.136 ]

Grundhöfer A and Iwai D. 2015. Robust, error-tolerant photometric projector compensation. IEEE Transactions on Image Processing, 24(12): 5086-5099 [DOI: 10.1109/TIP.2015.2478388]

Habe H, Saeki N and Matsuyama T. 2007. Inter-reflection compensation for immersive projection display//Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis, USA: IEEE: 1-2 [ DOI: 10.1109/CVPR.2007.383473 http://dx.doi.org/10.1109/CVPR.2007.383473 ]

Huang B Y and Ling H B. 2019a. CompenNet++: end-to-end full projector compensation//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul, Korea (South): IEEE: 7164-7173 [ DOI: 10.1109/ICCV.2019.00726 http://dx.doi.org/10.1109/ICCV.2019.00726 ]

HuangB Y and Ling H B. 2019b. End-to-end projector photometric compensation//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 6803-6812 [ DOI: 10.1109/CVPR.2019.00697 http://dx.doi.org/10.1109/CVPR.2019.00697 ]

Kuen J, Wang Z H and Wang G. 2016. Recurrent attentional networks for saliency detection//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE: 3668-3677 [ DOI: 10.1109/CVPR.2016.399 http://dx.doi.org/10.1109/CVPR.2016.399 ]

Li J N, Wei Y C, Liang X D, Dong J, Xu T F, Feng J S and Yan S C. 2017. Attentive contexts for object detection. IEEE Transactions on Multimedia, 19(5): 944-954 [DOI: 10.1109/TMM.2016.2642789]

Li Y Q, Yuan Q S and Lu D M. 2013. Perceptual radiometric compensation for inter-reflection in immersive projection environment//Proceedings of the 19th ACM Symposium on Virtual Reality Software and Technology. Singapore, Singapore: ACM: 201-208 [ DOI: 10.1145/2503713.2503720 http://dx.doi.org/10.1145/2503713.2503720 ]

Ma S L, Ma B K and Song M. 2012. Signal attenuation time series prediction method based on the chaos algorithm//Proceedings of 2012 International Symposium on Antennas, Propagation and EM Theory. Xi'an, China: IEEE: 802-806 [ DOI: 10.1109/ISAPE.2012.6408893 http://dx.doi.org/10.1109/ISAPE.2012.6408893 ]

Mejjati Y A, Richardt C, Tompkin J, Cosker D and Kim K I. 2018. Unsupervised attention-guided image-to-image translation//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Montréal, Canada: ACM: 3697-3707

Ng R, Ramamoorthi R and Hanrahan P. 2003. All-frequency shadows using non-linear wavelet lighting approximation//ACM SIGGRAPH 2003 Papers. San Diego, USA: ACM: 376-381 [ DOI: 10.1145/1201775.882280 http://dx.doi.org/10.1145/1201775.882280 ]

Ng T T, Pahwa R S, Bai J M, Tan K H and Ramamoorthi R. 2012. From the rendering equation to stratified light transport inversion. International Journal of Computer Vision, 96(2): 235-251 [DOI: 10.1007/s11263-011-0467-6]

Ronneberger O, Fischer P and Brox T. 2015. U-Net: Convolutional networks for biomedical image segmentation//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich, Germany: Springer: 234-241 [ DOI: 10.1007/978-3-319-24574-4_28 http://dx.doi.org/10.1007/978-3-319-24574-4_28 ]

Takeda S, Iwai D and Sato K. 2016. Inter-reflection compensation of immersive projection display by spatio-temporal screen reflectance modulation. IEEE Transactions on Visualization and Computer Graphics, 22(4): 1424-1431 [DOI: 10.1109/TVCG.2016.2518136]

Wang J P, Dong Y, Tong X, Lin Z C and Guo B N. 2009. Kernel Nyström method for light transport. ACM Transactions on Graphics, 28(3): #29 [DOI: 10.1145/1531326.1531335]

Xu H R, Guo J M, Liu Q and Ye L L. 2012. Fast image dehazing using improved dark channel prior//Proceedings of 2012 IEEE International Conference on Information Science and Technology. Wuhan, China: IEEE: 663-667 [ DOI: 10.1109/ICIST.2012.6221729 http://dx.doi.org/10.1109/ICIST.2012.6221729 ]

Zhao H, Gallo O, Frosio I and Kautz J. 2017. Loss functions for image restoration with neural networks. IEEE Transactions on Computational Imaging, 3(1): 47-57 [DOI: 10.1109/tci.2016.2644865]

Zhu J Y, Park T and Isola P. 2010. Unpaired image-to-image translation using cycle-consistent adversarial networks//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE: 183-202 [ DOI: 10.1109/ICCV.2017.244 http://dx.doi.org/10.1109/ICCV.2017.244 ]

Zou W H, Xu H S, Han B and Park D. 2008. A novel methodology for photometric compensation of projection display on patterned screen. Chinese Optics Letters, 6(7): 499-501

Alert me when the article has been cited

提交

Image relighting via depth-light direction joint modeling

Two-stage dual-branch model for 3D point cloud denoising

Image dehazing network based on dual-domain feature fusion

Multi-branch micro-expression recognition method based on multi-type features and multi-attention mechanism

Supervised attention-based oriented object detection in remote sensing images