浏览全部资源
扫码关注微信
Published： 16 October 2024 ，
移动端阅览
祁佳晨， 解利军， 阮文凯， 王孝强. 2024. 混合现实飞行模拟器的神经重光照方法. 中国图象图形学报， 29(10):3008-3021
Qi Jiachen， Xie Lijun， Ruan Wenkai， Wang Xiaoqiang. 2024. Neural relighting methods for mixed reality flight simulators. Journal of Image and Graphics， 29(10):3008-3021
目的
2
混合现实技术通过混合现实场景和虚拟场景，为飞行模拟器提供了沉浸式体验。由于现实场景和虚拟场景的光照条件不一致，混合结果往往使用户产生较强的不协调感，从而降低体验沉浸感。本文使用虚拟场景的光照条件对机舱现实图像场景进行重光照，解决光照不一致问题。
方法
2
受计算机图形学重要的渲染方法——预计算辐射传输法的启发，首次提出一种基于辐射传输函数估计的神经重光照方法。首先使用卷积神经网络估计输入图像中每个渲染点的辐射传输函数在球谐函数上的系数形式表达，同时将虚拟环境中提供光照信息的环境光贴图投影到球谐函数上，最后将对应球谐系数向量进行点乘，获得重光照渲染结果。
结果
2
目视评测，生成的重光照图像与目标光照条件匹配程度良好，同时保留原图中细节，未出现伪影等异常渲染结果。以本文生成的重光照数据集为基准进行测试，本文方法生成结果峰值信噪比达到28.48 dB，比相似方法高出7.5%。
结论
2
成功在多款战斗机模型中应用了上述方法，可以根据给定虚拟飞行场景中的光照条件，对现实机舱内部图像进行重光照，实现机舱内外图像光照条件一致，提升了应用混合现实的飞行模拟器的用户沉浸感。
Objective
2
The application of mixed reality （MR） in training environments， particularly in the field of aviation， marks a remarkable leap from traditional simulation models. This innovative technology overlays virtual elements onto the real world， creating a seamless interactive experience that is critical in simulating high-risk scenarios for pilots. Despite its advances， the integration of real and virtual elements often suffers from inconsistencies in lighting， which can disrupt the user’s sense of presence and diminish the effectiveness of training sessions. Prior attempts to reconcile these differences have involved static solutions that lack adaptability to the dynamic range of real-world lighting conditions encountered during flight. This study is informed by a comprehensive review of current methodologies， including photometric alignment techniques and the adaptation of CGI （computer-generated imagery） elements using standard graphics pipelines. Our analysis identified a gap in real-time dynamic relighting capabilities， which we address through a novel neural network-based approach.
Method
2
The methodological core of this research is the development of an advanced neural network architecture designed for the sophisticated task of image relighting. The neural network architecture proposed in this research is a convolutional neural network variant， specifically tailored to process high-fidelity images in a manner that retains critical details while adjusting to new lighting conditions. Meanwhile， an integral component of our methodology was the generation of a comprehensive dataset specifically tailored for the relighting of fighter jet cockpit environments. To ensure a high degree of realism， we synthesized photorealistic renderings of the cockpit interior under a wide array of atmospheric conditions， times of day， and geolocations across different latitudes and longitudes. This synthetic dataset was achieved by integrating our image capture process with an advanced weather simulation system， which allowed us to replicate the intricate effects of natural and artificial lighting as experienced within the cockpit. The resultant dataset presents a rich variety of lighting scenarios， ranging from the low-angle illumination of a sunrise to the diffused lighting of an overcast sky， providing our neural network with the nuanced training required to emulate real-world lighting dynamics accurately. The neural network is trained with this dataset to understand and dissect the complex interplay of lighting and material properties within a scene. The first step of the network involves a detailed decomposition of input images to separate and analyze the components affected by lighting， such as shadows， highlights， and color temperature. The geometry of the scene， the textures， and how objects occlude or reflect light must be deduced， extracting these elements into a format that can be manipulated independently of the original lighting conditions. To actualize the target lighting effect， the study leverages a concept adapted from the domain of precomputed radiance transfer——a technique traditionally used for rendering scenes with complex light interactions. By estimating radiance transfer functions at each pixel and representing these as coefficients over a series of spherical harmonic basis functions， the method facilitates a rapid and accurate recalculation of lighting across the scene. The environmental lighting conditions， captured through high dynamic range imaging techniques， are also projected onto these spherical harmonic functions. This approach allows for the real-time adjustment of lighting by simply recalculating the dot product of these coefficients， corresponding to the new lighting environment. This step is a computational breakthrough because it circumvents the need for extensive ray tracing or radiosity calculations， which are computationally expensive and often impractical for real-time applications. This method stands out for its low computational overhead， enabling near real-time relighting that can adjust dynamically as the simulated conditions change.
Result
2
The empirical results achieved through this method are substantiated through a series of rigorous tests and comparative analyses. The neural network’s performance was benchmarked against traditional and contemporary relighting methods across several scenarios reflecting diverse lighting conditions and complexities. The model consistently demonstrated superior performance， not only in the accuracy of light replication but also in maintaining the fidelity of the original textures and material properties. The visual quality of the relighting was assessed through objective performance metrics， including comparison of luminance distribution， color fidelity， and texture preservation against ground truth datasets. These metrics consistently indicated a remarkable improvement in visual coherence and a reduction in artifacts， ensuring a more immersive experience without the reliance on subjective user studies.
Conclusion
2
The implemented method effectively resolves the challenge of inconsistent lighting conditions in MR flight simulators. It contributes to the field by enabling dynamic adaptation of real-world images to the lighting conditions of virtual environments. This research not only provides a valuable tool for enhancing the realism and immersion of flight simulators but also offers insights that could benefit future theoretical and practical advancements in MR technology. The study utilized spherical harmonic coefficients of environmental light maps to convey lighting condition information and pioneered the extraction of scene radiance lighting functions’ spherical harmonic coefficients from real image data. This validated the feasibility of predicting scene radiance transfer functions from real images using neural networks. The limitations and potential improvements of the current method are discussed， outlining directions for future research. For example， considering the temporal continuity present in the relighted images， future efforts could exploit this characteristic to optimize the neural network architecture， integrating modules that enhance the stability of the prediction results.
重光照神经渲染方法辐射传输函数混合现实（MR）飞行模拟器
relightingneural rendering methodsradiance transfer functionsmixed reality（MR）flight simulator
Einabadi F， Guillemaut J Y and Hilton A. 2021. Deep neural models for illumination estimation and relighting： a survey. Computer Graphics Forum， 40（6）： 315-331 ［DOI： 10.1111/cgf.14283http://dx.doi.org/10.1111/cgf.14283］
El Helou M， Zhou R F， Barthas J and Süsstrunk S. 2020. VIDIT： virtual image dataset for illumination transfer. ［EB/OL］. ［2023-7-12］. https://doi.org/10.48550/arXiv.2307.06335https://doi.org/10.48550/arXiv.2307.06335
Eslami S M A， Rezende D J， Besse F， Viola F， Morcos A S， Garnelo M， Ruderman A， Rusu A A， Danihelka I， Gregor K， Reichert D P， Buesing L， Weber T， Vinyals O， Rosenbaum D， Rabinowitz N， King H， Hillier C， Botvinick M， Wierstra D， Kavukcuoglu K and Hassabis D. 2018. Neural scene representation and rendering. Science， 360（6394）： 1204-1210 ［DOI： 10.1126/science.aar6170http://dx.doi.org/10.1126/science.aar6170］
Feng Y， Xue B H， Liu M， Chen Q J and Fan R. 2023. D2NT： a high-performing depth-to-normal translator//Proceedings of 2023 IEEE International Conference on Robotics and Automation. London， UK： IEEE ［DOI： 10.1109/ICRA48891.2023.10161000http://dx.doi.org/10.1109/ICRA48891.2023.10161000］
Han C， Sun B， Ramamoorthi R and Grinspun E. 2007. Frequency domain normal map filtering//Proceedings of the ACM SIGGRAPH 2007. San Diego， USA： ACM， 2007： #28 ［DOI： 10.1145/1275808.1276412］
Isola P， Zhu J Y， Zhou T H and Efros A A. 2017. Image-to-image translation with conditional adversarial networks//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu， USA： IEEE： 5967-5976 ［DOI： 10.1109/CVPR.2017.632http://dx.doi.org/10.1109/CVPR.2017.632］
Kajiya J T. 1986. The rendering equation//Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques. Dallas， USA： ACM： 143-150 ［DOI： 10.1145/15922.15902http://dx.doi.org/10.1145/15922.15902］
Keskar N S， Mudigere D， Nocedal J， Smelyanskiy M and Tang P T P. 2017. On large-batch training for deep learning： generalization gap and sharp minima//Proceedings of the 5th International Conference on Learning Representations. Toulon， France： OpenReview.net
LeCun Y， Huang F J and Bottou L. 2004. Learning methods for generic object recognition with invariance to pose and lighting//Proceedings of 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington， USA： IEEE： II-104 ［DOI： 10.1109/CVPR.2004.1315150http://dx.doi.org/10.1109/CVPR.2004.1315150］
Loshchilov I and Hutter F. 2019. Decoupled weight decay regularization//Proceedings of the 7th International Conference on Learning Representations. New Orleans， USA： OpenReview.net
Milgram P and Kishino F. 1994. A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems， E77-D（12）： 1321-1329
Murmann L， Gharbi M， Aittala M and Durand F. 2019. A dataset of multi-illumination images in the wild//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul， Korea （South）： IEEE： 4080-4089 ［DOI： 10.1109/ICCV.2019.00418http://dx.doi.org/10.1109/ICCV.2019.00418］
Ng R， Ramamoorthi R and Hanrahan P. 2004. Triple product wavelet integrals for all-frequency relighting. ACM Transactions on Graphics， 23（3）： 477-487 ［DOI： 10.1145/1015706.1015749http://dx.doi.org/10.1145/1015706.1015749］
Nicolet B， Philip J and Drettakis G. 2020. Repurposing a relighting network for realistic compositions of captured scenes//Proceedings of the Symposium on Interactive 3D Graphics and Games. San Francisco， USA： ACM： #4 ［DOI： 10.1145/3384382.3384523http://dx.doi.org/10.1145/3384382.3384523］
Pandey R， Escolano S O， Legendre C， Häne C， Bouaziz S， Rhemann C， Debevec P and Fanello S. 2021. Total relighting： learning to relight portraits for background replacement. ACM Transactions on Graphics， 40（4）： #43 ［DOI： 10.1145/3450626.3459872http://dx.doi.org/10.1145/3450626.3459872］
Philip J， Gharbi M， Zhou T H， Efros A A and Drettakis G. 2019. Multi-view relighting using a geometry-aware network. ACM Transactions on Graphics， 38（4）： #78 ［DOI： 10.1145/3306346.3323013http://dx.doi.org/10.1145/3306346.3323013］
Philip J， Morgenthaler S， Gharbi M and Drettakis G. 2021. Free-viewpoint indoor neural relighting from multi-view stereo. ACM Transactions on Graphics， 40（5）： #194 ［DOI： 10.1145/3469842http://dx.doi.org/10.1145/3469842］
Raghavan N， Xiao Y， Lin K E， Sun T， Bi S， Xu Z， Li T M and Ramamoorthi R. 2023. Neural free‐viewpoint relighting for glossy indirect illumination. Computer Graphics Forum， 42（4）： #e14885 ［DOI： https://doi.org/10.1111/cgf.14885https://doi.org/10.1111/cgf.14885］
Rainer G， Bousseau A， Ritschel T and Drettakis G. 2022. Neural precomputed radiance transfer. Computer Graphics Forum， 41（2）： 365-378 ［DOI： 10.1111/cgf.14480http://dx.doi.org/10.1111/cgf.14480］
Ramamoorthi R and Hanrahan P. 2001. An efficient representation for irradiance environment maps//Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. Los Angeles， USA： ACM： 497-500 ［DOI： 10.1145/383259.383317http://dx.doi.org/10.1145/383259.383317］
Ren P R， Dong Y， Lin S， Tong X and Guo B N. 2015. Image based relighting using neural networks. ACM Transactions on Graphics， 34（4）： #111 ［DOI： 10.1145/2766899http://dx.doi.org/10.1145/2766899］
Ritschel T， Dachsbacher C， Grosch T and Kautz J. 2012. The state of the art in interactive global illumination. Computer Graphics Forum， 31（1）： 160-188 ［DOI： 10.1111/j.1467-8659.2012.02093.xhttp://dx.doi.org/10.1111/j.1467-8659.2012.02093.x］
Ronneberger O， Fischer P and Brox T. 2015. U-net： convolutional networks for biomedical image segmentation//Proceedings of the 18th Medical Image Computing and Computer-Assisted Intervention. Munich， Germany： Springer： 234-241 ［DOI： 10.1007/978-3-319-24574-4_28http://dx.doi.org/10.1007/978-3-319-24574-4_28］
Sengupta S， Gu J W， Kim K， Liu G L， Jacobs D and Kautz J. 2019. Neural inverse rendering of an indoor scene from a single image//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul， Korea （South）： IEEE： 8597-8606 ［DOI： 10.1109/ICCV.2019.00869http://dx.doi.org/10.1109/ICCV.2019.00869］
Sloan P P， Kautz J and Snyder J. 2002. Precomputed radiance transfer for real-time rendering in dynamic， low-frequency lighting environments. ACM Transactions on Graphics， 21（3）： 527-536 ［DOI： 10.1145/566654.566612http://dx.doi.org/10.1145/566654.566612］
Sun T C， Barron J T， Tsai Y T， Xu Z X， Yu X M， Fyffe G， Rhemann C， Busch J， Debevec P and Ramamoorthi R. 2019. Single image portrait relighting. ACM Transactions on Graphics， 38（4）： #79 ［DOI： 10.1145/3306346.3323008http://dx.doi.org/10.1145/3306346.3323008］
Tewari A， Fried O， Thies J， Sitzmann V， Lombardi S， Sunkavalli K， Martin-Brualla R， Simon T， Saragih J， Nießner M， Pandey R， Fanello S， Wetzstein G， Zhu J Y， Theobalt C， Agrawala M， Shechtman E， Goldman D B and Zollhöfer M. 2020. State of the art on neural rendering. Computer Graphics Forum， 39（2）： 701-727 ［DOI： 10.1111/cgf.14022http://dx.doi.org/10.1111/cgf.14022］
Xu Z L， Zeng Z， Wu L F， Wang L and Yan L Q. 2022. Lightweight neural basis functions for all-frequency shading//Proceedings of 2022 SIGGRAPH Asia Conference. Daegu， Korea（South）： ACM： #14 ［DOI： 10.1145/3550469.3555386http://dx.doi.org/10.1145/3550469.3555386］
Xu Z X， Sunkavalli K， Hadap S and Ramamoorthi R. 2018. Deep image-based relighting from optimal sparse samples. ACM Transactions on Graphics， 37（4）： #126 ［DOI： 10.1145/3197517.3201313http://dx.doi.org/10.1145/3197517.3201313］
Yan M， Wang J Z and Li J. 2022. Reliable binocular disparity estimation based on multi-scale similarity recursive search. Journal of Image and Graphics， 27（2）： 447-460
晏敏， 王军政， 李静. 2022. 多尺度相似性迭代查找的可靠双目视差估计. 中国图象图形学报， 27（2）： 447-460 ［DOI： 10.11834/jig.210551http://dx.doi.org/10.11834/jig.210551］
Yu Y， Meka A， Elgharib M， Seidel H P， Theobalt C and Smith W A P. 2020. Self-supervised outdoor scene relighting//Proceedings of the 16th European Conference on Computer Vision. Glasgow， UK： Springer： 84-101 ［DOI： 10.1007/978-3-030-58542-6_6http://dx.doi.org/10.1007/978-3-030-58542-6_6］
Yu Y and Smith W A P. 2019. InverseRenderNet： learning single image inverse rendering//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach， USA： IEEE： 3155-3164 ［DOI： 10.1109/CVPR.2019.00327http://dx.doi.org/10.1109/CVPR.2019.00327］
Zeng C， Chen G J， Dong Y， Peers P， Wu H Z and Tong X. 2023. Relighting neural radiance fields with shadow and highlight hints//Proceedings of 2023 ACM SIGGRAPH Conference. Los Angeles， USA： ACM： #73 ［DOI： 10.1145/3588432.3591482http://dx.doi.org/10.1145/3588432.3591482］
Zhan F N， Lu S J， Zhang C G， Ma F Y and Xie X S. 2020. Adversarial image composition with auxiliary illumination//Proceedings of the 15th Asian Conference on Computer Vision. Kyoto， Japan： Springer： 234-250 ［DOI： 10.1007/978-3-030-69532-3_15http://dx.doi.org/10.1007/978-3-030-69532-3_15］
Zhou H， Hadap S， Sunkavalli K and Jacobs D. 2019. Deep single-image portrait relighting//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul， Korea （South）： IEEE： 7193-7201 ［DOI： 10.1109/ICCV.2019.00729http://dx.doi.org/10.1109/ICCV.2019.00729］
Zhu Z L， Li Z， Zhang R X， Guo C L and Cheng M M. 2022. Designing an illumination-aware network for deep image relighting. IEEE Transactions on Image Processing， 31： 5396-5411 ［DOI： 10.1109/TIP.2022.3195366http://dx.doi.org/10.1109/TIP.2022.3195366］
相关文章
相关作者
相关机构
京公网安备11010802024621