浏览全部资源
扫码关注微信
收稿日期：2020-12-31，
修回日期：2021-03-13，
录用日期：2021-3-20，
纸质出版日期：2021-06-16
移动端阅览
可微绘制技术是当前虚拟现实、计算机图形学与计算机视觉领域研究的热点，其目标是改造计算机图形学中以光栅化或光线跟踪算法为主的真实感绘制流程，支持梯度信息回传以计算由输出图像的变化导致的输入几何、材质属性变化，通过与优化及深度学习技术等相结合支持从数据中学习绘制模型和逆向推理，是可微学习技术在计算机图形学绘制技术中的应用的具体体现，在增强/虚拟现实内容生成、三维重建、表观采集建模和逆向光学设计等领域中有广泛的应用前景。本文对可微绘制当前的发展状况进行调研，重点对该技术在真实感绘制、3维重建和表观采集建模中的研究和应用情况进行综述，并对可微绘制技术发展趋势进行展望，以期推动可微技术在学术界和产业界的进一步发展。
Differential rendering is currently a research focus in virtual reality
computer graphics
and computer vision. Its goal is to reform the rendering pipeline in computer graphics to support gradient backpropagation such that the change in the output image can be related to the change in input geometry or materials. The development of differential rendering technique is highly related to the deep learning
since neural networks are usually represented as computational graphs to support gradient backpropagation using the chain rule. Thus
the gradient backpropagation is the key to convert a computational procedure into a learnable process
which can significantly generalize the deep learning technique to a wide range of applications. Differential rendering follows this trend to integrate gradient backpropagation into rendering pipeline. It can significantly facilitate the gradient computation through auto-differential techniques. In fact
the derivatives of rendering results regarding to the mesh vertex have already computed in variational 3D reconstruction and shape from shading. However
differential rendering integrates the derivative computation into global rendering pipelines and neural networks. Therefore
the rendering process can be directly integrated into optimization or neural network training to approximate rendering pipeline or inverse graphics reasoning; it has wide applications in content creation in augmented/virtual reality
3D reconstruction
appearance modeling
and inverse design. The advantage of differential rendering over traditional rendering pipeline is that it allows to train neural networks to approximate the forward rendering pipeline. Once trained
the rendering results can be obtained through forward inference of the network
a much faster procedure in many situations. Moreover
the gradient information provided by differential rendering is helpful to improve the efficiency of the global rendering. For instance
the first- and second-order gradients can be used to guide the sampling process in Monte Carlo rendering. Another advantage of differential rendering is that it can directly be used in view interpolation or view synthesis through captured images
which traditional rendering pipeline needs geometry
appearance and lighting information simultaneously to render an image at specified viewpoints. In the application of differential rendering to view synthesis or image-based rendering
the implicit representation of a 3D scene is usually inferred from the captured images directly via deep neural networks supervised by differential rendering loss. Such a process falls into the category of self-supervised learning because ground truth 3D data are not provided during training. It bypasses the expensive multi-view 3D reconstruction and thus significantly simplifies the view synthesis procedure. Numerous representations
such as neural texture
neural volume
and neural implicit function
are proposed to handle freeview point rendering of a 3D scene. However
the training and rendering cost of these methods is still expensive. Thus
reducing their computational cost forms a new research direction. Differential rendering also enables the end-to-end inference of spatially variant bidirectional reflectance distribution function (BRDF) material properties from capture images. The BRDF parameters can be derived from a single image after training the deep neural network on a large amount of data by representing the material properties in a latent space. Moreover
with a differentiable pipeline
the layout of the light sources and projection patterns of dedicated appearance acquisition equipment can be optimized.The recent development of differential rendering
including its application in realistic rendering
3D reconstruction
and appearance modeling is comprehensively surveyed. We expect this study to further boost the research on differential rendering in academia and industry.
Bangaru S P, Li T M and Durand F. 2020. Unbiased warped-area sampling for differentiable rendering. ACM Transactions on Graphics, 39(6): #245[DOI:10.1145/3414685.3417833]
Bemana M, Myszkowski K, Seidel H P and Ritschel T. 2020. X-Fields: implicit neural view-, light- and time-image interpolation. ACM Transactions on Graphics, 39(6): #257[DOI:10.1145/3414685.3417827]
Chen W Z, Gao J, Ling H, Smith E J, Lehtinen J, Jacobson A and Fidler S. 2019. Learning to predict 3D objects with an interpolation-based differentiable renderer//Neural Information Processing Systems. Vancouver, Canada: [s.n.]: #8953765
Chen Z, Chen A P, Zhang G L, Wang C Y, Ji Y, Kutulakos K N and Yu J Y. 2020. A neural rendering framework for free-viewpoint relighting//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, USA: IEEE: 5598-5609[ DOI: 10.1109/CVPR42600.2020.00564 http://dx.doi.org/10.1109/CVPR42600.2020.00564 ]
Chen Z Q and Zhang H. 2019. Learning implicit fields for generative shape modeling//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 5932-5941[ DOI: 10.1109/CVPR.2019.00609 http://dx.doi.org/10.1109/CVPR.2019.00609 ]
Deschaintre V, Aittala M, Durand F, Drettakis G and Bousseau A. 2018. Single-image SVBRDF capture with a rendering-aware deep network. ACM Transactions on Graphics, 37(4): #128[DOI:10.1145/3197517.3201378]
Deschaintre V, Aittala M, Durand F, Drettakis G and Bousseau A. 2019. Flexible SVBRDF capture with a multi-image deep network. Computer Graphics Forum, 38(4): 1-13[DOI:10.1111/cgf.13765]
Dong Y. 2019. Deep appearance modeling: a survey. Visual Informatics, 3(2): 59-68[DOI:10.1016/j.visinf.2019.07.003]
Gao D, Chen G J, Dong Y, Peers P, Xu K and Tong X. 2020. Deferred neural lighting: free-viewpoint relighting from unstructured photographs. ACM Transactions on Graphics, 39(6): #258[DOI:10.1145/3414685.3417767]
Gao D, Li X, Dong Y, Peers P, Xu K and Tong X. 2019. Deep inverse rendering for high-resolution SVBRDF estimation from an arbitrary number of images. ACM Transactions on Graphics, 38(4): #134[DOI:10.1145/3306346.3323042]
Han Z Z, Chen C, Liu Y S and Zwicker M. 2020. DRWR: a differentiable renderer without rendering for unsupervised 3D structure learning from silhouette images//Proceedings of the 37th International Conference on Machine Learning (ICML). Virtual: [s.n.]
Hart J C. 1996. Sphere tracing: a geometric method for the antialiased ray tracing of implicit surfaces. The Visual Computer, 12(10): 527-545[DOI:10.1007/s003710050084]
Insafutdinov E and Dosovitskiy A. 2018. Unsupervised learning of shape and pose with differentiable point clouds//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Montréal, Canada: Curran Associates Inc. : 2807-2817
Jiang Y, Ji D T, Han Z Z and Zwicker M. 2020. SDFDiff: differentiable rendering of signed distance fields for 3D shape optimization//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 1248-1258[ DOI: 10.1109/CVPR42600.2020.00133 http://dx.doi.org/10.1109/CVPR42600.2020.00133 ]
Kajiya J T. 1986. The rendering equation//Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques. Virtual Event: Association for Computing Machinery: 143-150[DOI:10.1145/15922.15902]
Kang K Z, Chen Z M, Wang J P, Zhou K and Wu H Z. 2018. Efficient reflectance capture using an autoencoder. ACM Transactions on Graphics, 37(4): #127[DOI:10.1145/3197517.3201279]
Kang K Z, Xie C H, He C G, Yi M Q, Gu M Y, Chen Z M, Zhou K and Wu H Z. 2019. Learning efficient illumination multiplexing for joint capture of reflectance and shape. ACM Transactions on Graphics, 38(6): #165[DOI:10.1145/3355089.3356492]
Kato H, Ushiku Y and Harada T. 2018. Neural 3D mesh renderer//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 3907-3916[ DOI: 10.1109/CVPR.2018.00411 http://dx.doi.org/10.1109/CVPR.2018.00411 ]
Kim K, Gu J W, Tyree S, Molchanov P, Nieβner M and Kautz J. 2017. A lightweight approach for on-the-fly reflectance estimation//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE: 20-28[ DOI: 10.1109/ICCV.2017.12 http://dx.doi.org/10.1109/ICCV.2017.12 ]
Leng K X. 2020. Research on Generation of Adversarial Examples Based on Graphics. Chengdu: University of Electronic Science and Technology of China
冷凯轩. 2020. 基于图形的对抗样本生成技术研究. 成都: 电子科技大学
Li T M, Aittala M, Durand F and Lehtinen J. 2018a. Differentiable Monte Carlo ray tracing through edge sampling. ACM Transactions on Graphics, 37(6): #222[DOI:10.1145/3272127.3275109]
Li T M, Lehtinen J, Ramamoorthi R, Jakob W and Durand F. 2015. Anisotropic Gaussian mutations for metropolis light transport through Hessian-Hamiltonian dynamics. ACM Transactions on Graphics, 34(6): #209[DOI:10.1145/2816795.2818084]
Li X, Dong Y, Peers P and Tong X. 2017. Modeling surface appearance from a single photograph using self-augmented convolutional neural networks. ACM Transactions on Graphics, 36(4): #45[DOI:10.1145/3072959.3073641]
Li Z Q, Sunkavalli K and Chandraker M. 2018b. Materials for masses: SVBRDF acquisition with a single mobile phone image//Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer: 74-90[ DOI: 10.1007/978-3-030-01219-9_5 http://dx.doi.org/10.1007/978-3-030-01219-9_5 ]
Li Z Q, Xu Z X, Ramamoorthi R, Sunkavalli K and Chandraker M. 2018c. Learning to reconstruct shape and spatially-varying reflectance from a single image. ACM Transactions on Graphics, 37(6): #269[DOI:10.1145/3272127.3275055]
Lin C H, Kong C and Lucey S. 2018. Learning efficient point cloud generation for dense 3D object reconstruction//Proceedings of the 32nd AAAI Conference on Artificial Intelligence. New Orleans, USA: AAAI
Lin H L, Wang C Y and Lucey S. 2020. SDF-SRN: learning signed distance 3D object reconstruction from static images//Proceedings of the 34th Annual Conference on Neural Information Processing Systems. Vancouver, Canada: [s.n.]
Liu L J, Gu J T, Lin K Z, Chua T S and Theobalt C. 2020a. Neural sparse voxel fields//Proceedings of the 34th Conference on Neural Information Processing Systems. Vancouver, Canada: [s.n.]
Liu S C, Chen W K, Li T Y and Li H. 2019a. Soft rasterizer: a differentiable renderer for image-based 3D reasoning//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul, Korea (South): IEEE: 7707-7716[ DOI: 10.1109/ICCV.2019.00780 http://dx.doi.org/10.1109/ICCV.2019.00780 ]
Liu S C, Saito S, Chen W K and Li H. 2019b. Learning to infer implicit surfaces without 3D supervision//Proceedings of the 33rd Conference on Information Processing Systems. Vancouver, Canada: [s.n.]
Liu S H, Zhang Y D, Peng S Y, Shi B X, Pollefeys M and Cui Z P. 2020b. DIST: rendering deep implicit signed distance function with differentiable sphere tracing//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 2016-2025[ DOI: 10.1109/CVPR42600.2020.00209 http://dx.doi.org/10.1109/CVPR42600.2020.00209 ]
Lombardi S, Simon T, Saragih J, Schwartz G, Lehrmann A and Sheikh Y. 2019. Neural volumes: learning dynamic renderable volumes from images. ACM Transactions on Graphics, 38(4): #65[DOI:10.1145/3306346.3323020]
Loper M M and Black M J. 2014. OpenDR: an approximate differentiable renderer//Proceedings of the 13th European Conference on Computer Vision. Zurich, Switzerland: Springer: 154-169[ DOI: 10.1007/978-3-319-10584-0_11 http://dx.doi.org/10.1007/978-3-319-10584-0_11 ]
Loubet G, Holzschuch N and Jakob. 2019. Reparameterizing discontinuous integrands for differentiable rendering. ACM Transactions on Graphics, 38(6): #14228[DOI:10.1145/3355089.3356510]
Luan F J, Zhao S, Bala K and Gkioulekas I. 2020. Langevin Monte Carlo rendering with gradient-based adaptation. ACM Transactions on Graphics, 39(4): #140[DOI:10.1145/3386569.3392382]
Lyu J H, Wu B J, Lischinski D, Cohen-Or D and Huang H. 2020. Differentiable refraction-tracing for mesh reconstruction of transparent objects. ACM Transactions on Graphics, 39(6): #195[DOI:10.1145/3414685.3417815]
Meka A, Häne C, Pandey R, Zollhöfer M, Fanello S, Fyffe G, Kowdle A, Yu X M, Busch J, Dourgarian J, Denny P, Bouaziz S, Lincoln P, Whalen M, Harvey G, Taylor J, Izadi S, Tagliasacchi A, Debevec P, Theobalt C, Valentin J and Rhemann C. 2019. Deep reflectance fields: high-quality facial reflectance field inference from color gradient illumination. ACM Transactions on Graphics, 38(4): #77[DOI:10.1145/3306346.3323027]
Mescheder L, Oechsle M, Niemeyer M, Nowozin S and Geiger A. 2019. Occupancy networks: learning 3D reconstruction in function space//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 4455-4465[ DOI: 10.1109/CVPR.2019.00459 http://dx.doi.org/10.1109/CVPR.2019.00459 ]
Mildenhall B, Srinivasan P P, Tancik M, Barron J T, Ramamoorthi R and Ng R. 2020. NeRF: representing scenes as neural radiance fields for view synthesis//Proceedings of the 16th European Conference on Computer Vision. Glasgow, Scotland: Springer: 405-421[ DOI: 10.1007/978-3-030-58452-8_24 http://dx.doi.org/10.1007/978-3-030-58452-8_24 ]
Navaneet K L, Mandikal P, Agarwal M and Babu R V. 2019. CAPNet: continuous approximation projection for 3D point cloud reconstruction using 2D supervision//Proceedings of the 33rd AAAI Conference on Artificial Intelligence. Hawaii, USA: AAAI: 8819-8826
Nguyen-Phuoc T, Li C, Balaban S and Yang Y L. 2018. RenderNet: a deep convolutional network for differentiable rendering from 3D shapes//Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS). Montréal, Canada: [s.n.]
Niemeyer M, Mescheder L, Oechsle M and Geiger A. 2020. Differentiable volumetric rendering: learning implicit 3D representations without 3d supervision//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 3501-3512[ DOI: 10.1109/CVPR42600.2020.00356 http://dx.doi.org/10.1109/CVPR42600.2020.00356 ]
Nimier-David M, Speierer S, Ruiz B and Jakob W. 2020. Radiative backpropagation: an adjoint method for lightning-fast differentiable rendering. ACM Transactions on Graphics, 39(4): #146[DOI:10.1145/3386569.3392406]
Nimier-David M, Vicini D, Zeltner T and Jakob W. 2019. Mitsuba 2: a retargetable forward and inverse renderer. ACM Transactions on Graphics, 38(6): #203[DOI:10.1145/3355089.3356498]
Park J J, Florence P, Straub J, Newcombe R and Lovegrove S. 2019. DeepSDF: learning continuous signed distance functions for shape representation//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 165-174[ DOI: 10.1109/CVPR.2019.00025 http://dx.doi.org/10.1109/CVPR.2019.00025 ]
Rezende D J, Eslami S M A, Mohamed S, Battaglia P, Jaderberg M and Heess N. 2016. Unsupervised learning of 3D structure from images//Proceedings of the 30th International Conference on Neural Information Processing Systems. Barcelona, Spain: Curran Associates Inc. : 5003-5011
Shum H Y, Chan S C and Kang S B. 2007. Image-Based Rendering. Boston: Springer[ DOI: 10.1007/978-0-387-32668-9 http://dx.doi.org/10.1007/978-0-387-32668-9 ]
Sitzmann V, Thies J, Heide F, Nieβner M, Wetzstein G and Zollhöfer M. 2019a. DeepVoxels: learning persistent 3D feature embeddings//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE: 2432-2441[ DOI: 10.1109/CVPR.2019.00254 http://dx.doi.org/10.1109/CVPR.2019.00254 ]
Sitzmann V, Zollhöfer M and Wetzstein G. 2019b. Scene representation networks: continuous 3D-structure-aware neural scene representations//Proceedings of the 33rd Conference on Neural Information Processing Systems. Vancouver, Canada: [s.n.]
Thies J, Zollhöfer M and Nieβner M. 2019. Deferred neural rendering: image synthesis using neural textures. ACM Transactions on Graphics, 38(4): #66[DOI:10.1145/3306346.3323035]
Tulsiani S, Zhou T H, Efros A A and Malik J. 2017. Multi-view supervision for single-view reconstruction via differentiable ray consistency//Proceedingsof 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 209-217[ DOI: 10.1109/CVPR.2017.30 http://dx.doi.org/10.1109/CVPR.2017.30 ]
Veach E. 1997. Robust Monte Carlo Methods for Light Transport Simulation. Stanford: Stanford University
Vu H H, Labatut P, Pons J P and Keriven R. 2012. High accuracy and visibility-consistent dense multiview stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(5): 889-901[DOI:10.1109/TPAMI.2011.172]
Wang Y F, Serena F, Wu S H, Öztireli C and Sorkine-Hornung O. 2019. Differentiable surface splatting for point-based geometry processing. ACM Transactions on Graphics, 38(6): #230[DOI:10.1145/3355089.3356513]
Wu B J, Zhou Y, Qian Y M, Cong M L and Huang H. 2018. Full 3D reconstruction of transparent objects. ACM Transactions on Graphics, 37(4): #103[DOI:10.1145/3197517.3201286]
Wu J J, Wang Y F, Xue T F, Sun X Y, Freeman W T and Tenenbaum J B. 2017. MarrNet: 3D shape reconstruction via 2.5D sketches//Proceedings of the 31st International Conference on Neural Information Processing Systems. Los Angeles, USA: Curran Associates Inc. : 540-550
Xu Z X, Sunkavalli K, Hadap S and Ramamoorthi R. 2018. Deep image-based relighting from optimal sparse samples. ACM Transactions on Graphics, 37(4): #126[DOI:10.1145/3197517.3201313]
Yan X C, Yang J M, Yumer E, Guo Y J and Lee H. 2016. Perspective transformer nets: learning single-view 3D object reconstruction without 3D supervision//Proceedings of the 30th International Conference on Neural Information Processing Systems. Barcelona, Spain: Curran Associates Inc. : 1704-1712
Yariv L, Kasten Y, Moran D, Galun M, Atzmon M, Basri R and Lipman Y, 2020. Multiview neural surface reconstruction with implicit lighting and material[EB/OL]. [2020-12-01] . https://arxiv.org/pdf/2003.09852v2.pdf https://arxiv.org/pdf/2003.09852v2.pdf
Zhang C, Miller B, Yan K, Gkioulekas I and Zhao S. 2020. Path-space differentiable rendering. ACM Transactions on Graphics, 39(4): #143[DOI:10.1145/3386569.3392383]
Zhang C, Wu L F, Zheng C X, Gkioulekas I, Ramamoorthi R and Zhao S. 2019. A differential theory of radiative transfer. ACM Transactions on Graphics, 38(6): #227[DOI:10.1145/3355089.3356522]
Zhang R, Tsai P S, Cryer J E and Shah M. 1999. Shape-from-shading: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(8): 690-706[DOI:10.1109/34.784284]
Zhao S, Jakob W and Li T M. 2020. Physics-based differentiable rendering: from theory to implementation//ACM SIGGRAPH 2020 Courses. Virtual Event: Association for Computing Machinery: 14[ DOI: 10.1145/3388769.3407454 http://dx.doi.org/10.1145/3388769.3407454 ]
Zhu R, Galoogahi H K, Wang C Y and Lucey S. 2017.Rethinking reprojection: closing the loop for pose-aware shape reconstruction from a single image//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE: 57-65[ DOI: 10.1109/ICCV.2017.16 http://dx.doi.org/10.1109/ICCV.2017.16 ]
Zwicker M, Pfister H, van Baar J and Gross M. 2001. Surface splatting//Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. Virtual Event: Association for Computing Machinery: 371-378[ DOI: 10.1145/383259.383300 http://dx.doi.org/10.1145/383259.383300 ]
相关作者
相关机构
京公网安备11010802024621