Current Issue Cover
可微绘制技术研究进展

许威威1, 周漾2, 吴鸿智1, 过洁3(1.浙江大学CAD&CG国家重点实验室, 杭州 310058;2.深圳大学, 深圳 518057;3.南京大学, 南京 210093)

摘 要
可微绘制技术是当前虚拟现实、计算机图形学与计算机视觉领域研究的热点,其目标是改造计算机图形学中以光栅化或光线跟踪算法为主的真实感绘制流程,支持梯度信息回传以计算由输出图像的变化导致的输入几何、材质属性变化,通过与优化及深度学习技术等相结合支持从数据中学习绘制模型和逆向推理,是可微学习技术在计算机图形学绘制技术中的应用的具体体现,在增强/虚拟现实内容生成、三维重建、表观采集建模和逆向光学设计等领域中有广泛的应用前景。本文对可微绘制当前的发展状况进行调研,重点对该技术在真实感绘制、3维重建和表观采集建模中的研究和应用情况进行综述,并对可微绘制技术发展趋势进行展望,以期推动可微技术在学术界和产业界的进一步发展。
关键词
Differential rendering: a survey

Xu Weiwei1, Zhou Yang2, Wu Hongzhi1, Guo Jie3(1.State Key Laboratory of CAD&CG, Zhejiang University, Hangzhou 310058, China;2.Shenzhen University, Shenzhen 518057, China;3.Nanjing University, Nanjing 210093, China)

Abstract
Differential rendering is currently a research focus in virtual reality, computer graphics, and computer vision. Its goal is to reform the rendering pipeline in computer graphics to support gradient backpropagation such that the change in the output image can be related to the change in input geometry or materials. The development of differential rendering technique is highly related to the deep learning, since neural networks are usually represented as computational graphs to support gradient backpropagation using the chain rule. Thus, the gradient backpropagation is the key to convert a computational procedure into a learnable process, which can significantly generalize the deep learning technique to a wide range of applications. Differential rendering follows this trend to integrate gradient backpropagation into rendering pipeline. It can significantly facilitate the gradient computation through auto-differential techniques. In fact, the derivatives of rendering results regarding to the mesh vertex have already computed in variational 3D reconstruction and shape from shading. However, differential rendering integrates the derivative computation into global rendering pipelines and neural networks. Therefore, the rendering process can be directly integrated into optimization or neural network training to approximate rendering pipeline or inverse graphics reasoning; it has wide applications in content creation in augmented/virtual reality, 3D reconstruction, appearance modeling, and inverse design. The advantage of differential rendering over traditional rendering pipeline is that it allows to train neural networks to approximate the forward rendering pipeline. Once trained, the rendering results can be obtained through forward inference of the network, a much faster procedure in many situations. Moreover, the gradient information provided by differential rendering is helpful to improve the efficiency of the global rendering. For instance, the first- and second-order gradients can be used to guide the sampling process in Monte Carlo rendering. Another advantage of differential rendering is that it can directly be used in view interpolation or view synthesis through captured images, which traditional rendering pipeline needs geometry, appearance and lighting information simultaneously to render an image at specified viewpoints. In the application of differential rendering to view synthesis or image-based rendering, the implicit representation of a 3D scene is usually inferred from the captured images directly via deep neural networks supervised by differential rendering loss. Such a process falls into the category of self-supervised learning because ground truth 3D data are not provided during training. It bypasses the expensive multi-view 3D reconstruction and thus significantly simplifies the view synthesis procedure. Numerous representations, such as neural texture, neural volume, and neural implicit function, are proposed to handle freeview point rendering of a 3D scene. However, the training and rendering cost of these methods is still expensive. Thus, reducing their computational cost forms a new research direction. Differential rendering also enables the end-to-end inference of spatially variant bidirectional reflectance distribution function (BRDF) material properties from capture images. The BRDF parameters can be derived from a single image after training the deep neural network on a large amount of data by representing the material properties in a latent space. Moreover, with a differentiable pipeline, the layout of the light sources and projection patterns of dedicated appearance acquisition equipment can be optimized.The recent development of differential rendering, including its application in realistic rendering, 3D reconstruction, and appearance modeling is comprehensively surveyed. We expect this study to further boost the research on differential rendering in academia and industry.
Keywords

订阅号|日报