Current Issue Cover
基于物理的可微渲染综述

邢健开, 徐昆(清华大学)

摘 要
可微渲染技术是当前计算机图形学与计算机视觉方向的研究热点,其目标是将计算机图形学中的渲染流水线进行可微化改造以支持计算渲染的输出图像关于输入参数如几何、材质等参数的梯度。结合渲染图像与目标图像之间的损失函数,可微渲染允许我们在生成式分析的框架中,通过梯度下降的方式从图像中推理出场景参数,是解决三维重建、逆向渲染等领域问题的有效方法,并在虚拟现实、自动驾驶等领域有着广泛的应用前景。基于物理的可微渲染旨在对基于物理的渲染管线进行可微化改造,主要涉及对场景几何和材质的表达,以及光路传输模拟过程的梯度计算方法。本文对近年来基于物理的可微渲染领域的发展情况进行了调研,总结了基于物理的可微渲染研究进展。本文分为五个部分:第一部分总体介绍正向渲染和可微渲染的计算方法;第二部分介绍如何针对具体的几何、材质、相机的表达方式进行梯度计算;第三部分讨论如何提高可微渲染的效率和鲁棒性;第四部分展示可微渲染如何应用于实际任务中;在第五部分中,本文将展望可微渲染的发展趋势,期望推动该领域的进一步发展。
关键词
Physically based Differentiable Rendering: A Survey

Xingjiankai, Xukun(Tsinghua University)

Abstract
Rendering has been a prominent subject in the field of computer graphics for an extended period. It can be regarded as a function that accepts an abstract scene description as input and typically generates a 2D image as output. The theory and practice of rendering have significantly advanced through years of research. In recent years, inverse rendering has emerged as a new research focus in the field of computer graphics due to the development of digital technology. The objective of inverse rendering is to reverse the rendering process and deduce scene parameters from the output image, which is equivalent to solving the inverse function of the rendering function. This plays a crucial role in addressing perception problems in diverse advanced technological domains including virtual reality, autonomous driving, and robotics. Numerous methods exist for implementing inverse rendering, with the current mainstream framework being optimization through “analysis by synthesis” process. Firstly, it estimates a set of initial scene parameters, then performs forward rendering on the scene, compares the rendered result with the target image, and then minimizes the difference (loss function) by optimizing the scene parameters using gradient descent based method. This pipeline necessitates the ability to compute the derivatives of the output image in forward rendering with respect to the input parameters. Consequently, differentiable rendering has emerged to fulfill this requirement. Specifically, the research topic of differentiable rendering is to convert the forward rendering pipeline in computer graphics into a differentiable form, enabling the differentiation of the output image with respect to input parameters like geometry, material, light source, and camera. Currently, forward rendering can be broadly categorized into three types: rasterization-based rendering, physically based rendering, and the emerging neural rendering. Rasterization-based rendering is a fundamental technique in computer graphics that converts geometric shapes into pixels for display. It involves projecting 3D objects onto a 2D screen, performing hidden surface removal, shading, and texturing to create realistic images efficiently. While rasterization is fast and suitable for real-time applications, it may lack physical accuracy in simulating light interactions. Physically based rendering, on the other hand, aims to accurately simulate real-world light behavior by considering physical properties of materials, light sources, and the environment. It calculates how light rays interact with surfaces, accounting for reflections, refractions, and scattering to produce photorealistic visual results. This method prioritizes realism and is widely used in industries such as animation, gaming, and visual effects. Neural rendering is an emerging rendering technique in recent years, mainly used for image-based rendering tasks. Unlike traditional graphics rendering, image-based rendering does not require any explicit 3D scene information (geometry, materials, lighting, etc.), but instead implicitly encodes scenes through a sequence of 2D images sampled from different viewpoints, enabling the generation of images of the scene from any viewpoint. Accordingly, differentiable rendering can also be categorized into three types: differentiable rasterization, physically based differentiable rendering, and differentiable neural rendering. In differentiable rasterization, many works employ approximate methods to compute approximate derivatives of the rasterization process for backpropagation of gradients, or modify steps in the traditional rendering pipeline (usually rasterization and testing/blending steps) to make pixels differentiable with respect to vertices. Neural rendering is naturally differentiable because its rendering process is conducted through neural networks. For physically based differentiable rendering, because of the intricate nature of geometry, material, and light transmission processes, accurately calculating the gradient of the image concerning scene parameters is more challenging. Therefore, this article concentrates on recent research in the field of physically based differentiable rendering. The article is organized into the following sections: Chapter 1 introduces the computational methods of forward rendering and differentiable rendering from an abstract standpoint and two types of method for correctly computing boundary integral: edge sampling and reparameterization. Chapter 2 explores the attainment of differentiable rendering for distinct representations of geometry like volumetric representation, signed distance field, height field and vectorized geometry, material like volumetric material, parameterized BRDF(bidirectional reflectance distribution function), BSSRDF(bidirectional surface scattering reflectance distribution function) and continuously varying refractive index fields, and camera related parameters like pixel reconstruction filter and TOF(time of flight) camera. Chapter 3 focuses on enhancing the efficiency and robustness of differentiable rendering, including efficiently sampling, high efficiency system & framework & language for differentiable rendering and several techniques to enhance the robustness of differentiable rendering. Chapter 4 showcases the application of differentiable rendering in practical tasks, which can be generally divided in three types: single object reconstruction, object and environment light reconstruction and scene reconstruction. Lastly in Chapter 5, the future development trends of differentiable rendering are discussed, including improving efficiency, robustness of differentiable rendering and combining differentiable rendering with other methods.
Keywords

订阅号|日报