Current Issue Cover

邢健开, 徐昆(清华大学)

摘 要
Physically based Differentiable Rendering: A Survey

Xingjiankai, Xukun(Tsinghua University)

Rendering has been a prominent subject in the field of computer graphics for an extended period. It can be regarded as a function that accepts an abstract scene description as input and typically generates a 2D image as output. The theory and practice of rendering have significantly advanced through years of research. In recent years, inverse rendering has emerged as a new research focus in the field of computer graphics due to the development of digital technology. The objective of inverse rendering is to reverse the rendering process and deduce scene parameters from the output image, which is equivalent to solving the inverse function of the rendering function. This plays a crucial role in addressing perception problems in diverse advanced technological domains including virtual reality, autonomous driving, and robotics. Numerous methods exist for implementing inverse rendering, with the current mainstream framework being optimization through “analysis by synthesis” process. Firstly, it estimates a set of initial scene parameters, then performs forward rendering on the scene, compares the rendered result with the target image, and then minimizes the difference (loss function) by optimizing the scene parameters using gradient descent based method. This pipeline necessitates the ability to compute the derivatives of the output image in forward rendering with respect to the input parameters. Consequently, differentiable rendering has emerged to fulfill this requirement. Specifically, the research topic of differentiable rendering is to convert the forward rendering pipeline in computer graphics into a differentiable form, enabling the differentiation of the output image with respect to input parameters like geometry, material, light source, and camera. Currently, forward rendering can be broadly categorized into three types: rasterization-based rendering, physically based rendering, and the emerging neural rendering. Rasterization-based rendering is a fundamental technique in computer graphics that converts geometric shapes into pixels for display. It involves projecting 3D objects onto a 2D screen, performing hidden surface removal, shading, and texturing to create realistic images efficiently. While rasterization is fast and suitable for real-time applications, it may lack physical accuracy in simulating light interactions. Physically based rendering, on the other hand, aims to accurately simulate real-world light behavior by considering physical properties of materials, light sources, and the environment. It calculates how light rays interact with surfaces, accounting for reflections, refractions, and scattering to produce photorealistic visual results. This method prioritizes realism and is widely used in industries such as animation, gaming, and visual effects. Neural rendering is an emerging rendering technique in recent years, mainly used for image-based rendering tasks. Unlike traditional graphics rendering, image-based rendering does not require any explicit 3D scene information (geometry, materials, lighting, etc.), but instead implicitly encodes scenes through a sequence of 2D images sampled from different viewpoints, enabling the generation of images of the scene from any viewpoint. Accordingly, differentiable rendering can also be categorized into three types: differentiable rasterization, physically based differentiable rendering, and differentiable neural rendering. In differentiable rasterization, many works employ approximate methods to compute approximate derivatives of the rasterization process for backpropagation of gradients, or modify steps in the traditional rendering pipeline (usually rasterization and testing/blending steps) to make pixels differentiable with respect to vertices. Neural rendering is naturally differentiable because its rendering process is conducted through neural networks. For physically based differentiable rendering, because of the intricate nature of geometry, material, and light transmission processes, accurately calculating the gradient of the image concerning scene parameters is more challenging. Therefore, this article concentrates on recent research in the field of physically based differentiable rendering. The article is organized into the following sections: Chapter 1 introduces the computational methods of forward rendering and differentiable rendering from an abstract standpoint and two types of method for correctly computing boundary integral: edge sampling and reparameterization. Chapter 2 explores the attainment of differentiable rendering for distinct representations of geometry like volumetric representation, signed distance field, height field and vectorized geometry, material like volumetric material, parameterized BRDF(bidirectional reflectance distribution function), BSSRDF(bidirectional surface scattering reflectance distribution function) and continuously varying refractive index fields, and camera related parameters like pixel reconstruction filter and TOF(time of flight) camera. Chapter 3 focuses on enhancing the efficiency and robustness of differentiable rendering, including efficiently sampling, high efficiency system & framework & language for differentiable rendering and several techniques to enhance the robustness of differentiable rendering. Chapter 4 showcases the application of differentiable rendering in practical tasks, which can be generally divided in three types: single object reconstruction, object and environment light reconstruction and scene reconstruction. Lastly in Chapter 5, the future development trends of differentiable rendering are discussed, including improving efficiency, robustness of differentiable rendering and combining differentiable rendering with other methods.