发布时间: 2021-11-16 摘要点击次数: 全文下载次数: DOI: 10.11834/jig.200475 2021 | Volume 26 | Number 11 计算机图形学

 收稿日期: 2020-09-07; 修回日期: 2021-01-03; 预印本日期: 2021-01-10 基金项目: 国家自然科学基金项目（61702422）；中央高校基本科研业务费专项资金资助（2452018146） 作者简介: 任非儿, 1996年生, 女, 硕士研究生, 主要研究方向为计算机图形学。E-mail: renfir@nwafu.edu.cn 刘通, 男, 硕士研究生, 主要研究方向为计算机图形学。E-mail: tliu@nwafu.edu.cn 杨龙, 通信作者, 男, 副教授, 主要研究方向为计算机图形学、数字几何处理和3D计算机视觉。E-mail: yl@nwafu.edu.cn *通信作者: 杨龙  yl@nwafu.edu.cn 中图法分类号: TP399 文献标识码: A 文章编号: 1006-8961(2021)11-2713-10

# 关键词

3维重建; 从明暗恢复形状(SFS); 单幅图像; 图像骨架; 图像亮度统计; 距离场约束

3D reconstruction of a single plant leaf image
Ren Feier, Liu Tong, Yang Long
College of Information Engineering, Northwest A & F University, Yangling 712100, China
Supported by: National Natural Science Foundation of China (61702422);Fundamental Research Funds for the Central Universities (2452018146)

# Abstract

Objective In recent years, with the continuous improvement of computer hardware performance and the continuous in depth research of virtual plant modeling and landscape design in the fields of film and television and games, plant shape modeling has become possible and necessary. As one of the important organs of plants, leaves have complex physiological and morphological characteristics, which are difficult to represent in virtual scenes. Therefore, 3D reconstruction of plant leaves is a very challenging subject. Current 3D reconstruction methods have two main types. 1) External equipment such as laser or radar are used to measure the distance of the target by transmitting signals and then receiving the signals reflected by the target object to obtain the 3D shape of the surface. 2) Using the principle of binocular vision, two or more images are obtained from different perspectives in the same scene, then the 3D shape of the object is obtained according to the parallax between the images. The former requires the introduction of external equipment, which is costly and difficult to operate, whereas the latter needs to detect and match the feature points between the acquired image sequences due to the reconstruction based on multiple images. The acquisition of a single image is simple, and the problem of feature matching of multiple images is eliminated. However, due to the less information contained in a single image and to recover the 3D shape of plant leaves from the limited information, this paper is based on the method of the shape from shading (SFS) and preprocesses the image to add a priori to the 3D shape estimation. The brightness statistic information obtained from the image and the prior information of plant morphological characteristics are used to recover the final 3D shape of the leaf. Method When restoring the 3D shape of a single plant leaf image, the restoration of the surface shape is divided into two aspects: surface detail and surface macroscopic geometric shape. First, based on SFS, a distance field offset algorithm is designed according to the image skeleton to enhance the surface details of 3D shapes. The edge detection method is used to detect the leaf veins as the skeleton of the image, and the distance from the image skeleton is used as the constraint of the SFS minimization method to enhance the surface detail display. Then, to address the deficiencies of SFS in recovering the macroscopic geometric shape, various factors that affect the macroscopic geometric shape are considered, and the characteristics of leaf surface unevenness and curvature are finally realized. Selecting control points according to the statistical distribution of image brightness is proposed to control the change of the surface macroscopic geometric shape, and the distance field constraint of the blade central axis is used to restore the macroscopic geometric shape. According to the brightness statistics, the image is divided into bright-dark areas, the centroid of the unconnected areas of the bright and dark regions is used as the control point, and the cubic Bezier surface is used to generate the concave and convex characteristics of the blade surface. The overall bending of the blade can be estimated based on the distance from each point of the blade to the central axis because most blades have a certain degree of bending about the central axis and the curvature at the central axis changes the most. The two weights for the restoration of the surface macroscopic geometric shape are set based on the similarity between the restored reflection map and the input image, and the surface details are finally added to the macroscopic geometric shape to obtain the final 3D shape of the target object. Result Plant leaf images are selected for experiments, and their 3D restoration results are compared with those of other methods (including Tsai linear approach, Zheng minimization approach, SIRFS(shape, illumination, and reflectance from shading), and variational approach). Experimental results show that the method proposed enhances the display of surface details and has evident changes in macroscopic geometric shape. To verify the applicability of the method in the restoration of object surface details, namely, recovering the surface details of coins and dinosaurs, the experimental results prove that the proposed method of enhancing surface details is also applicable to other objects. In addition, using the ratio of error to information entropy is proposed to describe the effect of 3D reconstruction of the target. Error describes the accuracy of restoration, and information entropy describes the richness of information. The larger the entropy is, the greater the difference between the depth of restoration, which means larger surface macroscopic geometry changes. When the error is smaller and the information entropy is larger, the ratio of error to information entropy is smaller and the recovery effect is better. Conclusion To address the 3D reconstruction problem of a single plant leaf image, decomposing the problem into two aspects is proposed: surface details and macroscopic geometric shape. Based on SFS, the surface details are enhanced according to the skeleton feature, the surface macroscopic geometric shape is jointly restored using the statistical distribution of image brightness and the axial distance field constraints of the leaves, the final 3D shape is obtained by combining the surface details and the macroscopic geometric shape, and the feasibility of the proposed method is verified by multiple sets of experiments.

# Key words

3D restoration; shape from shading(SFS); single image; image skeleton; image brightness statistics; distance field constraint

# 1 研究基础

SFS从2维图像推导3维形状的原理是当入射光线照射到物体表面，表面会吸收或反射部分光，被表面反射的光进入相机镜头，在底片上成像，因此，图像亮度与光源、物体表面特性和表面形状相关，由于亮度图能获取的只有像素点的位置和亮度值，根据该反射原理，SFS的反射函数模型为

 $\begin{gathered} E(x, y) \approx R(p, q)= \\ \rho \frac{1+p \times p_{s}+q \times q_{s}}{\sqrt{p^{2}+q^{2}+1} \times \sqrt{p_{s}^{2}+q_{s}^{2}+1}} \end{gathered}$ (1)

 $p=\frac{\partial Z(x, y)}{\partial x}$ (2)

 $q=\frac{\partial Z(x, y)}{\partial y}$ (3)

SFS算法主要分为演化方法、局部方法、最小化方法和线性化方法(廖熠和赵荣椿，2001)。演化方法是从已知形状或者可以求得形状的点出发，向周围扩展，演化出整个表面。局部方法将局部表面的形状假设与反射模型相结合，构造线性偏微分方程组进行求解。最小化方法是将反射模型和其他约束条件构造为能量函数的形式，转化为最优化问题，对该问题求最小值，引入其他约束条件是为了限制解的范围。线性化方法通过线性化反射函数，将非线性问题简化为线性问题，便于求解。演化方法中，图像的奇点易与图像噪声混淆；局部方法的假设对自然表面很难满足(廖熠和赵荣椿，2001)。

SFS的线性化方法首先将函数$f$关于深度$Z$的线性逼近用泰勒级数展开，然后利用雅可比迭代求解，简化后可得

 $\begin{gathered} 0 \approx f(Z(x, y)) \approx f\left(Z^{n-1}(x, y)\right)+Z(x, y)- \\ Z^{n-1}(x, y) \frac{\mathrm{d} f\left(Z^{n-1}(x, y)\right)}{\mathrm{d} Z(x, y)} \end{gathered}$ (4)

 $\left.Z^{n}(x, y)\right)=Z^{n-1}(x, y)-\frac{\mathrm{d} f\left(Z^{n-1}(x, y)\right)}{\mathrm{d} Z(x, y)}$ (5)

SFS的最小化方法是通过构造包含亮度约束和其他约束的能量函数并使其最小化，从而求解出深度值。为了减少实际图像中噪声对结果的影响，约束方程引入光强误差作为亮度约束，具体为

 $E_{1}=\iint(I-R)^{2} \mathrm{~d} x \mathrm{d} y$ (6)

 $e=E_{1}+\lambda \times E_{2}+\mu \times E_{3}$ (7)

SFS的线性化方法计算简单，但由于该方法将反射函数线性化，导致最终结果只是对真正解的一种近似，始终存在系统误差，且受图像质量影响较大；最小化方法的鲁棒性相对较高，在理论上可以无限逼近正解(廖熠和赵荣椿，2001)。两种方法对表面细节的恢复较好，但几乎没有宏观几何形状的起伏，对全局重建效果不佳(苗旺，2020)。

# 2.2 2维图像推导3维形状

SFS恢复的3维形状有较清晰的表面细节，但是几乎没有明显的宏观几何形状表现。在图像预处理的基础上，利用图像骨架特征作为约束条件提升表面细节的恢复效果，同时，基于叶片在中轴处弯折的特性和图像亮度分布规律对宏观几何形状进行恢复。

# 2.2.1 表面细节恢复

SFS的最小化方法鲁棒性高，但相较于线性化方法求解速度慢，线性化方法的求解速度快，但易受图像噪声的影响(须明等，2004)，因此，将线性化方法与最小化方法结合，在图像预处理结果的基础上，利用线性化方法获得最小化方法的初始值。在已知光源方向和表面反射率等信息的条件下，先利用线性化方法初步恢复出物体的深度值，以该值作为最小化方法的初始值，迭代求解出最优深度值。

 $E_{2}=\iint\left(p_{x}^{2}+p_{y}^{2}+q_{x}^{2}+q_{y}^{2}\right) \mathrm{d} x \mathrm{d} y$ (8)

 $Z_{\mathrm{H}}=\lambda_{\mathrm{b}} \times Z_{\mathrm{b}}+\lambda_{\mathrm{mDC}} \times Z_{\mathrm{mDC}}$ (14)

 $Z=Z_{\mathrm{b}}+Z_{\mathrm{H}}$ (15)

# 3 实验结果与讨论

 $e r r=\frac{\sum\limits_{i=1, j=1}^{M, N}\left|R_{i, j}-I_{i, j}\right|}{M \times N \sum\limits_{Z=0}^{Z_{\max }}\left|P(Z) \log _{2} P(Z)\right|}$ (16)

Table 1 Error-Information entropy ratio of 3D reconstruction of leaves by different methods

 方法 叶片1 叶片2 叶片3 叶片4 叶片5 Tsai线性化(Tsai和Shah, 1994) 0.282 1 0.143 7 0.217 0 0.111 6 0.249 0 Zheng最小化(Zheng和Chellappa, 1991) 0.226 8 0.262 4 0.383 5 0.160 3 0.356 2 SIRFS(Barron和Malik, 2015) 0.107 1 0.141 0 0.124 2 0.152 4 0.283 5 变分法(Quéau等, 2017) 0.285 4 0.136 0 0.117 8 0.064 9 0.245 9 本文控制点 0.055 5 0.178 0 0.064 5 0.015 0 0.175 5 本文 0.053 2 0.085 0 0.092 7 0.082 6 0.191 4 注：加粗字体表示各列最优结果。

Table 2 Error-Information entropy ratio of 3D reconstruction of other objects by different methods

 方法 硬币 恐龙 Tsai线性化(Tsai和Shah, 1994) 0.028 9 0.146 6 Zheng最小化(Zheng和Chellappa, 1991) 0.330 1 0.335 2 本文 0.023 5 0.143 1 注：加粗字体表示各列最优结果。

# 参考文献

• Barron J T, Malik J. 2015. Shape, illumination, and reflectance from shading. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(8): 1670-1687 [DOI:10.1109/TPAMI.2014.2377712]
• Durou J D, Falcone M, Sagona M. 2008. Numerical methods for shape-from-shading: a new survey with benchmarks. Computer Vision and Image Understanding, 109(1): 22-43 [DOI:10.1016/j.cviu.2007.09.003]
• Guo Y B. 2014. Research on Intrinsic Image Decomposition Algorithm and Its Applications. Xiamen: Xiamen University (郭银斌. 2014. 本征图像分解算法研究及其应用. 厦门: 厦门大学)
• Horn B K P. 1970. Shape from Shading: A Method for Obtaining the Shape of a Smooth Opaque Object from One View. Technical Report 232. Massachusetts Institute of Technology: 1-198
• Hu Z Y, Liang F Z, Zhang X F, Zhou J. 2007. The current situation and trends of shape from shading. Journal of Zhengzhou University of Light Industry(Natural Science), 22(2/3): 182-185 (胡志勇, 梁发周, 张秀芬, 周洁. 2007. 明暗恢复形状技术研究进展. 郑州轻工业学院学报(自然科学版), 22(2/3): 182-185) [DOI:10.3969/j.issn.1004-1478.2007.02.053]
• Huang J, Wang C, Liu Y, Bi T T. 2019. The progress of monocular depth estimation technology. Journal of Image and Graphics, 24(12): 2081-2097 (黄军, 王聪, 刘越, 毕天腾. 2019. 单目深度估计技术进展综述. 中国图象图形学报, 24(12): 2081-2097) [DOI:10.11834/jig.190455]
• Li S Q, Zhu Y P, Liu H L, Li S J, Liu D, Zhang H Y, Li Z Y. 2016. Research and realization of wheat leaf three-dimensional visualization based on NURBS surface. Journal of Agricultural Science and Technology, 18(3): 89-95 (李书钦, 诸叶平, 刘海龙, 李世娟, 刘丹, 张红英, 李中阳. 2016. 基于NURBS曲面的小麦叶片3维可视化研究与实现. 中国农业科技导报, 18(3): 89-95) [DOI:10.13304/j.nykjdb.2015.641]
• Liao Y, Zhao R C. 2001. Analysis and evaluation of several typical SFS algorithms. Journal of Image and Graphics, 6(10): 953-961 (廖熠, 赵荣椿. 2001. 从明暗恢复形状(SFS)的几类典型算法分析与评价. 中国图象图形学报, 6(10): 953-961) [DOI:10.3969/j.issn.1006-8961.2001.10.002]
• Lu G R, Tang J F, Jiang M. 2014. Research on image highlight removal based on fast bilateral filter. Computer Engineering and Applications, 50(10): 176-179, 207 (卢桂荣, 汤景凡, 姜明. 2014. 基于快速双边滤波的图像高光去除研究. 计算机工程与应用, 50(10): 176-179, 207) [DOI:10.3778/j.issn.1002-8331.1310-0359]
• Lu S L, Guo X Y, Li C F. 2009. Research on techniques for accurate modeling and rendering 3D plant leaf. Journal of Image and Graphics, 14(4): 731-737 (陆声链, 郭新宇, 李长锋. 2009. 3维植物叶片精确建模和绘制技术研究. 中国图象图形学报, 14(4): 731-737) [DOI:10.11834/jig.20090426]
• Miao W. 2020. 3D Reconstruction of Soft Tissue Image Based on Binocular Endoscope. Chengdu: University of Electronic Science and Technology of China (苗旺. 2020. 基于双目内窥镜的软组织图像3维重建. 成都: 电子科技大学)
• Quéau Y, Mélou J, Castan F, Cremers D and Durou J D. 2017. A variational approach to shape-from-shading under natural illumination//Proceedings of the International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition. Venice, Italy: Springer: 342-357[DOI: 10.1007/978-3-319-78199-0_23]
• Sun Y Y. 2011. A survey of 3D reconstruction based on single image. Journal of North China University of Technology, 23(1): 9-13 (孙宇阳. 2011. 基于单幅图像的3维重建技术综述. 北方工业大学学报, 23(1): 9-13) [DOI:10.3969/j.issn.1001-5477.2011.01.002]
• Tsai P S, Shah M. 1994. Shape from shading using linear approximation. Image and Vision Computing, 12(8): 487-498 [DOI:10.1016/0262-8856(94)90002-7]
• Wang L C, Huai Y J, Yang G, Luo D. 2010. Research on realism leaves modeling and rendering in virtual botany. Computer Simulation, 27(5): 204-208 (王立臣, 淮永建, 杨刚, 罗岱. 2010. 虚拟植物叶片的可视化建模技术研究. 计算机仿真, 27(5): 204-208) [DOI:10.3969/j.issn.1006-9348.2010.05.050]
• Wang L P, He H J, Yang H Y. 2017. Three-dimensional shape modeling and real-time color rendering of rice leaf. Computer Engineering and Applications, 53(24): 187-190 (汪丽萍, 何火娇, 杨红云. 2017. 水稻叶片3维建模与叶色渲染. 计算机工程与应用, 53(24): 187-190) [DOI:10.3778/j.issn.1002-8331.1606-0328]
• Xu M, Liao Y, Zhang Y N, Zhao R C. 2004. Analyzing the minimization methods of shape from shading. Computer Engineering and Applications, 40(2): 20-22 (须明, 廖熠, 张艳宁, 赵荣椿. 2004. Shape From Shading最小化方法分析. 计算机工程与应用, 40(2): 20-22) [DOI:10.3321/j.issn:1002-8331.2004.02.007]
• Zhang R, Tsai P S, Cryer and Shah. 1994. Analysis of shape from shading techniques//Proceedings of 1994 IEEE Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 377-384[DOI: 10.1109/CVPR.1994.323854]
• Zhang R, Tsai P S, Cryer J E, Shah M. 1999. Shape-from-shading: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(8): 690-706 [DOI:10.1109/34.784284]
• Zhang S J, Liu W X. 2018. Skeleton-based single image 3D reconstruction. Computer and Digital Engineering, 46(5): 997-1002 (张淑军, 刘文晓. 2018. 基于骨架的单幅图像3维建模. 计算机与数字工程, 46(5): 997-1002) [DOI:10.3969/j.issn.1672-9722.2018.05.030]
• Zhao Z Y, Jiang M R, Huang Y Q, Hao J Y, Zeng K. 2019. Single image depth estimation algorithm based on SFS and binocular model. Computer Science, 46(S1): 161-164 (赵子阳, 蒋慕蓉, 黄亚群, 郝健宇, 曾科. 2019. 结合SFS和双目模型的单幅图像深度估计算法. 计算机科学, 46(S1): 161-164)
• Zheng Q, Chellappa R. 1991. Estimation of illuminant direction, albedo, and shape from shading. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(7): 680-702 [DOI:10.1109/34.85658]
• Zhong F J, Xiao Z T, Zhang F. 2012. Shape recovery algorithm based on both shading and texture information. Journal of Tianjin Polytechnic University, 31(3): 60-64 (钟方洁, 肖志涛, 张芳. 2012. 一种结合明暗信息和纹理信息的形状恢复算法. 天津工业大学学报, 31(3): 60-64) [DOI:10.3969/j.issn.1671-024X.2012.03.015]