Current Issue Cover
基于网格形变的立体图像内容重组

柴雄力, 邵枫, 姜求平, 蒋刚毅(宁波大学信息科学与工程学院, 宁波 315211)

摘 要
目的 近年来,随着数字摄影技术的飞速发展,图像增强技术越来越受到重视。图像构图作为图像增强中影响美学的重要因素,一直都是研究的热点。为此,从立体图像布局调整出发,提出一种基于Delaunay网格形变的立体图像内容重组方法。方法 首先将待重组的一对立体图像记为源图像,将用于重组规则确定的一幅图像记为参考图像;然后对源图像需要调整的目标、特征线和其他区域进行取点操作,建立Delaunay网格。将源图像的左图与参考图像进行模板匹配操作,得到源图像与参考图像在结构布局上的对应关系;最后利用网格形变的特性,移动和缩放目标对象,并对立体图像的深度进行自适应调整。结果 针对目标对象的移动、缩放和特征线调整几方面进行优化。当只涉及目标对象的移动或特征线调整时,立体图像视差保持不变;当目标对象缩放时,立体图像中目标对象的视差按照缩放比例变化而背景视差保持不变。实验结果表明,重组后的立体图像构图与参考图像一致且深度能自适应调整。与最新方法比较,本文方法在目标对象分割精度和图像语义保持方面具有优势。结论 根据网格形变相关理论,构建图像质量、布局匹配和视差适应3种能量项,实现了立体图像的内容重组。与现有需要提取和粘贴目标对象的重组方法不同,本文方法对目标对象的分割精度要求不高,不需要图像修复和混合技术,重组后的立体图像没有伪影和语义错误出现。用户可以通过参考图像来引导立体图像的布局调整,达到期望的图像增强效果。
关键词
Stereoscopic image recomposition based on mesh deformation

Chai Xiongli, Shao Feng, Jiang Qiuping, Jiang Gangyi(Faculty of Information Science and Engineering, Ningbo University, Ningbo 315211, China)

Abstract
Objective Image enhancement techniques have received increasing attention along with the rapid development of digital photography in recent years. Image enhancement aims to improve the visual quality of an image through tone mapping, denoising, and recomposition. In image aesthetic assessment, the main factors that influence aesthetics, namely, object relationships and geometric structure, are considered in recomposition. As a research hotspot, image recomposition utilizes photographic composition rules, such as the rule of thirds, visual balance, diagonal dominance, and object size, to capture aesthetically pleasing content. Nonphotographic experts can also acquire photographic images that conform to image aesthetics using image recomposition technology. Stereoscopic technologies have undergone a tremendous boom in recent years. Various stereoscopic services and applications are currently available which present great demands for the availability of 3D contents. A growing demand for stereoscopic image layout adjustment has been observed due to the separation of stereoscopic content production and display. However, the content modification of 3D images is difficult compared with the conventional 2D image modification. The former requires additional care due to the additional depth dimension. The misalignment between left and right images may result in uncomfortable 3D viewing and cause eye strain and headache. In consideration of the above factors, starting from stereoscopic image layout adjustment, this study proposes a stereoscopic image recomposition method based on Delaunay mesh deformation and depth adaptation. Method A pair of stereoscopic images to be recomposed is first recorded as source images, including left and right images, and a binary image used for rule determination is recorded as a reference image. Alpha matting is used to obtain a precise region with opacity value for each object in the left image and calculate the significance of stereoscopic images. We then detect feature points from the left image and use the Delaunay triangulation algorithm to generate meshes as follows. We employ an edge detection operator, such as Canny operator, and utilize a corner detection algorithm, such as Harris corner detection, to extract feature points in a target object. Hough transform is used to detect feature lines and select points in the left image to evenly sample feature points in the feature lines. Here, the location of the target object and the feature line are classified into three categories:intersect, above separate, and below separate. When intersecting with the feature line, the target object moves along with the feature line, and the rest of the situation can be considered separately. The left image boundary is evenly discretized to use all the points there as part of the feature points and sample the remaining area to gain feature points. The Delaunay triangulation mesh can be automatically generated based on the feature points. After the left image meshes are established, the meshes in the right image are mapped by disparity from the left image meshes, and the reference image meshes are also built similar to the operation of the left image. A template-matching operation is performed on the contents of the left and reference images to obtain the corresponding relationship between the source and reference images in the layout. In the optimization process, we construct energy terms from three aspects, namely, image quality, layout adjustment, and depth adaptation. Finally, the target object is moved and scaled based on the characteristics of mesh deformation, and the depth of the stereoscopic image is adaptively adjusted. The disparity change ratio of the target object is consistent with the size scaling. Result This study conducts an experimental design from two aspects of single object and multiple objects, which prove that the proposed method is applicable to all objects. When the movement of the target object or the adjustment of the feature line is involved, the disparity of the stereoscopic image is maintained. When the target object is scaled, the disparity of the target object in the stereoscopic image varies according to the scaling ratio, while the background disparity remains unchanged. Experimental results show that the stereoscopic image after recomposition is consistent with the layout of the reference image for the target object movement, scaling, and feature line adjustment and that the depth can be adjusted adaptively. The coefficients of different energy terms are also adjusted to prove that the new framework proposed in this study can achieve satisfactory stereoscopic content recomposition. Compared with the latest method, the optimization method used in this study has advantages in the segmentation accuracy of the target object and the preservation of image semantics. Conclusion In this study, image quality, layout matching, and disparity adaptive energy terms are constructed based on the theory of mesh deformation, and content recomposition of the stereoscopic image is achieved according to energy term optimization. Unlike the existing recomposition method that needs to extract and paste the target object, the proposed method does not require high accuracy in the segmentation of the target object. Image-inpainting and -blending techniques are not needed. The stereoscopic image after recomposition has no artifacts and semantic errors. The user can guide the layout adjustment of the stereoscopic image by using the reference image to achieve the image enhancement desired by the user. In the future, mesh deformation and cropping technologies can be combined to enhance the efficiency and flexibility of stereoscopic image recomposition.
Keywords

订阅号|日报