浏览全部资源
扫码关注微信
收稿：2018-06-14，
修回：2018-8-24，
纸质出版：2019-03-16
移动端阅览
目的
2
近年来，随着数字摄影技术的飞速发展，图像增强技术越来越受到重视。图像构图作为图像增强中影响美学的重要因素，一直都是研究的热点。为此，从立体图像布局调整出发，提出一种基于Delaunay网格形变的立体图像内容重组方法。
方法
2
首先将待重组的一对立体图像记为源图像，将用于重组规则确定的一幅图像记为参考图像；然后对源图像需要调整的目标、特征线和其他区域进行取点操作，建立Delaunay网格。将源图像的左图与参考图像进行模板匹配操作，得到源图像与参考图像在结构布局上的对应关系；最后利用网格形变的特性，移动和缩放目标对象，并对立体图像的深度进行自适应调整。
结果
2
针对目标对象的移动、缩放和特征线调整几方面进行优化。当只涉及目标对象的移动或特征线调整时，立体图像视差保持不变；当目标对象缩放时，立体图像中目标对象的视差按照缩放比例变化而背景视差保持不变。实验结果表明，重组后的立体图像构图与参考图像一致且深度能自适应调整。与最新方法比较，本文方法在目标对象分割精度和图像语义保持方面具有优势。
结论
2
根据网格形变相关理论，构建图像质量、布局匹配和视差适应3种能量项，实现了立体图像的内容重组。与现有需要提取和粘贴目标对象的重组方法不同，本文方法对目标对象的分割精度要求不高，不需要图像修复和混合技术，重组后的立体图像没有伪影和语义错误出现。用户可以通过参考图像来引导立体图像的布局调整，达到期望的图像增强效果。
Objective
2
Image enhancement techniques have received increasing attention along with the rapid development of digital photography in recent years. Image enhancement aims to improve the visual quality of an image through tone mapping
denoising
and recomposition. In image aesthetic assessment
the main factors that influence aesthetics
namely
object relationships and geometric structure
are considered in recomposition. As a research hotspot
image recomposition utilizes photographic composition rules
such as the rule of thirds
visual balance
diagonal dominance
and object size
to capture aesthetically pleasing content. Nonphotographic experts can also acquire photographic images that conform to image aesthetics using image recomposition technology. Stereoscopic technologies have undergone a tremendous boom in recent years. Various stereoscopic services and applications are currently available which present great demands for the availability of 3D contents. A growing demand for stereoscopic image layout adjustment has been observed due to the separation of stereoscopic content production and display. However
the content modification of 3D images is difficult compared with the conventional 2D image modification. The former requires additional care due to the additional depth dimension. The misalignment between left and right images may result in uncomfortable 3D viewing and cause eye strain and headache. In consideration of the above factors
starting from stereoscopic image layout adjustment
this study proposes a stereoscopic image recomposition method based on Delaunay mesh deformation and depth adaptation.
Method
2
A pair of stereoscopic images to be recomposed is first recorded as source images
including left and right images
and a binary image used for rule determination is recorded as a reference image. Alpha matting is used to obtain a precise region with opacity value for each object in the left image and calculate the significance of stereoscopic images. We then detect feature points from the left image and use the Delaunay triangulation algorithm to generate meshes as follows. We employ an edge detection operator
such as Canny operator
and utilize a corner detection algorithm
such as Harris corner detection
to extract feature points in a target object. Hough transform is used to detect feature lines and select points in the left image to evenly sample feature points in the feature lines. Here
the location of the target object and the feature line are classified into three categories:intersect
above separate
and below separate. When intersecting with the feature line
the target object moves along with the feature line
and the rest of the situation can be considered separately. The left image boundary is evenly discretized to use all the points there as part of the feature points and sample the remaining area to gain feature points. The Delaunay triangulation mesh can be automatically generated based on the feature points. After the left image meshes are established
the meshes in the right image are mapped by disparity from the left image meshes
and the reference image meshes are also built similar to the operation of the left image. A template-matching operation is performed on the contents of the left and reference images to obtain the corresponding relationship between the source and reference images in the layout. In the optimization process
we construct energy terms from three aspects
namely
image quality
layout adjustment
and depth adaptation. Finally
the target object is moved and scaled based on the characteristics of mesh deformation
and the depth of the stereoscopic image is adaptively adjusted. The disparity change ratio of the target object is consistent with the size scaling.
Result
2
This study conducts an experimental design from two aspects of single object and multiple objects
which prove that the proposed method is applicable to all objects. When the movement of the target object or the adjustment of the feature line is involved
the disparity of the stereoscopic image is maintained. When the target object is scaled
the disparity of the target object in the stereoscopic image varies according to the scaling ratio
while the background disparity remains unchanged. Experimental results show that the stereoscopic image after recomposition is consistent with the layout of the reference image for the target object movement
scaling
and feature line adjustment and that the depth can be adjusted adaptively. The coefficients of different energy terms are also adjusted to prove that the new framework proposed in this study can achieve satisfactory stereoscopic content recomposition. Compared with the latest method
the optimization method used in this study has advantages in the segmentation accuracy of the target object and the preservation of image semantics.
Conclusion
2
In this study
image quality
layout matching
and disparity adaptive energy terms are constructed based on the theory of mesh deformation
and content recomposition of the stereoscopic image is achieved according to energy term optimization. Unlike the existing recomposition method that needs to extract and paste the target object
the proposed method does not require high accuracy in the segmentation of the target object. Image-inpainting and -blending techniques are not needed. The stereoscopic image after recomposition has no artifacts and semantic errors. The user can guide the layout adjustment of the stereoscopic image by using the reference image to achieve the image enhancement desired by the user. In the future
mesh deformation and cropping technologies can be combined to enhance the efficiency and flexibility of stereoscopic image recomposition.
Yao L, Suryanarayan P, Qiao M, et al. OSCAR:On-site composition and aesthetics feedback through exemplars for photographers[J]. International Journal of Computer Vision, 2012, 96(3):353-383.[DOI:10.1007/s11263-011-0478-3]
Islam M B, Wong L K, Wong C O. A survey of aesthetics-driven image recomposition[J]. Multimedia Tools and Applications, 2017, 76(7):9517-9542.[DOI:10.1007/s11042-016-3561-5]
Bhattacharya S, Sukthankar R, Shah M. A framework for photo-quality assessment and enhancement based on visual aesthetics[C]//Proceedings of the 18th ACM International Conference on Multimedia. Firenze, Italy: ACM, 2010: 271-280.[ DOI:10.1145/1873951.1873990 http://dx.doi.org/10.1145/1873951.1873990 ]
Liu L G, Chen R J, Wolf L, et al. Optimizing photo composition[J]. Computer Graphics Forum, 2010, 29(2):469-478.[DOI:10.1111/j.1467-8659.2009.01616.x]
Wang W N, Liu J C, Xu X M, et al. Aesthetic enhancement of images based on photography composition guidelines[J]. Journal of South China University of Technology:Natural Science Edition, 2015, 43(5):51-58.
王伟凝, 刘剑聪, 徐向民, 等.基于构图规则的图像美学优化[J].华南理工大学学报:自然科学版, 2015, 43(5):51-58.[DOI:10.3969/j.issn.1000-565X.2015.05.009]
Liang Y, Wang X T, Zhang S H, et al. PhotoRecomposer:interactive photo recomposition by cropping[J]. IEEE Transactions on Visualization and Computer Graphics, 2017, 24(10):2728-2742.[DOI:10.1109/TVCG.2017.2764895]
Bhattacharya S, Sukthankar R, Shah M. A holistic approach to aesthetic enhancement of photographs[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2011, 7S(1):#21.[DOI:10.1145/2037676.2037678]
Park J, Lee J Y, Tai Y W, et al. Modeling photo composition and its application to photo re-arrangement[C]//Proceedings of the 19th IEEE International Conference on Image Processing. Orlando, FL, USA: IEEE, 2012: 2741-2744.[ DOI:10.1109/ICIP.2012.6467466 http://dx.doi.org/10.1109/ICIP.2012.6467466 ]
Guo G J, Wang H Z, Shen C H, et al. Automatic image cropping for visual aesthetic enhancement using deep neural networks and cascaded regression[J]. IEEE Transactions on Multimedia, 2018, 20(8):2073-2085.[DOI:10.1109/TMM.2018.2794262]
Zhang F L, Wang M, Hu S M. Aesthetic image enhancement by dependence-aware object recomposition[J]. IEEE Transactions on Multimedia, 2013, 15(7):1480-1490.[DOI:10.1109/TMM.2013.2268051]
Chang H T, Wang Y C F, Chen M S. Transfer in photography composition[C]//Proceedings of the 22nd ACM International Conference on Multimedia. Orlando, Florida, USA: ACM, 2014: 957-960.[ DOI:10.1145/2647868.2654976 http://dx.doi.org/10.1145/2647868.2654976 ]
Li K, Yan B, Li J, et al. Seam carving based aesthetics enhancement for photos[J]. Signal Processing:Image Communication, 2015, 39:509-516.[DOI:10.1016/j.image.2015.07.005]
Wong L K, Low K L. Enhancing visual dominance by semantics-preserving image recomposition[C]//Proceedings of the 20th ACM International Conference on Multimedia. Nara, Japan: ACM, 2012: 845-848.[ DOI:10.1145/2393347.2396327 http://dx.doi.org/10.1145/2393347.2396327 ]
Chang H T, Pan P C, Wang Y C F, et al. R2P: Recomposition and retargeting of photographic images[C]//Proceedings of the 23rd ACM International Conference on Multimedia. Brisbane, Australia: ACM, 2015: 927-930.[ DOI:10.1145/2733373.2806366 http://dx.doi.org/10.1145/2733373.2806366 ]
Jin Y, Wu Q B, Liu L G. Aesthetic photo composition by optimal crop-and-warp[J]. Computers&Graphics, 2012, 36(8):955-965.[DOI:10.1016/j.cag.2012.07.007]
Niu Y Z, Liu F, Feng W C, et al. Aesthetics-based stereoscopic photo cropping for heterogeneous displays[J]. IEEE Transactions on Multimedia, 2012, 14(3):783-796.[DOI:10.1109/TMM.2012.2186122]
Tong R F, Zhang Y, Cheng K L. StereoPasting:interactive composition in stereoscopic images[J]. IEEE Transactions on Visualization and Computer Graphics, 2013, 19(8):1375-1385.[DOI:10.1109/TVCG.2012.319]
Luo S J, Shen I C, Chen B Y, et al. Perspective-aware warping for seamless stereoscopic image cloning[J]. ACM Transactions on Graphics, 2012, 31(6):#182.[DOI:10.1145/2366145.2366201]
Islam M B, Wong L K, Low K L, et al. Aesthetics-driven stereoscopic 3-D image recomposition with depth adaptation[J]. IEEE Transactions on Multimedia, 2018, 20(11):2964-2979.[DOI:10.1109/TMM.2018.2820324]
Islam M B, Wong L K, Wong C O, et al. Stereoscopic image warping for enhancing composition aesthetics[C]//Proceedings of the 3rd IAPR Asian Conference on Pattern Recognition. Kuala Lumpur, Malaysia: IEEE, 2015: 645-649.[ DOI:10.1109/ACPR.2015.7486582 http://dx.doi.org/10.1109/ACPR.2015.7486582 ]
Gastal E S L, Oliveira M M. Shared sampling for real-time alpha matting[J]. Computer Graphics Forum, 2010, 29(2):575-584.[DOI:10.1111/j.1467-8659.2009.01627.x]
Wang W G, Shen J B, Yu Y Z, et al. Stereoscopic thumbnail creation via efficient stereo saliency detection[J]. IEEE Transactions on Visualization and Computer Graphics, 2017, 23(8):2014-2027.[DOI:10.1109/TVCG.2016.260059]
Fernandes L A F, Oliveira M M. Real-time line detection through an improved Hough transform voting scheme[J]. Pattern Recognition, 2008, 41(1):299-314.[DOI:10.1016/j.patcog.2007.04.003]
Zhang G X, Cheng M M, Hu S M, et al. A shape-preserving approach to image resizing[J]. Computer Graphics Forum, 2010, 28(7):1897-1906.[DOI:10.1111/j.1467-8659.2009.01568.x]
Chang C H, Liang C K, Chuang Y Y. Content-aware display adaptation and interactive editing for stereoscopic images[J]. IEEE Transactions on Multimedia, 2011, 13(4):589-601.[DOI:10.1109/TMM.2011.2116775]
Niu Y Z, Feng W C, Liu F. Enabling warping on stereoscopic images[J]. ACM Transactions on Graphics, 2012, 31(6):#183.[DOI:10.1145/2366145.2366202]
Ju R, Liu Y, Ren T W, et al. Depth-aware salient object detection using anisotropic center-surround difference[J]. Signal Processing:Image Communication, 2015, 38:115-126.[DOI:10.1016/j.image.2015.07.002]
相关作者
相关机构
京公网安备11010802024621
