浏览全部资源
扫码关注微信
Published： 16 December 2023 ，
移动端阅览
廖远鸿， 钱文华， 曹进德. 2023. 风格强度可变的人脸风格迁移网络. 中国图象图形学报， 28(12):3784-3796
Liao Yuanhong， Qian Wenhua， Cao Jinde. 2023. MStarGAN： a face style transfer network with changeable style intensity. Journal of Image and Graphics， 28(12):3784-3796
目的
2
针对人脸风格迁移算法StarGAN（star generative adversarial network）、MSGAN（mode seeking generative adversarial network）等存在细节风格学习不佳、迁移效果单一和生成图像失真等缺点，提出一种能够降低失真并生成不同风格强度图像的人脸风格迁移算法MStarGAN（multilayer StarGAN）。
方法
2
首先，通过特征金字塔网络（feature pyramid network，FPN）构建前置编码器，生成蕴含有图像细节特征的多层特征向量，增强生成图像在风格传输时能学习到的风格图像的细节风格；其次，使用前置编码器对原图像及风格图像各生成一个风格向量并进行组合，利用组合后的风格向量进行风格传输，使生成图像具有不同的风格迁移强度；最后，采用权重解调算法作为生成器中的风格传输模块，通过对卷积权重的操作代替在特征图上的归一化操作，消除特征图中的特征伪影，减少生成图像中的失真。
结果
2
在Celeba_HQ数据集上进行实验，与MSGAN、StarGAN v2等对比算法相比，在参考引导合成实验中，MStarGAN的FID（Frechét inception distance score）指标分别降低了18.9和3.1，LPIPS（learned perceptual image patch similarity）指标分别提升了0.094和0.018。在潜在引导合成实验中，MStarGAN的FID指标分别降低了20.2和0.8，LPIPS指标分别提升了0.155和0.92，并能够生成具有不同风格强度的结果图像。
结论
2
提出的算法能够传输图像的细节风格，生成具有不同强度的输出图像，并减少生成图像的失真。
Objective
2
The style transfer algorithm can transfer the style from the art image to the original natural image. The style image provides certain features， such as style texture and stroke， while the content image provides the contour structure. The goal of the style transfer algorithm is to synthesize a new stylized image with the texture stroke of the style image and the contour structure of the content image. The early face style transfer algorithm applies mathematical modeling to build a filter that counts the local features of the target image to understand its style. This algorithm then establishes a statistical model to describe the image style. However， the face style transfer algorithm only generates a single style， the resulting image style is not obvious， and needs to be modeled manually， thereby limiting its efficiency. With the rise of deep learning， the style transfer algorithm has started using the deep learning model as its core. Given that generative adversarial network （GAN） can generate images that satisfy certain distribution laws， we can generate a target image that is similar to the real image by training GAN. Therefore， GAN has been widely used in image style transfer algorithms. The main image style transfer algorithms are divided into two categories. The algorithms in the first category only improve GAN without using a pre-encoder， such as pix2pix and CycleGAN， while those in the second category use a pre-encoder. Due to the addition of encoders before the GAN structure， the resulting network structure becomes complex yet achieves highly realistic results， such as StyleGAN and StarGAN. To overcome the shortcomings of some face style transfer algorithms， such as StarGAN and MSGAN， which have poor detail style learning， insignificant style transfer effect， and generation of distorted images， we present a face style migration algorithm called multi-layer StarGAN （MStarGAN） with controllable style intensity.
Method
2
First， we construct the pre-encoder through the feature pyramid network （FPN） to generate multi-layer feature vectors containing image detail features. Compared with the original 1 × 64 feature vector， the pre-encoder constructed by FPN can output a 6 × 256 feature vector， which contains additional details of the original image. Therefore， the generated image can learn the detailed style of the style image during style transmission. Second， we use the pre-encoder to generate style vectors for the original and style images and then combine these vectors. We then use the combined style vector for style transmission. We can also adjust the number of layers of this vector so that the style of the generated image is biased to either the original or style image， hence resulting in different style transfer intensities for the generated image. Third， we introduce a new loss function to maintain balance in the style of the generated image and ensure that the image will not be too biased toward either the original or style image. Fourth， we apply the weight demodulation algorithm as our style transmission module in the generator. The traditional method AdaIN has been proven to distort the generated image. By replacing the normalization operation on the feature map with the operation of convolution weight， we eliminate the feature artifacts in the feature map and reduce the distortion in the generated image.
Result
2
We implement our model in Python and test it on the Celeba_HQ dataset with RTX2080Ti. Our model not only generates high-quality random face images but also makes the generated images learn the style of style images， including hair and skin color. Compared with the multimodal unsupervised image-to-image translation）， diverse image-to-image translation， MSGAN， and StarGAN V2 algorithms， in the latent-guided synthesis experiment， the Frechét inception distance （FID） index of the proposed algorithm is reduced by 18.5， 39.2， 20.2， and 0.8， respectively， while its learned perceptual image patch similarity （LPIPS） index is increased by 0.181， 0.366， 0.155， and 0.092 respectively. In the reference-guided synthesis experiment， the FID index of the proposed algorithm is reduced by 86.4， 32.6， 18.9， and 3.1， respectively， while its LPIPS index is increased by 0.23， 0.095， 0.094， and 0.018， respectively. In sum， our algorithm can generate result images with different styles and intensities.
Conclusion
2
The proposed algorithm can transmit the detail style of the image， control the style intensity of the output image， and reduce the distortion of the generated image.
人脸风格迁移网络StarGAN风格强度特征金字塔网络（FPN）权重解调
face style transfer networkStarGANstyle intensityfeature pyramid network （FPN）weight demodulation
Choi Y， Choi M， Kim M， Ha J W， Kim S and Choo J. 2018. StarGAN： unified generative adversarial networks for multi-domain image-to-image translation//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City， USA： IEEE： 8789-8797 ［DOI： 10.1109/CVPR.2018.00916http://dx.doi.org/10.1109/CVPR.2018.00916］
Choi Y， Uh Y， Yoo J and Ha J W. 2020. StarGAN v2： diverse image synthesis for multiple domains//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， USA： IEEE： 8185-8194 ［DOI： 10.1109/CVPR42600.2020.00821http://dx.doi.org/10.1109/CVPR42600.2020.00821］
Denton E， Chintala S， Szlam A and Fergus R. 2015. Deep generative image models using a Laplacian pyramid of adversarial networks//Proceedings of the 28th International Conference on Neural Information Processing Systems. Montreal， Canada： MIT Press： 1486-1494
Elgammal A， Liu B C， Elhoseiny M and Mazzone M. 2017. CAN： creative adversarial networks generating “art” by learning about styles and deviating from style norms ［EB/OL］. ［2022-12-22］. https://arxiv.org/pdf/1706.07068.pdfhttps://arxiv.org/pdf/1706.07068.pdf
Gatys L A， Ecker A S and Bethge M. 2015a. Texture synthesis using convolutional neural networks//Proceedings of the 28th International Conference on Neural Information Processing Systems. Montreal， Canada： MIT Press： 262-270
Gatys L A， Ecker A S and Bethge M. 2015b. A neural algorithm of artistic style ［EB/OL］. ［2022-12-22］. https://arxiv.org/pdf/1508.06576.pdfhttps://arxiv.org/pdf/1508.06576.pdf
Goodfellow I， Pouget-Abadie J， Mirza M， Xu B， Warde-Farley D， Ozair S， Courville A and Bengio Y. 2020. Generative adversarial networks. Communications of the ACM， 63（11）： 139-144 ［DOI： 10.1145/3422622http://dx.doi.org/10.1145/3422622］
He K M， Zhang X Y， Ren S Q and Sun J. 2016a. Identity mappings in deep residual networks//Proceedings of the 14th European Conference on Computer Vision. Amsterdam， the Netherlands： Springer： 630-645 ［DOI： 10.1007/978-3-319-46493-0_38http://dx.doi.org/10.1007/978-3-319-46493-0_38］
He K M， Zhang X Y， Ren S Q and Sun J. 2016b. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas， USA： IEEE： 770-778 ［DOI： 10.1109/CVPR.2016.90http://dx.doi.org/10.1109/CVPR.2016.90］
Heusel M， Ramsauer H， Unterthiner T， Nessler B and Hochreiter S. 2017. GANs trained by a two time-scale update rule converge to a local Nash equilibrium ［EB/OL］. ［2022-12-22］. https://arxiv.org/pdf/1706.08500.pdfhttps://arxiv.org/pdf/1706.08500.pdf
Huang R， Zhang S， Li T Y and He R. 2017. Beyond face rotation： global and local perception GAN for photorealistic and identity preserving frontal view synthesis//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice， Italy： IEEE： 2458-2467 ［DOI： 10.1109/ICCV.2017.267http://dx.doi.org/10.1109/ICCV.2017.267］
Huang X and Belongie S. 2017. Arbitrary style transfer in real-time with adaptive instance normalization//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice， Italy： IEEE： 1510-1519 ［DOI： 10.1109/ICCV.2017.167http://dx.doi.org/10.1109/ICCV.2017.167］
Huang X， Liu M Y， Belongie S and Kautz J. 2018. Multimodal unsupervised image-to-image translation//Proceedings of the 15th European Conference on Computer Vision. Munich， Germany： Springer： 179-196 ［DOI： 10.1007/978-3-030-01219-9_11http://dx.doi.org/10.1007/978-3-030-01219-9_11］
Isola P， Zhu J Y， Zhou T H and Efros A A. 2017. Image-to-image translation with conditional adversarial networks//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu， USA： IEEE： 5967-5976 ［DOI： 10.1109/CVPR.2017.632http://dx.doi.org/10.1109/CVPR.2017.632］
Karras T， Laine S， Aittala M， Hellsten J， Lehtinen J and Aila T. 2020. Analyzing and improving the image quality of StyleGAN//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， USA： IEEE： 8107-8116 ［DOI： 10.1109/CVPR42600.2020.00813http://dx.doi.org/10.1109/CVPR42600.2020.00813］
Lee H Y， Tseng H Y， Huang J B， Singh M and Yang M H. 2018. Diverse image-to-image translation via disentangled representations//Proceedings of the 15th European Conference on Computer Vision. Munich， Germany： Springer： 36-52 ［DOI： 10.1007/978-3-030-01246-5_3http://dx.doi.org/10.1007/978-3-030-01246-5_3］
Li T T， Qian R H， Dong C， Liu S， Yan Q， Zhu W W and Lin L. 2018. BeautyGAN： instance-level facial makeup transfer with deep generative adversarial network//Proceedings of the 26th ACM International Conference on Multimedia. Seoul， Korea （South）： ACM： 645-653 ［DOI： 10.1145/3240508.3240618http://dx.doi.org/10.1145/3240508.3240618］
Lin T Y， Dollár P， Girshick R， He K M， Hariharan B and Belongie S. 2017. Feature pyramid networks for object detection//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu， USA： IEEE： 936-944 ［DOI： 10.1109/CVPR.2017.106http://dx.doi.org/10.1109/CVPR.2017.106］
Mao Q， Lee H Y， Tseng H Y， Ma S W and Yang M H. 2019. Mode seeking generative adversarial networks for diverse image synthesis//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach， USA： IEEE： 1429-1437 ［DOI： 10.1109/CVPR.2019.00152http://dx.doi.org/10.1109/CVPR.2019.00152］
Mescheder L， Geiger A and Nowozin S. 2018. Which training methods for GANs do actually converge?//Proceedings of the 35th International Conference on Machine Learning. Stockholm， Sweden： PMLR： 3481-3490
Mirza M and Osindero S. 2014. Conditional generative adversarial nets ［EB/OL］. ［2022-12-22］. https://arxiv.org/pdf/1411.1784.pdfhttps://arxiv.org/pdf/1411.1784.pdf
Nirkin Y， Masi I， Tuan A T， Hassner T and Medioni G. 2018. On face segmentation， face swapping， and face perception//Proceedings of the 13th IEEE International Conference on Automatic Face and Gesture Recognition. Xi’an， China： IEEE： 98-105 ［DOI： 10.1109/FG.2018.00024http://dx.doi.org/10.1109/FG.2018.00024］
Portilla J and Simoncelli E P. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision， 40（1）： 49-70 ［DOI： 10.1023/A：1026553619983http://dx.doi.org/10.1023/A：1026553619983］
Sbai O， Elhoseiny M， Bordes A， LeCun Y and Couprie C. 2018. DesIGN： design inspiration from generative networks//Proceedings of 2018 European Conference on Computer Vision. Munich， Germany： Springer： 37-44 ［DOI： 10.1007/978-3-030-11015-4_5http://dx.doi.org/10.1007/978-3-030-11015-4_5］
Shi C J， Tu D J and Liu J Y. 2021. Re-GAN： residual generative adversarial network algorithm. Journal of Image and Graphics， 26（3）： 594-604
史彩娟， 涂冬景， 刘靖祎. 2021. Re-GAN： 残差生成式对抗网络算法. 中国图象图形学报， 26（3）： 594-604 ［DOI： 10.11834/jig.200069http://dx.doi.org/10.11834/jig.200069］
Shih Y C， Paris S， Barnes C， Freeman W T and Durand F. 2014. Style transfer for headshot portraits. ACM Transactions on Graphics， 33（4）： #148 ［DOI： 10.1145/2601097.2601137http://dx.doi.org/10.1145/2601097.2601137］
Shrivastava A， Pfister T， Tuzel O， Susskind J， Wang W D and Webb R. 2017. Learning from simulated and unsupervised images through adversarial training//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu， USA： IEEE： 2242-2251 ［DOI： 10.1109/CVPR.2017.241http://dx.doi.org/10.1109/CVPR.2017.241］
Simonyan K and Zisserman A. 2015. Very deep convolutional networks for large-scale image recognition ［EB/OL］. ［2022-12-22］. https://arxiv.org/pdf/1409.1556.pdfhttps://arxiv.org/pdf/1409.1556.pdf
Wang X Y， Bo L F and Fuxin L. 2019. Adaptive wing loss for robust face alignment via heatmap regression//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul， Korea （South）： IEEE： 6970-6980 ［DOI： 10.1109/ICCV.2019.00707http://dx.doi.org/10.1109/ICCV.2019.00707］
Xie B， Wang N and Fan Y W. 2020. Correlation alignment total variation model and algorithm for style transfer. Journal of Image and Graphics， 25（2）： 241-254
谢斌， 汪宁， 范有伟. 2020. 相关对齐的总变分风格迁移新模型. 中国图象图形学报， 25（2）： 241-254 ［DOI： 10.11834/jig.190199http://dx.doi.org/10.11834/jig.190199］
Yang S， Jiang L M， Liu Z W and Loy C C. 2022. Pastiche master： exemplar-based high-resolution portrait style transfer ［EB/OL］. ［2022-12-22］. https://arxiv.org/pdf/2203.13248.pdfhttps://arxiv.org/pdf/2203.13248.pdf
Zhang R， Isola P， Efros A A， Shechtman E and Wang O. 2018. The unreasonable effectiveness of deep features as a perceptual metric//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City， USA： IEEE： 586-595 ［DOI： 10.1109/CVPR.2018.00068http://dx.doi.org/10.1109/CVPR.2018.00068］
Zheng Z T and Liu J Y. 2020. P2-GAN： efficient style transfer using single style image ［EB/OL］. ［2022-12-22］. https://arxiv.org/pdf/2001.07466.pdfhttps://arxiv.org/pdf/2001.07466.pdf
Zhu J Y， Park T， Isola P and Efros A A. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice， Italy： IEEE： 2242-2251 ［DOI： 10.1109/ICCV.2017.244http://dx.doi.org/10.1109/ICCV.2017.244］
相关作者
相关机构
京公网安备11010802024621