Objective

2

IQA (image quality assessment) is one of the fundamental research topics in the fields of computer vision and image processing. Traditional quality assessment methods are mainly based on low-level visual features and generally ignore high-level semantic information. Traditional IQA methods mainly rely on single pixel intensity or low-level visual features

such as image contrast

image edges

etc

.

to assess images. PSNR (peak signal-to-noise ratio) is a basic and commonly used tool for directly comparing the differences of pixel intensities between the test image and the reference image By contrast

human visual systems extract structural information from visual scenes. The PSNR cannot accurately measure the subjective visual quality. To extract the structure information and attain a better evaluation

various kinds of improved IQA methods have been proposed. Many methods first decompose an image into different aspects to extract information that effectively measures image quality. However

these traditional methods still omit the high-level semantic information. With the rapid development of deep learning algorithms

high-level semantic information can be effectively extracted by deep networks. Given their special hierarchical structure

deep networks can analyze and understand images in different levels. In recent years

perceptual loss based on deep network has been widely used in many computer vision applications

such as image style-transfer

non-photorealistic rendering

image restoration

etc. By utilizing a pre-trained deep network to decompose an image into different semantic levels

satisfactory results can be produced for related tasks. Inspired by the perceptual loss

we proposed a multi-layer perceptual decomposition-based full-reference image quality assessment method.

Method

2

First

a pre-trained deep network was used to decompose the input image and extract the multi-layer feature maps. Many pre-trained deep networks could be employed for this purpose. On the basis of previous studies on perceptual loss

the VGG-19 network was selected because of its effectiveness. VGG-19 is composed of several different layers

including the convolutional

activation function

pool

dropout

fully connected

and softmax layers. These elements are stacked in a specific order to form a completed network model. This network has been widely applied because it can achieve impressive results in many recognition tasks. To reduce complexity

several layers were set as the abstraction layer for extracting feature maps. Second

the proposed method calculated not only the similarity between the test image and the reference image but also the similarity between their multi-level feature maps. The feature maps at the lower level can reflect the differences of the image in the edge

detail

texture

and some low-level features

whereas the feature maps at the higher level can reflect the saliency and semantic differences of the image in the region of interest. Finally

an image quality score that considered the similarity of high-level semantic was obtained. Compared with existing DNN (deep neural network)-based IQA methods

the pre-trained deep network was merely utilized to decompose the image rather than to fit the subjective mean opinion scores. Thus

the proposed method did not need to train a new IQA network in contrast to other DNN-based methods. Moreover

the proposed method was an open and elastic framework that improved the performance of traditional methods by extracting additional high-level semantic information. Therefore

numerous traditional full reference IQA methods can be further improved by exploiting the proposed framework. In this paper

a number of typical and efficient traditional IQA methods were improved and evaluated by proposed method. These IQA methods included the PSNR

the SSIM (structure similarity)

and its two effective variants

namely

MS-SSIM (multi-scale structure similarity) and FSIM (feature similarity). Other full reference IQA methods can also be improved by the proposed semantic decomposition-based framework.

Result

2

The experimental data were derived from the TID2013 dataset

which includes 25 reference images and 3 000 distorted images. Compared with other existing databases

TID2013 has more images and distortion types

guaranteeing more reliable results. The experimental results of the selected traditional methods

namely

PSNR

SSIM

MS-SSIM

and FSIM

showed that the proposed method can effectively improve the performance of traditional image quality assessment methods and achieve corresponding improvements in many objective criteria

such as SRCC (Spearman rank order correlation coefficient)

KRCC (Kendall rank order correlation coefficient)

PLCC (Pearson linear correlation coefficient)

and RMSE (root mean squared error). The SRCC indicators were increased by 0.02

0.07

0.06

and 0.04 for PSNR

SSIM

MS-SSIM

and FSIM

respectively

on the TID2013 dataset. SRCC and KRCC measure the predicting monotonicity. PLCC is calculated to predict accuracy. RMSE is used to predict consistency. These traditional assessments can attain higher SRCC

KRCC

and PLCC values by using the proposed method. For the RMSE

the proposed methods can achieve much lower results than those of the corresponding conventional IQA methods. In addition

the results for different distortion types demonstrated that the proposed method can effectively improve the performance.

Conclusion

2

This paper proposed a full-reference image quality assessment method based on perceptual decomposition that combined the benefits of traditional methods and deep learning methods. By simultaneously considering the low-level visual features and high-level semantic information

the proposed method effectively improved the evaluation performance of traditional methods. By incorporating the additional high-level semantic information

the IQA results became more consistent with the subjective visual perception. Furthermore

the proposed evaluation framework can also be applied to other traditional full reference IQA methods.