Objective

2

Foreground segmentation is an essential research in the field of image understanding

which is a pre-processing step for saliency object detection

semantic segmentation

and various pixel-level learning tasks. Given an image

this task aims to provide each pixel a foreground or background annotation. For fully supervision-based methods

satisfactory results can be achieved via multi-instance-based learning. However

when facing the problem under unsupervised conditions

achieving a stable segmentation performance based on fixed rules or a single type of feature is difficult because different images and instances always have variable expressions. Moreover

we find that different types of method have different advantages and disadvantages on different aspects. On the one hand

semantic feature-based learning methods could provide accurate key region extraction of foregrounds but could not generate complete object region and edges in detail. On the other hand

richer detailed expression can be obtained based on an apparent feature-based framework

but it cannot be suitable for variable kinds of cases.

Method

2

Based on the observations

we propose an unsupervised foreground segmentation method based on semantic-apparent feature fusion. First

given a sample

we encode it as semantic and apparent feature map. We use a class activation mapping model pretrained on ImageNet for semantic heat map generation and select saliency and edge maps to express the apparent feature. Each kind of semantic and apparent feature can be used

and the established framework is widely adaptive for each case. Second

to combine the advantages of the two type of features

we split the image as super pixels

and set the expression of four elements as unary and binary semantic and apparent feature

which realizes a comprehensive description of the two types of expressions. Specifically

we build two binary relation matrices to measure the similarity of each pair of super pixels

which are based on apparent and semantic feature. For generating the binary semantic feature

we use the apparent feature-based similarity measure as a weight to provide the element for each super pixel

in which semantic-feature-based similarity measure is utilized for binary apparent feature calculation. Based on the different view for feature encoding

the two types of information could be fused for the first time. Then

we propose a method for adaptive parameter learning to calculate the most suitable feature weights and generate the foreground confidence score map. Based on the four elements

we could establish an equation to express each super pixel's foreground confidence score using the least squares method. For an image

we first select super pixels with higher confident scores of unary semantic and apparent feature on foreground or background. Then

we can learn weights of the four elements and bias' linear combination by least squares estimation. Based on the adaptive parameters

we can achieve a better confidence score inference for each super pixel individually. Third

we use segmentation network to learn foreground common features from different instances. In a weakly supervised semantic segmentation task

the fully supervision-based framework is used for improving pseudo annotations for training data and providing inference results. Inspired by the idea

we use the convolution network to mine foreground common feature from different instances. The trained model could be utilized to optimize the quality of foreground segmentation for both images used for network training and new data directly. A better performance can be achieved by fusing semantic and apparent features as well as cascading the modules of intra image adaptive feature weight learning and inter-image common feature learning.

Result

2

We test our methods on the pattern analysis

statistical modelling and computational learning visual object classes(PASCAL VOC)2012 training and evaluation set

which include 10 582 and 1 449 samples

respectively. Precision-recall curve as well as F-measure are used as indicators to evaluate the experimental results. Compared with typical semantic and apparent feature-based foreground segmentation methods

the proposed framework achieves superior improvement of baselines. For PASCAL VOC 2012 training set

the F-measure has a 3.5% improvement

while a 3.4% increase is obtained on the validation set. We also focus on the performance on visualized results for analysis the advantages of fusion framework. Based on comparison

we can find that results with accurate

detailed expression can be achieved based on the adaptive feature fusion operation

while incorrect cases can further be modified via multi-instance-based learning framework.

Conclusion

2

In this study

we propose a semantic-apparent feature fusion method for unsupervised foreground segmentation. Given an image as input

we first calculate the semantic and apparent feature of the unary region of each super pixel in image. Then

we integrate two types of features through the cross-use of similarity measure of apparent and semantic feature. Next

we establish a context relationship for each pair of super pixels to calculate the binary feature of each region. Further

we establish an adaptive weight learning strategy. We obtain the weighting parameters for optimal foreground segmentation and achieve the confidence in the image foreground by automatically adjusting the influence of each dimensional feature on the foreground estimation in each specific image instance. Finally

we build a foreground segmentation network model to learn the common features of foreground between different instances and samples. Using the trained network model

the image can be re-inferred to obtain more accurate foreground segmentation results. The experiments on the PASCAL VOC 2012 training set and validation set prove the effectiveness and generalization ability of the algorithm. Moreover

the method proposed can use other foreground segmentation methods as a baseline and is widely used to improve the performance of tasks such as foreground segmentation and weakly supervised semantic segmentation. We also believe that to consider the introduction of various types of semantic and apparent feature fusion as well as adopt alternate iterations to mine the internal spatial context information of image and the common expression features between different instance is a feasible way to improve the performance of foreground segmentation further and an important idea for semantic segmentation tasks.