Objective

2

Human action recognition has a very wide application prospect in fields such as video surveillance

human-computer interface

environment-assisted life

human-computer interaction

and intelligent driving. In image or video analysis

most of these tasks use color and texture cues in 2D images for recognition. However

due to occlusion

shadows

illumination changes

perspective changes

scale changes

intra-class variations

and similarities between classes

the recognition rate of human behavior is not ideal. In recent years

with the release of 3D depth cameras

such as Microsoft Kinect

3D depth data can provide pictures of scene changes

thereby improving the recognition rates for the first three challenges of human recognition. In addition

3D depth cameras provide powerful human motion capture technology

which can output the human skeleton of a 3D joint point position. Therefore

much attention has been paid to skeleton-based action recognition. The linear dynamical system (LDS) is the most common method for encoding spatio-temporal time-series data in various disciplines due to its simplicity and efficiency. A new method is proposed to obtain the parameters of a tensor-based LDS with forward and inverse action sequences to construct a complete observation matrix. The linear subspace of the observation matrix

which maps to a point on Grassmann manifold for human action recognition

is obtained. In this manner

an action can be expressed as a subspace spanned by columns of the matrix corresponding to a point on the Grassmann manifold. On the basis of such action

classification can be performed using dictionary learning and sparse coding.

Method

2

Considering the dynamics and persistence of human behavior

we do not vectorize the time series according to the general method but retain its own tensor characteristics

that is

we transform the high-dimensional vector into a low-dimensional subspace to analyze the factors affecting actions from various angles (modules). In this method

human skeletons are modeled using human joint points

which are initially extracted from a depth camera recording. To preserve the original spatio-temporal information of an action video and enhance the accuracy of human action recognition

we develop a time series of skeleton motions on the basis of the data in a three-order tensor and convert the skeleton into a two-order tensor. With this action representation

Tucker tensor decomposition methods are applied to obtain dimensionality reduction. Using the tensor-based LDS model with forward and inverse action sequence

we learn a parameter tuple (

A

F

A

I

C

)

in which

C

represents the spatial appearance of skeleton information

A

F

describes the dynamics of the forward time series

and

A

I

describes the dynamics of the inversion time series. We consider using an

m

-order observable matrix to approximate the extended observable matrix because human behavior has a limited duration and does not extend indefinitely in time. When

$$m$$

is small

it is insufficient to describe the entire action sequence. In case of cyclic sub-actions in human behavior

even adding

$$m$$

cannot simulate the follow-up action of human behavior. When combined with the observable matrix of an inverse action sequence

the description of cyclic sub-actions evidently makes up for this shortcoming

improves the performance of the system

increases the completeness of the finite observation matrix

and reduces computational complexity. Thus

the finite observability matrix can then be adopted as the feature descriptor for an action sequence with forward and inverse actions. In classifying points on the Grassmann manifolds

a simple method is to insert the Grassmann manifolds into a Euclidean space through tangent bundles of manifolds. This method does not necessarily provide an accurate estimate

and it requires intensive computation. Sparse coding and dictionary learning are carried out to classify points on the Grassmann manifolds by maintaining the Grassmann projection distance (chord metric) in differential homeomorphism. Sparse coding on Grassmann manifolds finds a set of linear subspaces to represent each linear subspace as a linear combination of these linear subspaces.

Result

2

The MSR-Action 3D dataset comprises depth sequences captured by depth cameras. It includes time-segment action sequences that have been preprocessed to remove the background. The dataset contains 20 actions performed by 10 different objects

with each action repeated thrice without any interaction with objects. The UT-Kinect dataset is a 200-frame depth sequence acquired indoors by using Kinect sensors. It contains 10 actions

namely

walking

standing up

picking up

moving

waving

throwing

pushing

sitting down

pulling

and clapping. Each action is repeated twice by 10 different people. To assess the effects of different subspace dimensions on recognition rate

we test the subspace dimensions ranging from 1 to 20. Experiments using the MSR-Action 3D and UT-Kinect datasets demonstrate the excellent performance of our proposed method. Through an extensive set of experimental assessments

we verify that the tensor-based LDS (tLDS) model with forward and inverse action sequences significantly improves the rate of human action recognition. Results show that the rate of the algorithm is 13.55% higher than that of the joint eigenvalue decomposition algorithm

2.79% higher than that of LTBSVM (local tangent bundle support vector machine) algorithm

and 1% higher than that of the tLDS algorithm on the MSR-Action 3D dataset. For the UT-Kinect dataset

the recognition rate of the proposed algorithm is 5.8% higher than that of the LTBSVM algorithm and 1.3% higher than that of tLDS algorithm.

Conclusion

2

We develop a novel action representation

namely

the tensor-based LDS model with forward and inverse action sequences. The proposed model translates 3D human skeleton sequences into tensor time series without having to unfold the skeletons onto column vectors. Tucker decomposition is used to estimate the parameters of the model as action descriptors. Through an extensive set of experimental assessments

we verify that the tensor-based LDS model with forward and inverse action sequences significantly improves the rate of human action recognition. Major contributions enabled by the proposed method include several novel skeleton-based tensor representations. Our next intended approach in subsequent research is to apply the tensor-based LDS model with forward and inverse action sequences to multi-person interactions.