Human behavior understanding is a challenging area in computer vision and machine intelligence. A basic problem in this area is the semantic gap between observable actions and human behavior

which should be bridged via context based reasoning. In this paper

we proposed a method to model the daily knowledge about action behavior

environment and their relationship. A novel progressive reasoning method is further built for striding over the semantic gap based on an extendable environment and action model. At first

models about the relation between features

complex features and behavior are built. Feature extraction modules process the continuously sensor data and forward the results to the reasoning module

in which a set of possible behaviors is selected via the feature-behavior models. The set of behaviors is further used as a condition in feature extraction for finding more features to support or to discriminate the behaviors in the set. A system is developed to continuously progressively reason human behavior by the proposed models. Primary experiment shown the system can continuously understand concurrent human behaviors.