Objective Since smoke often occurs earlier than flame when fire breaks out, smoke detection provides earlier fire alarms than flame detection does. The color, shape and movement of smoke are susceptible to external environment, so existing smoke features lack discriminative ability and robustness. These factors make image-based smoke recognition a difficult task. In order to decrease false alarm rates (FARs) and error rates (ERRs) of smoke recognition without dropping detection rates (DRs), we propose a Gabor-based hierarchy (GaborNet) in this paper. Method First, a Gabor convolutional unit, which consists of a set of learning-free convolutional filters and condensing modules, is constructed. The Gabor filters with fixed parameters generate a set of response maps from an original image as a multi-scale and multi-orientation representation. Besides, the condensing module conducts max-pooling across channels of every response map to further capture condensed and scale-invariant information. Then, condensed response maps, i.e. the outputs of the above-mentioned Gabor convolution unit, are encoded both within and across channels. LBP (Local Binary Pattern) encoding method is leveraged to describe texture distribution in every channel of a condensed map, and Hash binary encoding is used to capture the relations across map channels. The binarization in encoding module helps the representation be robust to local changes. Thereafter, histogram calculation is applied to encoded maps to obtain statistical features, known as basic features. The aforementioned Gabor convolution unit, including encoding module and histogram calculation, forms a basic Gabor layer. In addition, this Gabor layer is provided with two extensive modules. One is to futher explore the invariance of texture distributions, and the other is to enrich the pattern of response maps. The former restores and encodes the indices of max responses in the Gabor convolutional unit. The latter holisticly learns a set of projection vectors from condensed response maps to construct a feature space. Once being projected into this feature space, the texture representation not only becomes more seperable, but also carries more patterns. The extensive features improve the robustness and discriminative ability of basic features since holistic information and more patterns are characterized. At last, smoke features of a Gabor layer are generated by concatenating basic features and extensive ones. Through stacking several Gabor layers on top of each other, a feedforwad network, termed GaborNet, can be built. Consequently, the concatenation of features acquired from every Gabor layer constitutes multi-scale, multi-orientation and hierachical features. As a network goes deeper, the features becomes more high-level and less explicable. Thus, the extension, which explicitly improves basic features, is conducted only on the first Gabor layer that carries low-level features. Besides, once holistic learning in extension is implemented, this step is no required any more in subsequent steps. Result This paper conducted ablation experiments to gain insights to the extensive features. Then, comparison experiments for smoke recognition were carried out to present the performance of the proposed GaborNet. Since this algorithm utilizes texture representations to present smoke, texture classification was also conducted as a supplement to the experiment. Experimental results demonstrate that the proposed GaborNet achieves powerful generalization ability. Smoke features extracted by the GaborNet descrease false alarm rates and error rates without dropping detection rates, thus the result of GaborNet ranks first among state-of-the-art methods. While results of texture classfication rank at first and second place respectively in two widely-used texture datasets. In summary, the GaborNet provides better texture representation than most of the existing texture descriptors in both smoke recognition and texture classification. Conclusion The proposed GaborNet can extract multi-scale, multi-orientation and hierachical representations for textures, and consequently helps improve the performance of smoke recognition and increases the accuracy of texture classification. Future researches should focus on eliminating the redundancy in features to gain compactness, and on exploring and utilizing the relations between features in different layers to enchance transform invariance. Eventually, this method is expected to be practically applied in real-time video smoke recognition.