语义分割是计算机视觉领域的一项像素级别的感知任务，目的是为图像中的每个像素分配相应类别标签，有广泛的应用。许多语义分割网结构复杂，计算量和参数量较大，在对高分辨率图像的进行像素层次的理解时具有较大的延迟，这极大的限制了其在资源受限环境下的应用，如自动驾驶、辅助医疗、移动设备等；因此，实时推理的语义分割网络得到了广泛的关注。本文对深度学习中实时语义分割算法进行了全面的论述和分析。首先，介绍了语义分割和实时语义分割任务的基本概念、应用场景和面临的问题。其次，详细介绍了实时语义分割算法中常用的技术和设计，包括模型压缩技术、高效CNN模块和高效Transformer模块。再次，全面地整理和归纳了现阶段的实时语义分割算法，包括单分支网络、双分支网络、多分支网络、U型网络、神经架构搜索网络五种类别的实时语义分割方法，涵盖基于CNN、基于Transformer、基于混合框架的分割网络，并分析了各类实时语义分割算法的特点和局限性。接着，提供了完整的实时语义分割评价体系，包括相关数据集和评价指标、现有方法性能汇总、以及领域主流方法的同设备比较，为后续的研究者提供了统一的比较标准。最后，给出结论并分析了实时语义分割领域仍存在的挑战，对实时语义分割领域未来可能的研究方向提出了相应的见解。本文提及的算法、数据集和评估指标已汇总至https://github.com/xzz777/Awesome-Real-time-Semantic-Segmentation，以便后续研究者使用。

Semantic segmentation is a fundamental task in the field of computer vision which aims to assign a category label to each pixel in the input image. Many semantic segmentation networks have complex structures, high computational costs, and massive parameters. As a result, they introduce significant latency when performing pixel-level scene understanding on high-resolution images. These limitations greatly restrict the applicability of these methods in resource-constrained scenarios, such as autonomous driving, medical applications, and mobile devices, etc. Therefore, real-time semantic segmentation methods, which produce high-precision segmentation masks with fast inference speeds, receive widespread attention. This paper provides a systematic and critical review of real-time semantic segmentation algorithms based on deep learning techniques to explore the development of real-time semantic segmentation in recent years. This paper covers three key aspects of real-time semantic segmentation: real-time semantic segmentation networks, mainstream datasets, and common evaluation indicators. In addition, this paper conducts a quantitative evaluation of the real-time semantic segmentation methods discussed and also provides some insights into the future development in this field. First of all, this paper provides an introduction to the task of semantic segmentation and real-time semantic segmentation tasks, as well as their application scenarios and challenges. The key challenge in real-time semantic segmentation mainly lies in how to extract high-quality semantic information with high efficiency. Secondly, this paper introduces in detail some preliminary knowledge for studying real-time semantic segmentation algorithms. Specifically, this paper introduces four kinds of general model compression methods: network pruning, neural architecture search, knowledge distillation, and parameter quantification. This paper also introduces some popular efficient CNN modules in real-time semantic segmentation networks, such as MobileNet, ShuffleNet, EfficientNet, and efficient Transformer modules, such as External Attention, SeaFormer, MobileViT, etc. Then, this paper comprehensively organizes and summarizes the existing real-time semantic segmentation algorithms. According to the characteristics of the overall network structure, existing works are categorized into five categories: single-branch network, two-branch network, multi-branch network, U-shaped network, and neural architecture search network. Specifically, the encoder of a single-branch network is a single-branch hierarchical backbone network, and its decoder is usually very lightweight and does not involve complex fusion of multi-scale features. The two-branch network adopts a two-branch encoder structure, using one branch to capture spatial detail information and the other branch to model semantic context information. Multi-branch networks are characterized by a multi-branch structure in the encoder part of the network, or a network with multi-resolution inputs, where the input of each resolution passes through a different sub-network. The U-shaped network has a contracting encoder and an expansive decoder which is roughly symmetrical to the encoder. Most works of these aforementioned four categories are manually designed, while the neural architecture search networks are obtained using network architecture search technology based on the four types of architectures These five categories of real-time semantic segmentation methods cover almost all real-time semantic segmentation algorithms based on deep learning, including CNN-based, Transformer-based, and hybrid-architecture-based segmentation networks. Moreover, we also introduce commonly used datasets and evaluation indicators of accuracy, speed, and model size for real-time segmentation. We divided popular datasets into the autonomous driving scene and general scene datasets, and the evaluation indicators are divided into accuracy indicators and efficiency descriptors. In addition, using relevant evaluation indicators, this paper quantitatively evaluates various real-time semantic segmentation algorithms mentioned on multiple datasets. To avoid the interference of different devices to conduct a quantitative comparison between real-time semantic segmentation algorithms, this paper compares the performance of advanced methods of each category with the same devices and configuration and establishes a fair and integral real-time semantic segmentation evaluation system for subsequent research, contributing to a unified standard for comparison. Finally, we discuss current challenges in real-time semantic segmentation and envision possible future directions for improvements, such as the utilization of transformers, applications on edge devices, the knowledge transfer of visual foundation models, diversity of evaluation indicators, variety of datasets, utilization of multi-modal data and weakly supervised methods, combination with incremental learning. The algorithms, datasets, and evaluation indicators mentioned in this paper are summarized at https://github.com/xzz777 /Awesome-Real-time-Semantic-Segmentation for the convenience of subsequent researchers.