摘要:3D Gaussian splatting (3DGS) has recently emerged as an efficient scene-representation paradigm for real-time rendering of complex 3D environments. By modeling scenes with anisotropic Gaussian primitives and optimizing them through differentiable rasterization, 3DGS achieves high rendering quality and computational efficiency compared with neural radiance fields (NeRF). Despite these advantages, standard 3DGS pipelines remain primarily appearance driven and lack explicit semantic understanding, which limits their usefulness in intelligent vision tasks. To address the fragmented state of current research, this paper proposes a three-stage taxonomy of semantic information flow, categorizing existing methods into semantic perception, semantic binding, and semantic querying. The survey systematically reviews more than 80 representative papers published between 2023 and 2026, comparing their data types, algorithmic structures, and performance indicators such as mean intersection over union (mIoU) and rendering speed. The first stage, semantic perception, focuses on acquiring language-aligned semantic features. Most existing methods rely on pretrained vision-language models (VLMs), including CLIP, BLIP, OpenCLIP, and SigLIP, to extract representations. These models provide strong open-vocabulary capability and generalization across diverse scenes. Beyond direct feature extraction, many approaches incorporate geometric cues to improve cross-view consistency. Depth maps, estimated surface normals, and epipolar constraints are used to guide semantic alignment across camera viewpoints, while cross-view self-distillation further suppresses semantic noise by enforcing consistency in overlapping regions. Through contrastive learning, multi-view fusion, and latent-space alignment, semantic perception modules build a unified embedding manifold that supports both visual coherence and linguistic flexibility, thereby establishing the foundation for downstream semantic reasoning. The second stage, semantic binding, concerns how semantic information is encoded, organized, and embedded into the continuous Gaussian representation. Current research emphasizes three main binding procedures: feature-distillation-based binding, 2D-3D mask-enhancement-based binding, and single-forward feedforward-network-based binding. 1) Feature-distillation-based binding adopts a teacher-student paradigm in which rich semantic features extracted by pretrained VLMs are distilled into the 3DGS representation. In these approaches, multi-view image features are projected or aggregated and then used to supervise Gaussian-associated semantic descriptors through distillation losses that encourage cross-view consistency. This strategy benefits from the generalization of large pretrained models and reduces dependence on per-scene annotation, but it can also inherit noisy or biased cues from the teacher and requires careful training to prevent semantic drift. 2) 2D-3D mask-enhancement-based binding emphasizes mask-level alignment between 2D renderings and 3D Gaussians. Rendered feature maps or preliminary 2D predictions are converted into semantic masks and lifted into 3D through visibility-aware back-projection or probabilistic fusion; conversely, 3D-consistent priors are used to refine 2D masks in an iterative enhancement loop. This bidirectional process improves localization, reduces artifacts caused by occlusion, and yields sharper instance boundaries and per-Gaussian semantic assignments. It is especially effective when high-quality 2D segmentations are available or can be jointly refined, although it depends heavily on accurate rendering and mask alignment. 3) Single-forward feedforward-network-based binding uses a dedicated lightweight network that predicts per-Gaussian semantic descriptors in a single forward pass from rendered multi-view features or geometric cues. This one-shot design prioritizes runtime efficiency and deterministic behavior: given current Gaussian parameters and rendered observations, the network outputs semantic vectors that are attached to Gaussians without iterative optimization. Such methods are appealing for real-time or interactive applications because of their low latency and fixed memory footprint, but they require carefully designed architectures and training procedures to generalize across scenes and preserve cross-view coherence. Collectively, these binding strategies reflect different trade-offs among semantic fidelity, computational cost, and implementation complexity. Feature distillation leverages large pretrained priors, mask enhancement enforces explicit geometric-semantic consistency, and one-shot feedforward binding prioritizes efficiency. The third stage, semantic querying, examines how enriched semantics within 3DGS can be retrieved, manipulated, and interpreted efficiently. Current work mainly distinguishes between 2D-space and 3D-space querying. In 2D-space querying, methods such as LangSplat, LEGaussians, and feature splatting compute similarity between text embeddings and rendered semantic feature maps. The resulting relevance heatmaps are then thresholded to obtain semantic masks. More advanced approaches, including SLGaussian and econSG, introduce cross-view memory banks or language-feature caches to improve query speed and stability in open-vocabulary segmentation. GOI further proposes an optimizable semantic hyperplane that adapts similarity thresholds automatically, making semantic querying differentiable and content adaptive. In 3D-space querying, methods operate directly in Gaussian latent space. Techniques such as OpenGaussian, Dr.Splat, and seconGS compute text-to-Gaussian similarity volumetrically, enabling multi-scale semantic retrieval in 3D. Some approaches also incorporate re-ranking or transformer-based refinement to improve discrimination among visually similar objects. Because they query semantics directly in 3D space, these methods reduce artifacts caused by partial occlusion and inconsistent view coverage and thus support more geometry-consistent semantic reasoning. Beyond static scenes, recent work has begun extending semantic expansion to dynamic 3D Gaussian splatting by introducing temporal coherence and motion-aware semantic propagation. These dynamic semantic representations support applications such as semantic tracking, action-driven editing, and language-guided dynamic scene interaction. Although this direction is still in its early stages, it is expected to drive the development of unified 4D semantic scene-understanding frameworks. In summary, this paper provides a structured review of semantic expansion methods in 3D Gaussian splatting through the lens of semantic information flow. By analyzing existing work in terms of semantic perception, semantic binding, and semantic querying, the survey clarifies the design principles and interdependencies of current methods. It also discusses key challenges, including multi-view semantic alignment, embedding efficiency, dataset bias, open-vocabulary scalability, and the balance between semantic richness and real-time performance. Finally, the paper highlights promising future directions such as cross-modal pretraining for Gaussian fields, lightweight semantic-binding strategies for mobile deployment, and hierarchical semantic querying, thereby supporting the development of next-generation semantic-aware 3D vision systems.
摘要:Multivariate time-series (MTS) classification is a foundational problem in modern data science and underpins critical applications ranging from intelligent transportation and industrial IoT monitoring to financial surveillance and medical diagnosis. Unlike univariate time series, MTS data exhibit two intertwined forms of dependency: temporal dynamics within each variable and complex spatial correlations among variables. Traditional approaches, such as dynamic time warping (DTW) with k-nearest neighbors or statistical feature-extraction methods, often suffer from poor scalability and strong dependence on manual feature engineering. Although deep-learning models such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs) automate feature extraction effectively, they are fundamentally designed for Euclidean data and therefore struggle to capture the complex non-Euclidean topological relationships—such as sensor networks, biological interactions, or stock-market linkages—that characterize many real-world systems. In recent years, graph neural networks (GNNs) have emerged as a transformative framework for structured data, representing variables as nodes and their interdependencies as edges. Through message passing, GNNs can explicitly model geometric structure and thereby capture spatial dependencies that conventional models often miss. However, despite the rapid growth of GNN-based methods for MTS tasks, the literature remains fragmented. Most existing surveys focus primarily on forecasting or imputation, with far less attention given specifically to classification. Moreover, the field lacks a unified theoretical framework for organizing the diverse strategies used to model coupled spatial and temporal features. To address this gap, this paper proposes a unified analytical framework, termed the “spatio-temporal dependency coupling paradigm”. Under this framework, existing models are categorized into three philosophical approaches. 1) The decoupled paradigm: this divide-and-conquer strategy treats spatial and temporal modeling as separate stages. Typically, a GNN extracts spatial features at each time step, and those features are then passed to a temporal model such as an LSTM or TCN. Although this approach offers strong interpretability and modularity, it often fails to capture the intricate simultaneous interactions between space and time. 2) The coupled paradigm: represented by models such as STGCN and DCRNN, this paradigm treats spatial and temporal dependencies as inseparable. By integrating graph convolutions directly with temporal convolutions or recurrent units, these models capture local spatio-temporal correlations more effectively, though often at the cost of greater parameter complexity. 3) The evolutionary paradigm: this frontier direction addresses the non-stationarity of real-world systems. Unlike early models that assume a static graph structure, methods such as EvolveGCN and TodyNet explicitly model the dynamic evolution of the graph itself, reflecting the fact that relationships among variables can change over time—for example, when traffic patterns shift after an accident or stock correlations reverse during a financial crisis. A critical determinant of GNN performance in MTS classification is how the graph structure is defined, especially when no explicit topology is available. We systematically review four graph-construction strategies: 1) structure-driven methods, which rely on prior knowledge such as physical road networks in traffic tasks; 2) similarity-driven methods, which define edges using statistical criteria such as Pearson correlation or DTW; 3) attention-driven methods, which use mechanisms such as graph attention networks (GATs) to learn edge weights dynamically; and 4) latent graph learning, which infers hidden dependency structures end to end without manual intervention, as in MTSGNN. This survey goes beyond taxonomy to provide a comprehensive analysis of applications in four major domains: transportation, healthcare, industry, and finance. We synthesize general principles for model selection. For example, in transportation and industry, where physical sensors determine topology, structure-driven static graphs are often robust and efficient. By contrast, in finance and healthcare, where relationships are abstract, invisible, and highly non-stationary—such as information flow among stocks or physiological interactions in ICU monitoring—dynamic and latent graph-learning strategies are often essential. We also highlight representative case studies, including the use of the concurrency hypothesis in traffic analysis and knowledge-empowered dynamic graphs (KEDGN) in medical diagnosis, to illustrate how domain knowledge can improve GNN performance.Despite rapid progress, several open challenges continue to hinder the wider adoption of GNNs for MTS classification. 1) Scalability: the computational cost of graph construction remains a bottleneck for large-scale systems such as massive IoT networks. 2) Heterogeneity and non-stationarity: real-world data frequently involve irregular sampling and distribution shifts, or concept drift, which static models cannot handle well. 3) Interpretability versus performance: although GNNs offer richer topological insight than CNNs, deep GNNs and current explanation methods still struggle to provide causal interpretations of their predictions. Looking ahead, we propose a research agenda focused on causal spatio-temporal representation learning. Future models should move beyond capturing correlations and toward identifying causal structure, thereby improving robustness to spurious dependencies. We also discuss the promise of integrating large language models (LLMs) to infer semantic relationships among variables, self-supervised learning to address label scarcity, and privacy-preserving federated graph learning for sensitive medical and financial data. This paper provides a holistic overview of the GNN-based MTS classification landscape, bridging the gap between theoretical modeling and practical application. By clarifying the trade-offs among model flexibility, computational efficiency, and dynamic adaptability, we aim to guide researchers in the selection and design of next-generation intelligent systems. To facilitate further research and reproducibility, the algorithms, datasets, and evaluation metrics discussed in this paper have been summarized and are available at https://github.com/sunjieseu/GNN-MTSC-Survey.
关键词:multivariate time series classification(MTSC);graph neural network(GNN);spatio-temporal data mining;graph representation learning;deep learning
摘要:ObjectiveHigh-resolution hyperspectral imaging is critically important across diverse application domains, including remote sensing monitoring, precision agriculture, mineral exploration, and environmental assessment. However, current imaging sensors face an inherent trade-off between spatial and spectral resolution due to physical constraints and manufacturing limitations. Specifically, increasing the number of spectral sampling channels reduces the light energy received by individual detectors, thereby decreasing the signal-to-noise ratio and limiting improvements in spatial sampling density. Conversely, enhancing spatial resolution requires reducing the number of spectral channels to maintain sufficient signal strength. Therefore, existing sensors can typically provide either high-spatial-resolution multispectral images (3–10 bands) or high-spectral-resolution images with low spatial resolution (dozens to hundreds of bands), making it difficult to simultaneously achieve both high spatial and high spectral resolution.MethodTo address the limitations of existing hyperspectral and multispectral image fusion algorithms, particularly the insufficient spectral reconstruction strategies and the lack of effective modality fusion mechanisms, this paper proposes HMIF-SRNet (hyperspectral and multispectral image fusion with sparse reconstruction networks), a novel fusion framework based on sparse reconstruction networks. The proposed method adopts an end-to-end architecture that achieves high-quality fusion through four carefully designed modules: preprocessing, feature extraction, core fusion, and feature reconstruction. The preprocessing module standardizes and aligns the two modalities via bicubic interpolation upsampling and spectral response function mapping, ensuring consistency in both spatial and spectral dimensions. The feature extraction module employs a dual-branch parallel architecture to process hyperspectral images (HSI) and multispectral images (MSI) separately. The HSI branch captures joint spectral-spatial features using multi-layer three-dimensional convolutions, while the MSI branch focuses on extracting spatial details through two-dimensional convolutions with residual connections, thereby maintaining gradient flow and accelerating network convergence. To ensure efficient utilization and precise preservation of spectral information, as well as effective complementarity between modalities, two key modules are designed: the spectral reconstruction unit (SRU) and the cross-reference unit (CRU). The SRU addresses the lack of effective sparse reconstruction strategies in the spectral domain by introducing a dual statistical gating strategy combined with a dynamic sparse reconstruction mechanism. It first performs global statistical analysis on fused features using both global average pooling and global maximum pooling to capture average response intensity and peak response characteristics. These complementary statistics provide rich information for subsequent weight learning. A multi-layer perceptron (MLP) is then used to learn adaptive spectral channel weights, enabling dynamic adjustment of each channel’s contribution. Finally, a spatial gating mechanism is incorporated to form a dual-gating strategy that jointly regulates spectral channels and spatial positions, ensuring both efficient utilization and high-fidelity preservation of spectral information. The CRU addresses the absence of explicit guidance mechanisms in cross-modal fusion by introducing a cross-guidance strategy for complementary feature integration. The module takes three inputs: the original F_hsi and F_msi from the feature extraction stage (used for global context modeling), and the optimized features F_stage1 generated by the SRU. Global context features are first computed via global average pooling, producing channel-wise descriptors that reflect the statistical properties of each modality. These descriptors are then passed through a two-layer linear transformation with nonlinear activation to generate cross-guidance weights, which encode the relative importance of each modality across channels. Using these weights, the CRU performs channel-wise weighted fusion of SRU-optimized features and downsampled MSI features, dynamically adapting the fusion strategy to different scenes and achieving an optimal balance between spectral fidelity and spatial detail. The SRU and CRU modules are cascaded to enable progressive feature refinement. First, the SRU performs dynamic sparse reconstruction in the spectral domain using a frequency decomposition strategy that separates features into high-frequency and low-frequency components. The high-frequency branch preserves fine spectral details through spatial attention and dynamic sparse selection, while the low-frequency branch captures the main spectral structure via global max pooling. These branches are then dynamically fused and downsampled to produce features with enhanced spectral representation. This stage primarily reduces spectral redundancy and selects the most discriminative features. Subsequently, the CRU performs cross-modal fusion by generating channel-wise adaptive weights based on global context and applying them to combine SRU-optimized features with MSI features. The final output integrates the spectral fidelity provided by SRU with the spatial detail contributed by MSI.ResultComparative experiments with 10 mainstream fusion methods on the CAVE and Harvard standard datasets demonstrate that HMIF-SRNet achieves superior performance in terms of PSNR, ERGAS and UIQI. For the spectral angle mapper(SAM) metric, the proposed method obtains the best result among deep learning-based fusion approaches, with traditional transformation methods yielding lower SAM values yet poor overall reconstruction quality. On the cave multispectral image database(CAVE) dataset, PSNR improves by 4.41 dB (a relative improvement of 9.4%) compared to the second-best method SSRNet, reaching 51.124 4 dB. ERGAS decreases by 9.5% to 0.472 6, SAM is reduced by 32.0% to 2.014 3, and UIQI increases to 0.983 2. On the Harvard dataset, which represents more complex real-world scenarios, HMIF-SRNet maintains superior performance, with ERGAS decreasing by 4.44% compared to the second-best method SSFCNN and UIQI reaching 0.942 2. Ablation studies further confirm the effectiveness of each core module, demonstrating that both SRU and CRU contribute significantly to performance improvements across all metrics.ConclusionThrough a lightweight network design, the proposed method effectively reduces computational complexity while maintaining excellent fusion performance. Visualization results show that HMIF-SRNet produces fusion images that are closest to the ground truth, with minimal error distribution, demonstrating strong robustness in real-world scenarios. Overall, the results validate the effectiveness of the dynamic sparse reconstruction strategy and the cross-guidance fusion mechanism, confirming that the proposed method outperforms existing techniques in spectral fidelity, spatial detail preservation, and overall fusion quality.
摘要:ObjectiveImage steganography is a key branch of information-hiding technology whose core objective is to embed secret information into cover images for covert communication, and it is important in areas such as military communication and privacy protection. In recent years, robust image-steganography methods based on encoder-decoder networks have made notable progress in balancing payload capacity, robustness, and visual quality through end-to-end training. However, most existing methods focus mainly on global visual-quality optimization and lack adaptive capacity for local spatial features, which leads to visible artifacts in smooth regions and loss of detail in textured regions. At the same time, limited error-correction capability reduces the reliability of information recovery under complex channel interference. To address these issues, this paper proposes a robust image-steganography framework with multi-adversarial guidance and hash-based error correction.MethodBuilding on the TransStego architecture, the proposed method introduces a steganalyzer that, combined with a discriminator, forms a multi-adversarial guidance mechanism. Through adversarial training, the encoder learns to exploit the steganalyzer’s high sensitivity to smooth regions and to embed secret information preferentially into textured regions with lower perceptual sensitivity. In this way, the discriminator preserves global visual fidelity, while the steganalyzer suppresses local statistical anomalies. Their collaborative optimization significantly reduces embedding artifacts and improves both visual quality and statistical invisibility in the resulting stego images. In addition, a dual-level error-correction mechanism based on Bose-Chaudhuri-Hocquenghem (BCH) coding and hash mapping is designed. At the sender side, the secret message is first protected with BCH coding. If residual errors remain beyond the correction capability of BCH, the error-location information is encoded into a robust hash sequence through a pre-synchronized hash-generation model and a shared image database. This hash sequence is transmitted together with the stego image after matching the corresponding natural image in the database. To avoid hash collisions, an unambiguous hash-mapping scheme is further introduced: the hash-generation model first produces initial hash sequences for all database images and sorts them in ascending order; for subsets with collisions, secondary ascending sorting based on average pixel value is used to resolve ambiguity, while the remaining images retain their original order. All images are then merged into a unified sequence, assigned consecutive unique IDs, and mapped to binary sequences as final hash codes. At the receiver side, the system uses both hash values and average pixel values for precise localization and combines this information with BCH decoding to recover the secret message accurately even under channel interference.ResultWe evaluate the proposed method in terms of imperceptibility, robustness, security, and ablation. In imperceptibility, it outperforms TransStego with a 5.89% increase in peak signal-to-noise ratio (PSNR), a 3.11% improvement in multi-scale structural similarity index (MS-SSIM), and a 30.21% reduction in learned perceptual image patch similarity (LPIPS). Under common non-geometric attacks, including Gaussian blur, median filtering, Joint Photographic Experts Group (JPEG) compression, and image scaling, the method achieves near-perfect secret extraction accuracy. Under more challenging geometric attacks such as affine transformation, rotation, rotation with cropping, and rotation with scaling and cropping, it still maintains strong recovery capability and attains an average extraction accuracy of 79.19%. This exceeds four representative baselines—TransStego (52.97%), StegaStamp (54.43%), RoSteALS (47.32%), and FNNS-R (49.97%)—by an average of 28.02 percentage points. Notably, under the severe composite attack of 60° rotation followed by scaling and cropping, the proposed method improves extraction accuracy by 30.88 percentage points over TransStego. In security evaluation against the advanced steganalyzer UCNet, it achieves the lowest and most stable detection rate, outperforming PRIS, FNNS-R, and TransStego. Ablation studies show that the two-stage error-correction mechanism improves robustness, while steganalysis-guided embedding enhances both imperceptibility and security. Their combination yields the highest extraction accuracy across diverse attacks, with performance gains exceeding 30% under geometric perturbations.ConclusionThis method achieves synergistic optimization of high visual fidelity and strong robustness under complex interference through steganalysis-guided local adaptive embedding and a BCH-hash dual-level error-correction mechanism. By using the steganalyzer as an embedding guide, it directs secret information toward more disturbance-tolerant texture regions and reduces distortion through gradient-based feedback, while the dual-level error-correction strategy improves extraction accuracy. Experimental results demonstrate that this design significantly improves the visual quality and robustness of stego images, offering a new paradigm for constructing practical and highly reliable robust image-steganography systems.
摘要:ObjectiveOccluded person re-identification (Re-ID) aims to match individuals across non-overlapping camera views despite frequent partial obstruction, making it a challenging problem for intelligent surveillance and public security. The key difficulties arise from two sources: the loss of discriminative identity information in occluded regions and the intrusion of noise from occluders. Although many methods have been proposed, most remain limited to single-view paradigms and therefore struggle to model channel-wise feature importance adequately, avoid overfitting to the most salient regions, and enforce stable feature selection under severe visual ambiguity. These limitations reduce both robustness and generalizability.MethodTo address these issues holistically, this paper proposes a novel multi-view gated and soft-regularized network (MGSR-Net), a synergistic framework designed to improve feature discriminability, diversity, and stability. MGSR-Net consists of an efficient single-view inference branch and a robust multi-view regularization branch, both of which share a squeeze-and-excitation ResNet (SE-ResNet) backbone for channel-aware feature enhancement. The first key component is the Top-k saliency-guided contextual erasure (TSCE) module. Unlike random erasure or hard masking, TSCE uses class activation maps (CAMs) to identify the top-k most influential spatial regions and then applies soft suppression to reduce their dominance. This encourages the network to learn from secondary but complementary regions, thereby alleviating overfitting and improving feature diversity. The second key component is a gated regularization mechanism that imposes an entropy constraint on channel-selection probabilities. This encourages sparse and stable feature-channel selection so that the model consistently relies on a robust set of core channels even under noisy or occluded conditions. During training, features from multiple views are processed by TSCE, weighted by stabilized gates, and fused. A feature-alignment loss then distills the resulting robust multi-view knowledge back into the single-view branch through L2 minimization, enabling efficient single-view inference at test time. The total loss combines identity-classification losses for both branches, the alignment loss, and the gate-regularization loss.ResultExtensive experiments validate the effectiveness of MGSR-Net. On the Occluded-REID dataset, it achieves state-of-the-art performance with 83.4% mean average precision (mAP) and 92.8% Rank-1 accuracy, surpassing the previous leading multi-view method, multi-view information integration and propagation (MVIIP), by 1.7% mAP and 4.8% Rank-1. On P-DukeMTMC and Market-1501, the model reaches 89.3% Rank-1/77.6% mAP and 95.0% Rank-1/88.0% mAP, respectively, demonstrating strong transferability. Most notably, on the large-scale, multi-scene MSMT17 dataset, MGSR-Net achieves 89.6% Rank-1 and 71.4% mAP, outperforming MVIIP by 5.7% Rank-1 and 10.0% mAP, which highlights its exceptional generalization ability. Ablation studies confirm the contribution of each component: SE-ResNet provides a strong channel-enhanced backbone, TSCE is crucial for feature diversity, gated regularization stabilizes channel selection, and the alignment loss enables effective knowledge distillation from the multi-view branch to the single-view branch. Visualizations further show that MGSR-Net produces more holistic and adaptive attention maps than competing methods, shifting focus to visible informative regions more effectively under heavy occlusion.ConclusionThis paper proposes MGSR-Net, an integrated framework that combines channel enhancement, saliency-guided soft feature perturbation, and entropy-constrained feature-selection stabilization within a multi-view learning paradigm. It effectively overcomes the limitations of single-view modeling, mitigates overfitting, and promotes robust feature selection under occlusion. Extensive results on both occlusion-specific and large-scale general benchmarks show that MGSR-Net establishes a new state of the art, offering improved discriminability, robustness, and generalization. Its design philosophy also provides a valuable reference for other vision tasks affected by partial information loss. Future work will explore more adaptive multi-view fusion strategies.
关键词:Occluded person re-identification;Discriminative feature mining;Channel attention enhancement;Saliency-guided erasure;Gated regularization
摘要:ObjectiveMultimodal emotion recognition aims to understand the emotional states of specific subjects by fusing data from multiple modalities, such as text, vision, and audio. However, the inherent heterogeneity in the representation forms and distribution laws of different modalities leads to emotional semantics in the latent feature space often being entangled and coupled with modality-specific, non-emotional noise. This feature entanglement phenomenon not only hinders the model’s effective learning of key emotional features but also reduces the interpretability of its decision-making process. Furthermore, existing feature fusion strategies adopt simple concatenation operations or coarse-grained attention mechanisms, making it difficult to successfully capture fine-grained emotional semantic interaction cues between modalities in complex cross-modal contexts, ultimately resulting in fused emotional representations that lack sufficient discriminability. To this end, this study introduces an interpretable invertible disentanglement and adaptive fusion method for multimodal emotion recognition.MethodFirst, aiming to minimize semantic information loss during the feature learning phase and achieve structured feature disentanglement, an invertible attention mask-based disentanglement (IAMD) module is designed. Based on invertible neural networks (INNs), a bidirectional invertible mapping structure is constructed between the latent representations of each modality’s features and emotional semantic factors, and an attention mask mechanism is integrated to disentangle latent features in the channel dimension into two parts: one part captures shared features with semantic consistency across modalities, and the other part retains specific features containing unique attributes of each modality. Second, a mutual information constraint (MIC) mechanism is constructed to further enhance the disentanglement effect from an information-theoretic level. The semantic consistency of emotional features in the shared subspace is enhanced by calculating and maximizing the mutual information between shared features as well as that between shared features and emotion labels. Meanwhile, by minimizing the mutual information between specific features and emotion labels conditioned on shared features, the model is constrained to remove modality-specific attributes that are not directly related to the emotion task into the specific feature subspace. This approach reduces the interference of modality-redundant noise on emotional semantics. Finally, a semantic-guided adaptive feature fusion (SGAFF) module is designed to address the limited interaction during the feature fusion phase. The cross-modal consistent emotional semantic information captured in the shared subspace is leveraged as contextual cues to perform fine-grained semantic correction of modality-specific features through residual connections. A dual-branch prediction structure is also constructed, in which a gating mechanism adaptively assigns weights to the shared branch and the specific-guided branch, thereby enhancing the discriminability of the fused representation.ResultExtensive comparative experiments and ablation studies were conducted on the CMU-MOSI, CMU-MOSEI, and UR-FUNNY datasets. Specially, on the CMU-MOSI dataset, compared with the DLF (disentangled language focused) model, our model reduces the mean absolute error (MAE) by 2.4% and improves the 7-class accuracy (Acc-7) by 2.9%. On the CMU-MOSEI dataset, compared with the TMBL (Transformer-based multimodal binding learning) model, the MAE is reduced by 2.6% and the Pearson correlation coefficient (Corr) is improved by 1.7%. Furthermore, on the UR-FUNNY dataset, the model achieved an improved F1-score (F1) of 5.6% compared to the modality-invariant and specific analysis model. Additionally, detailed ablation experiments verified the necessity of the IAMD, MIC, and SGAFF modules for improving model performance. Feature visualization analysis using t-distributed stochastic neighbor embedding confirms that the model successfully separates emotional semantics and modality noise in the latent space. Furthermore, visualization of fusion weights confirms that the model adaptively assigns higher contributions to the specific-guided branch, confirming the role of fine-grained complementary cues in final emotion judgment.ConclusionOverall, the proposed method achieves interpretable disentanglement of emotional semantic information from modality-specific noise based on INNs and mutual information constraints. Meanwhile, through a semantic-guided adaptive fusion strategy, the method realizes deep and fine-grained interactions among cross-modal emotional semantic features, thereby improving the accuracy and robustness of multimodal emotion recognition tasks in complex scenarios. Although the proposed method achieves remarkable progress in model interpretability, the introduction of invertible transformations and multiple mutual information constraints increases its computational complexity. This method is applicable to multimodal scenarios with complex modal heterogeneity, as well as tasks that have strict requirements for quantitative emotion recognition metrics. Future work will focus on lightweight disentanglement strategies for emotion recognition tasks, aiming to further improve inference efficiency and model generalization in scenarios with limited computational resources. The source code has been archived at https://doi.org/10.57760/sciencedb.j00240.00138.
摘要:ObjectiveVisual object tracking is a fundamental capability in computer vision and underpins important applications such as autonomous driving and unmanned aerial vehicle surveillance. Although Transformer-based trackers have demonstrated strong performance under normal illumination, their effectiveness drops sharply at night. This degradation stems mainly from two factors. First, low-light conditions cause the loss of texture detail and edge information, leading to feature degradation and making it difficult for the feature extractor to distinguish the target from the background. Second, nighttime scenes often contain high-frequency background noise and intense distractors such as streetlights and vehicle headlights. Because these distractors can exhibit visual characteristics similar to those of the target, the tracker may incorrectly assign them high confidence, causing a phenomenon known as high-score drift. Once this occurs, the dynamic template becomes contaminated and tracking may fail irreversibly. Existing solutions fall mainly into two categories: low-light enhancement and domain adaptation. Enhancement-based methods introduce extra image-processing modules, increasing latency and sometimes amplifying noise. Domain-adaptation methods attempt to align nighttime features with daytime domains, but they often fail to preserve the fine-grained geometric structure required for precise localization. Although recent prompt-learning approaches such as DCPT try to inject darkness cues into trackers, they mainly operate at the input level. As features propagate through deep Transformer layers, these prompts become diluted and lose their semantic guidance capability. Existing methods also lack dedicated mechanisms for correcting attention dispersion caused by bright background distractors. To address these limitations, this study proposes a framework called nighttime object tracking via cross-layer consistent darkness perception prompts (ProDAPT). Its primary goal is to bridge the gap between feature extraction and decision making in low-light scenarios by constructing a full-link darkness-perception mechanism within a frozen Transformer backbone.MethodProDAPT is built on the ProContEXT baseline and uses a standard ViT-Base backbone pretrained on daytime datasets. To preserve generalization and maintain parameter efficiency, the backbone remains frozen and only the newly introduced prompt-related modules are fine-tuned. The method contains three main innovations. First, a cross-layer consistent prompt generator is introduced to counter feature dilution. Unlike methods that rely on simple pixel-level superposition, this generator uses an iterative back-projection mechanism composed of emphasizing and undermining blocks to estimate the residual difference between target structure and background noise. A cross-layer semantic constraint is further imposed by enforcing consistency between deep-layer prompts and shallow-layer prompt projections. This encourages the network to preserve fine-grained geometric cues such as edges and shapes from shallow layers and transmit them to deeper semantic layers, thereby recovering target details buried in noise. Second, a prompt semantic calibration attention module is designed to mitigate attention dispersion. Under low illumination, standard self-attention often assigns high weights to bright background distractors. This module integrates generated prompts into self-attention by computing the cosine similarity between template prompts and search-region prompts. The resulting structural correlation matrix is injected into the original query-key attention map as an explicit structural bias. By increasing the attention weights of regions with similar prompt structure, the module suppresses responses to bright distractors even when they have high pixel intensity. Third, an energy-aware dual gating update strategy is proposed to improve template updating. Conventional update mechanisms rely only on classification scores, which are unreliable at night. ProDAPT introduces prompt energy, computed using the norm of prompt features, as an additional metric for structural integrity. The underlying idea is that a true target maintains high prompt energy because of cross-layer structural consistency, whereas background noise yields low energy after prompt filtering. A dual-gating rule permits template updates only when both the classification score and prompt energy exceed predefined thresholds, thereby preventing template-bank contamination by high-confidence distractors.ResultExperiments were conducted on three mainstream nighttime tracking benchmarks—NAT2021, LLOT, and UAVDark135—and on a self-collected real-world dataset recorded by autonomous-driving test vehicles in Suzhou, China. The self-collected dataset includes 4 K Ultra-HD video (3 840 × 2 160 pixels at 30 frames/s) captured by high-specification onboard cameras and covers challenging conditions such as glare from oncoming vehicles, rain reflections on road surfaces, and temporary occlusion during car-following. Across the evaluated benchmarks, ProDAPT achieved competitive performance. On NAT2021, the success rate (AUC) reached 0.557, which is 3.1% higher than that of the prompt-learning baseline DCPT and better than competing domain-adaptation methods. On LLOT and UAVDark135, the AUC scores reached 0.585 and 0.608, respectively. Ablation studies confirmed that the cross-layer consistent prompt generator, prompt semantic calibration attention, and energy-aware dual gating update modules each contribute to performance, with the full model achieving the best center precision and overlap accuracy. In terms of efficiency, ProDAPT runs at 76.8 frames/s on an NVIDIA A100 GPU, exceeding the real-time threshold of 30 frame/s. Although the additional modules introduce a modest speed reduction relative to the baseline, the trade-off is justified by the accuracy gains. Only 6.51 million parameters are trained, accounting for 7.03% of the total model parameters, which reflects a favorable balance between performance and computational cost compared with full fine-tuning. Qualitative results on the 4K self-collected dataset further show that ProDAPT maintains stable bounding boxes on leading vehicles in the presence of severe glare, whereas baseline trackers drift toward nearby streetlights.ConclusionThis study presents ProDAPT, a unified framework that addresses feature dilution and decision unreliability in nighttime tracking. By introducing a cross-layer consistent prompt generator, the framework preserves darkness cues throughout deep network layers. Prompt semantic calibration attention and the energy-aware dual gating update strategy further improve robustness to distractors and prevent erroneous template updates. Extensive experiments on public benchmarks and high-quality self-collected 4K data demonstrate the effectiveness and generalization ability of the proposed method. ProDAPT therefore offers a feasible and efficient solution for nighttime visual perception in autonomous driving and unmanned aerial vehicle applications. Future work will focus on edge deployment on embedded devices and on incorporating multi-modal data, such as infrared or thermal imaging, into prompt generation to further improve tracking reliability under extremely dark conditions.
摘要:ObjectiveThe advancement of autonomous driving technology imposes stringent demands on precise environmental perception. While pure vision-based bird’s-eye view(BEV) 3D object detection has garnered significant attention due to its cost-effectiveness, it still confronts two major bottlenecks in dynamic traffic scenarios. Firstly, existing methods often inadequately exploit temporal information when processing multi-frame images, leading to insufficient perception capabilities for moving objects and occluded areas. Secondly, the prevalent fixed voxel sampling strategies struggle to adapt to the variability of scene complexity, failing to balance computational efficiency in sparse scenes with detection accuracy in dense, crowded environments. This paper aims to address these critical limitations by constructing an efficient and robust pure vision-based 3D detection framework specifically designed for dynamic traffic conditions.MethodThis paper proposes a novel framework for 3D object detection in dynamic traffic scenes, introducing two core technical innovations that address key challenges in temporal modeling and computational efficiency. The temporal grouping fusion module revolutionizes temporal feature integration by processing consecutive BEV features through a grouped architecture inspired by Res2Net principles. This module partitions temporal sequences into multiple groups, where features within each group are concatenated and compressed via shared 1 × 1 convolutions to generate compact representations. The breakthrough lies in the residual connections established between groups, allowing subsequent groups to directly inform preceding ones before applying shared 3 × 3 convolutions. This sophisticated design captures fine-grained temporal dependencies at local levels while enabling comprehensive information propagation across the entire temporal sequence. The architecture significantly enhances the model’s capacity to model complex dynamic interactions, particularly in handling scene transitions and occlusion patterns, while maintaining parameter efficiency through weight sharing across all processing groups. Complementing this, the adaptive voxel feature sampling strategy introduces an intelligent sampling mechanism governed by a comprehensive scene complexity assessment pipeline. The system begins by processing multi-view image features through a meticulous normalization procedure involving clamping, rectification, and maximum-value normalization. It then computes both global feature intensity (mean) and local texture complexity (standard deviation) statistics, which are intelligently fused using optimized weighting coefficients. The framework incorporates temporal consistency through historical reference integration and exponential moving average smoothing, producing robust complexity scores that reliably classify scenes into “simple” or “complex” categories. This classification dynamically orchestrates the sampling strategy, seamlessly switching between computationally efficient Fast-Ray transformations for straightforward scenarios and sophisticated deformable attention mechanisms for challenging environments. In the integrated framework, multi-camera images are used as input. Surround-view images first pass through a backbone network to extract multi-scale 2D(two-dimensional) features, which are then fed into a feature pyramid network (FPN) and an adaptive voxel feature sampling strategy respectively. In the adaptive voxel feature sampling strategy, a scoring function is employed to calculate the complexity of the current frame, thereby adaptively selecting the sampling method. Secondly, the outputs of the two branches are lifted from multi-view 2D perspective features to the 3D space through geometric projection in the 2D-3D module to generate a BEV representation. Finally, after decoding by the temporal grouping fusion module and the 3D detection head, the final object detection results are obtained. The adaptive voxel feature sampling strategy effectively balances the trade-off between detection accuracy and computational efficiency. In addition, the temporal grouping fusion module further enhances the model’s stability in dynamic traffic environments and its perception capability for occluded objects.ResultExtensive experiments were conducted on the large-scale public autonomous driving dataset, nuScenes. The evaluation adheres to the official nuScenes metrics, including mean average precision (mAP) and the comprehensive nuScenes detection score (NDS). Under a lightweight configuration using ResNet50 as the backbone network and an input resolution of 256 × 704, our proposed method achieved a mAP of 36.2% and an NDS of 49.5%. This performance surpasses contemporary state-of-the-art pure vision methods such as Fast-BEV (35.4% mAP, 48.7% NDS), and demonstrates substantial improvements over other mainstream methods like BEVDet and BEVDepth. Under a higher-performance configuration employing a ResNet101 backbone and an input resolution of 900 × 1 600, our method’s performance was further elevated, reaching 42.9% mAP and 55.8% NDS, ranking it among the top-performing approaches in the comparison. Ablation studies systematically validated the individual and collective contributions of the two core modules. Removing either module led to a noticeable performance drop, confirming that both the temporal grouping fusion and the adaptive sampling strategy are pivotal to the overall success. Qualitative results from visualization further corroborated the quantitative findings, demonstrating that our method produces more complete detections and more accurately aligned 3D bounding boxes, particularly in challenging scenarios involving occlusions and fast-moving objects, compared to the baseline model.ConclusionThis paper addresses two fundamental challenges in vision-based 3D object detection for dynamic traffic scenes: the effective utilization of temporal information and the adaptive adjustment of feature sampling under varying scene complexities. To this end, we propose a unified and efficient framework integrating a temporal grouping fusion module and an adaptive voxel feature sampling strategy. The proposed temporal grouping design enables fine-grained feature interaction and multi-scale information propagation across consecutive BEV representations, ensuring consistent spatiotemporal perception even in complex and occluded environments. Meanwhile, the adaptive sampling mechanism dynamically aligns computational effort with scene complexity, thereby achieving a more balanced trade-off between efficiency and accuracy. Comprehensive experiments conducted on the nuScenes benchmark confirm the superior generalization and robustness of our framework. The proposed approach not only surpasses existing pure vision-based methods in both mAP and NDS but also demonstrates remarkable scalability across different backbone architectures and resolutions. More importantly, it maintains real-time inference capability, showing great promise for deployment in resource-constrained autonomous driving systems. Beyond quantitative results, qualitative visualization further illustrates the framework’s ability to preserve object integrity, refine 3D box alignment, and maintain temporal consistency during rapid motion or heavy occlusion. In future work, we plan to enhance the framework by introducing region-aware adaptive mechanisms and shorter temporal fusion pipelines for low-latency applications. Moreover, integrating self-supervised temporal learning and multi-modal knowledge distillation could further improve the robustness and cross-domain adaptability of the system. Ultimately, this research provides a feasible, scalable, and cost-effective perception paradigm, contributing to the broader goal of realizing safe, intelligent, and human-centered autonomous driving technology. The source code has been https://www.scidb.cn/s/rA7ZFf.
摘要:ObjectiveIn recent years, autonomous driving technology has developed rapidly, with the main objective of reducing driver workload and enhancing driving safety. Within autonomous driving systems, 3D object detection plays a crucial role in the environmental perception module, as its performance directly affects the accuracy and reliability of subsequent path planning and decision-making. Among existing 3D object detection tasks, LiDAR-based methods have attracted considerable attention in academia and industry because they can directly capture high-precision spatial geometric information and exhibit strong robustness to environmental disturbances such as lighting changes and adverse weather conditions. Among various LiDAR-based detection methods, pillar-based approaches have been widely adopted in practical autonomous driving scenarios due to their high computational efficiency and real-time performance. These methods employ a unique pillarization process, in which original unstructured point cloud data is first discretized into a series of regularly arranged vertical pillars. The resulting 3D pillar features are then aggregated and projected onto a 2D plane to generate pseudo-image representations. This transformation converts sparse and irregular 3D point clouds into structured 2D grid data, allowing the use of well-developed 2D convolutional neural networks for efficient feature extraction and object detection, thereby substantially reducing computational complexity. However, the process of mapping point clouds to 2D pseudo-images often leads to information loss due to compression and dimensionality reduction, resulting in the degradation of fine-grained geometric details in 3D space. This issue is particularly prominent in sparse point cloud regions and scenes involving small objects. Moreover, during pseudo-image feature extraction, conventional 2D convolutional neural networks are limited by their fixed local receptive fields, making it difficult to fully capture long-range dependencies and local features. This phenomenon results in insufficient feature representation, further constraining detection performance.Aiming to address these challenges, this paper proposes a 3D object detection model with pseudo-image features enhanced by dynamic spatial global attention.MethodFirst, a dynamic spatial global attention mechanism is designed. This module generates sample-adaptive convolutional kernels to support targeted modeling of local geometric structures. Meanwhile, a global self-attention mechanism is introduced to enhance spatial feature interactions and long-range dependencies. The combination of local and global representations helps preserve additional geometric details in pseudo-images, effectively mitigating information loss during the pillarization process and improving contextual awareness for small or occluded objects. Second, an efficient multiscale pseudo-image feature fusion network is constructed. The network adopts a three-stage hierarchical feature extraction structure to balance fine-grained local details and high-level semantic information, while incorporating a feature pyramid network for multiscale feature fusion. Specifically, during the feature fusion stage, a top-down upsampling and lateral connection strategy is employed to unify features from different scales to the same resolution, followed by channel-wise fusion. This design effectively integrates shallow spatial details with deep semantic representations, enhancing the overall representational capability of pseudo-image features and notably improving their suitability for downstream 3D detection tasks. Finally, a dual perception dynamic upsampling module is proposed. Based on DySample, an edge-awareness branch and a small-object saliency enhancement branch are introduced to jointly optimize the spatial detail recovery during upsampling. This design improves the modeling performance for small objects and complex boundaries in 3D object detection.ResultAiming to systematically evaluate the performance of the proposed method, comprehensive comparison experiments were conducted on two public datasets: Karlsruhe Institute of Technology and Toyota Technological Institute at Chicago (KITTI) and DAIR-V2X-V. On the KITTI dataset, the proposed model was compared with various representative 3D object detection methods, including multi-modal and LiDAR-based approaches. The compared methods include aggregate view object detection (AVOD), F-PointNet, VeloFCN, sparsely embedded convolutional detection (SECOND), Associate-3Det, VoxelNet, PointRCNN, HS-Pillar, TANet, CenterPoint, VoxelNeXt, MAT-PointPillars, PointBi-FPN, XPillars, and PointPillars. Experimental results show that, compared with the baseline model PointPillars, the proposed method achieves improvements in average precision (AP) for the car category of 1.31%, 2.51%, and 2.76% at easy, moderate, and hard difficulty levels, respectively. For pedestrian detection, the corresponding improvements are 6.96%, 6.07%, and 6.10%, while for cyclist detection, the improvements are 10.62%, 10.33%, and 9.81%. At the moderate difficulty level, the proposed algorithm achieves a mean average precision (mAP) of 65.44%, which is 6.3 percentage points higher than that of the baseline model. Similarly, on the DAIR-V2X-V test set, the proposed method is evaluated against several representative 3D object detection methods, including Part-A2, SECOND, PointRCNN, and PointPillars. The experimental results indicate that, compared with PointPillars, the proposed method improves AP for the car category by 0.08%, 0.07%, and 0.30% at easy, moderate, and hard difficulty levels, respectively. For pedestrian detection, the improvements are 7.81%, 6.48%, and 6.04%, while for cyclist detection, the improvements are 7.45%, 8.70%, and 8.64%. The overall mAP across all categories reaches 44.37%, which is 5.09 percentage points higher than that of the baseline model.ConclusionAiming to address the issues of geometric detail loss during the pillar-based projection of point clouds into pseudo-images, as well as the insufficient modeling during pseudo-image feature extraction, this paper proposes a 3D object detection model with pseudo-image features enhanced by dynamic spatial global attention. First, a dynamic spatial global attention mechanism is introduced, combining dynamic convolution kernel generation with a global self-attention mechanism to strengthen the spatial representation of pseudo-image features. Second, an efficient multiscale pseudo-image feature fusion network is designed to enhance the model’s capability to extract multiscale features while preserving fine-grained local details and global semantic information. Finally, a dual-perception dynamic upsampling module is proposed. The module facilitates sampling point distribution through a dual-path design that incorporates edge awareness and small-object saliency enhancement, optimizing the upsampling process and improving spatial detail recovery and detection performance. Experimental results on the KITTI and DAIR-V2X-V datasets demonstrate that the proposed algorithm achieves superior detection performance. Ablation studies and visualization results further verify the effectiveness of each module. Although the proposed method substantially improves detection accuracy, the dynamic spatial global attention computation and multiscale feature fusion mechanism result in increased model complexity and parameter size. Future work will focus on lightweight model designs to further enhance the algorithm’s real-time performance, robustness, and applicability in diverse and complex scenarios.
摘要:ObjectiveFacial action unit (AU) detection is a fundamental yet challenging task in affective computing and computer vision. Defined within the facial action coding system (FACS), AUs correspond to the subtle activation of specific facial muscles, providing objective indicators of human emotions, cognitive states, and mental health conditions. Automatic AU detection plays a critical role in a wide range of applications, including advanced emotion recognition systems, non-intrusive mental health monitoring, and natural human-computer interaction. However, despite considerable progress in recent years, existing methods still suffer from two primary limitations. First, spatial relationship modeling often relies on fixed, predefined priors or captures only shallow feature correlations. Such approaches are insufficient for capturing the complex and nonlinear co-activation and antagonistic patterns inherent in facial expressions, such as the synergistic relationship between AUs when smiling or the mutually exclusive nature of certain brow movements. Second, temporal dynamic modeling typically depends on computationally intensive techniques, such as optical flow or manually crafted constraints. These methods are not only prone to noise and tracking errors but are also difficult to deploy in real-time applications due to their high computational cost. Aiming to overcome these issues, a unified and lightweight spatiotemporal fusion framework for AU detection is introduced. This framework jointly captures intra-frame spatial dependencies and inter-frame temporal dynamics in an efficient and scalable manner.MethodThe proposed framework employs a ResNet-18 architecture as the backbone to extract robust per-frame facial feature representations. Based on these features, the model integrates two lightweight core modules: spatial relationship modeling (SRM) and temporal relationship modeling (TRM). The SRM module leverages graph neural networks (GNNs) to explicitly model intra-frame AU structures. Instead of relying on static priors, this module dynamically computes pairwise similarities between AU features via cosine similarity. A Top-K pruning mechanism is then applied to the resulting graph, retaining only the strongest connections to suppress spurious correlations and ensure a compact, interpretable, and efficient AU graph structure. Notably, this dynamic graph construction does not require additional anatomical labels. The TRM module focuses on temporal dynamics by first generating motion-sensitive features through simple frame-difference operations. This step effectively minimizes redundant texture information while highlighting regions of change. A temporal GNN is then applied to these motion features to propagate information across a sequence of frames, allowing the model to distinguish transient noise from genuine AU activations that continuously evolve over time. A spatiotemporal feature fusion (SFF) strategy is introduced to effectively integrate the complementary strengths of spatial context and temporal motion. This module concatenates the features from the SRM and TRM branches and employs an adaptive attention mechanism to dynamically balance their contributions for each input instance. Finally, AU recognition is performed using a cosine-similarity-based classification module, which is robust to feature scale. During training, a weighted cross-entropy loss function is employed to effectively address the pervasive class imbalance problem by assigning greater emphasis to the learning of infrequent AUs.ResultExtensive experiments were conducted to validate the proposed framework on two benchmark datasets: Binghamton protocol 4-dimensional (BP4D) and Denver intensity of spontaneous facial action (DISFA). Results indicate that the proposed approach achieves state-of-the-art or highly competitive performance while maintaining low computational and memory costs. On the BP4D dataset, the proposed model reached an average F1-score of 66.00%, outperforming recent methods such as RE-Net and AC2D. On the DISFA dataset, the framework achieved an average F1-score of 65.34%, closely approaching the best-reported result. Comprehensive ablation studies further confirmed the necessity and effectiveness of each module. Specifically, removing the SRM module substantially impaired the capability of the model to detect weak or infrequently occurring AU. Eliminating the TRM module degraded performance on dynamic AU recognition, while excluding the SFF fusion strategy weakened the model’s adaptability and overall feature integration efficiency. The results indicate that the synergy of SRM, TRM, and SFF is essential for achieving superior detection performance. Notably, the proposed framework excelled in detecting dynamic AUs, particularly AU6 (cheek raiser) and AU7 (lid tightener), demonstrating its effectiveness in modeling temporal consistency and motion patterns. Statistical tests further verified the importance of the improvements achieved over baselines. Additionally, these findings highlight that the model is particularly robust under varying illumination conditions and subtle expression changes, showcasing its strong generalization capability for unconstrained environments.ConclusionOverall, the proposed spatiotemporal fusion framework effectively integrates spatial and temporal modeling to enhance the accuracy and robustness of AU detection, while maintaining a lightweight and computationally efficient design suitable for real-world applications. The framework offers a reliable solution for AU detection in complex and dynamic environments and shows strong potential for deployment in affective computing and human-computer interaction. A key contribution of this model lies in its capability to dynamically balance spatial and temporal features based on the input context. Future research will emphasize fine-grained local feature modeling for specific AUs, multiscale temporal fusion strategies, and the integration of multimodal data (e.g., audio and text) to further improve generalization and performance in unconstrained real-world scenarios. Additionally, future studies will validate the framework across diverse demographic groups and clinical populations.
摘要:ObjectiveCo-speech holistic motion generation aims to simultaneously produce expressive gestures and accurately synchronized facial expressions. However, the two tasks have fundamentally different characteristics. Gesture generation is non-deterministic, representing a one-to-many mapping in which the same speech can correspond to various natural motions, thus requiring high diversity. Conversely, facial expression generation, especially lip movements, is deterministic and represents a one-to-one mapping that requires precise correspondence with phonemes, thus demanding high accuracy. Existing methods suffer from three critical limitations. First, numerous methods employ fixed architectural designs, such as unidirectional conditional flows, imposing rigid task relationships and limiting the model’s capability to capture the true dynamic connections between gestures and expressions. Second, the use of manually designed static loss weights prevents the framework from adapting to dynamic changes in task importance during training. Third, over-relying on minimizing differences with ground-truth data often leads to gesture overfitting, thereby suppresses diversity. These limitations force existing methods into making unavoidable tradeoffs between facial synchronization and gesture diversity. Aiming to address this issue, this research introduces a unified adaptive framework that autonomously models and dynamically balances the relationship between the two tasks through learnable uncertainty mechanisms. The proposed framework aims to simultaneously achieve gesture diversity and expression accuracy without requiring manual intervention.MethodA novel diffusion-based framework is introduced, leveraging uncertainty-based multitask learning for adaptive task balancing in holistic motion generation. This approach represents the first application of uncertainty-based loss weighting to speech-driven holistic motion synthesis. The core idea is to treat gesture and facial expression generation as distinct tasks within a unified framework, allowing their relationship to naturally emerge during training. The framework employs a denoising diffusion probabilistic model that operates on concatenated gesture and facial expression representations. The architecture incorporates shared features, including WavLM audio representations, word embeddings, speaker identity, and timestep encoding, alongside task-specific features such as Gaussian noise vectors and seed motion sequences, to capture commonalities and distinct requirements of each task. Cross-local attention mechanisms are adopted to capture long-range dependencies across timesteps and modalities, while self-attention layers refine task-specific patterns. A key innovation of this framework lies in the introduction of learnable parameters that represent task-dependent homoscedastic uncertainty for gestures and expressions. The overall training objective integrates the losses of both tasks and dynamically weights them according to these uncertainty parameters. This formulation automatically balances task contributions: larger uncertainty values reduce penalties to encourage diversity, whereas smaller values increase penalties to enforce precision. The uncertainty parameters are jointly optimized with the model parameters, allowing the framework to automatically determine the optimal task weighting without manual intervention.ResultComprehensive evaluations performed on the 76-h BEAT dataset, featuring 30 speakers and a 98/16/16 data split, demonstrate notable improvements. The proposed method achieved the highest gesture diversity score (52.5), outperforming MambaTalk (51.6), diffusion-based approach for speech-driven holistic 3D expression and gesture generation (DiffSHEG) (47.4), DiffuseStyleGesture (DSG) (48.5), and cascaded motion network (CaMN) (43.2). Additionally, the method achieved the best semantic relevance score (SRGR: 0.324). For facial expressions, the proposed framework achieved the lowest Fréchet distance, outperforming MambaTalk, DiffSHEG, and SAiD. Ablation studies confirm the critical role of uncertainty-based weighting, as removing it decreased gesture diversity from 52.5 to 47.2 and increased facial FD from 9.18 to 10.5. The learned uncertainty parameters converged to weights of 0.506 and 0.494 for gestures and expressions, respectively, demonstrating autonomous task balancing. When applied to DiffSHEG and MambaTalk, the proposed mechanism further improved gesture diversity, validating its strong generalizability. Qualitative analysis revealed that the generated gestures exhibit substantially greater diversity than those produced by baselines, closely imitating ground truth. Additionally, user studies involving 17 participants evaluating nine groups of videos exhibited a clear preference for the proposed method in terms of gesture diversity, facial synchronization, and overall quality dimensions.ConclusionThis research presents a novel adaptive diffusion framework that successfully addresses the fundamental challenge of simultaneously achieving precise facial synchronization and diverse gesture generation. By introducing uncertainty-based learnable parameters within a multitask learning paradigm, the proposed method automatically optimizes task relationships without requiring manual parameter tuning, thereby achieving superior performance in deterministic expression synthesis and non-deterministic gesture generation. Experimental results demonstrate substantial improvements in facial accuracy (FD: 9.18), gesture diversity (52.5), and semantic relevance (SRGR: 0.324), with user studies confirming enhanced realism. These findings indicate that the proposed framework offers an effective solution for creating lifelike virtual agents and introduces new research directions for holistic motion generation through adaptive multitask learning. The codebase of the paper:https://doi.org/10.57760/sciencedb.j00240.00175.
摘要:ObjectiveSelf-supervised learning (SSL) has become a cornerstone of modern representation learning, allowing models to extract powerful features from vast unlabeled data. SSL-based methods have achieved performance comparable to or even exceeding fully supervised paradigms across various downstream tasks. This success has driven the development of large-scale vision models but often comes at the cost of immense computational resources, posing substantial barriers to research and practical applications. Data pruning, which aims to enhance training efficiency by removing less informative samples, provides a viable solution to this efficiency issue. Although dynamic data pruning methods are conceptually suited for the label-free nature of SSL, they face a fundamental challenge. Specifically, these methods typically assume that samples with low loss values are “simple” and can be safely pruned. However, this assumption does not hold reliably in the context of SSL. Owing to the complex and inherently stochastic characteristics of SSL tasks, such as masked image modeling with high masking ratios and contrastive learning, the loss associated with individual samples can exhibit extreme and frequent volatility throughout training. Consequently, critical samples might momentarily yield low loss values due to stochastic optimization dynamics and be erroneously and permanently removed from the training set, resulting in irreversible degradation in model performance. Furthermore, prematurely removing genuinely simple samples at the beginning of training can destabilize the learning process, as these samples are crucial for the model to learn fundamental low-level features. This work addresses the inherent conflict between existing dynamic pruning techniques and the intrinsic properties of SSL by introducing a novel pruning framework to ensure a stable, efficient, and lossless pre-training process.MethodAiming to address these challenges, curriculum-driven dynamic data pruning (CDDP), a novel framework redesigning the core dynamic pruning pipeline for SSL, is introduced. The methodology is based on two key innovations: a robust sample importance metric and a curriculum-based data scheduling strategy. First, the exponential loss importance (ELI) metric is introduced to address the unreliability of instantaneous loss. Instead of relying on a single, noisy loss value at a specific time step, ELI provides a temporally smoothed and stable assessment of sample importance by calculating the exponential moving average of its historical loss values. This approach effectively removes high-frequency fluctuations and captures the long-term contribution of the sample to learning, thereby preventing important samples from being mistakenly pruned during transient periods of low loss. ELI is highly efficient and introduces minimal computational and memory overhead. Second, inspired by curriculum learning principles, the dynamic curriculum pruning (DCP) strategy is developed. Rather than making simple binary decisions to retain or discard samples, DCP reframes data pruning as a dynamic data scheduling problem, guiding the model from simple to complex samples through a structured learning path. At the start of each epoch, DCP initially performs two steps: 1) applying an efficient 1D k-means clustering algorithm to partition the entire dataset into a predefined number of “courses” based on current ELI scores; and 2) sorting these courses by difficulty and defining a curriculum window that specifies the subset of courses to be used for training in the current epoch. As training progresses, this window gradually shifts from the simplest courses to the most difficult ones, ensuring that the model builds a solid foundation of basic features before transitioning to more complex examples. Aiming to further control the pruning ratio precisely over the training process, the framework incorporates a random pruning mechanism to achieve the target pruning rate. Additionally, an annealing strategy is introduced to deactivate pruning in the final training stages, allowing the model to converge on the full dataset and mitigating potential gradient bias.ResultComprehensive evaluations validate the effectiveness, generalizability, and robustness of CDDP across multiple data modalities and learning paradigms. The experiments encompassed 2D image datasets (CIFAR-10, ImageNet-1K) and 3D point cloud datasets (ShapeNet), utilizing various self-supervised pre-training schemes, including masked autoencoder (MAE) with vision transformer (ViT) backbones, momentum contrast (MoCo) with residual network (ResNet) backbones, and point cloud masked autoencoder (PointMAE) for 3D data. Performance was assessed across a wide range of downstream tasks, such as image classification on CIFAR-100, semantic segmentation on ADE20K, 3D object classification on ModelNet40 and ScanObjectNN, and 3D part segmentation on the ShapeNetPart dataset. The proposed CDDP framework was compared against the full-dataset baseline and state-of-the-art dynamic data pruning methods, such as InfoBatch and SeTa. Results unequivocally demonstrate the superiority of the proposed approach. CDDP achieved lossless performance compared with the full-dataset baseline, markedly outperforming competitors on challenging transfer learning and large-scale 2D and 3D tasks. Notably, on the large-scale ImageNet-1K dataset, CDDP matched baseline performance across all downstream tasks while pruning training data by 30.6%. This superior performance extended to the 3D domain, where the framework achieved virtually identical results to the baseline across all downstream tasks while pruning over 32% of the data. Furthermore, detailed ablation studies confirmed the individual contributions of the ELI metric and the DCP strategy, demonstrating that their synergistic combination is crucial for achieving lossless and stable pruned training.ConclusionThis paper introduces a novel CDDP framework specifically designed for self-supervised learning. Experiment results demonstrate that the proposed method notably improves training efficiency while achieving lossless performance compared with the full-dataset baseline, outperforming existing state-of-the-art pruning methods.
关键词:self-supervised learning (SSL);dynamic data pruning;dynamic curriculum learning;exponential loss metric;efficient pre-training
摘要:ObjectiveDongba paintings, as a valuable cultural heritage of the Naxi people, embody rich and diverse emotional expressions through their distinctive symbolic systems, color compositions, and brushstroke techniques. These artworks integrate religious beliefs, mythological narratives, ritual functions, and collective cultural values, forming highly abstract, symbolic, and stylized visual representations that differ significantly from natural images and everyday visual scenes. In Dongba paintings, emotional meanings are often conveyed implicitly through color arrangements, line directions, spatial structures, and brushstroke textures rather than explicit semantic objects. Such characteristics pose substantial challenges for automated emotion understanding and multi-label classification. However, most existing vision-language models are pre-trained under paradigms primarily oriented toward generic object recognition and natural image understanding, emphasizing object-level semantics while overlooking artistic aesthetics. When directly applied to culturally specific artworks such as Dongba paintings, these models struggle to establish reliable and fine-grained mappings between multi-dimensional emotional semantics and the aesthetic characteristics of color structures and brushstroke textures. This discrepancy leads to emotional semantic misalignment caused by domain shift, which severely restricts both classification performance and model interpretability in ethnic art scenarios. To overcome these limitations, this paper proposes an emotion classification method for Dongba paintings that explicitly integrates color and brushstroke features to enhance domain adaptability, representation robustness, and semantic interpretability.MethodFocusing on the unique visual characteristics of Dongba paintings, particularly their color structures and brushstroke textures, a dual-domain color-brushstroke feature extraction mechanism is designed. This mechanism jointly leverages spatial-domain representations and frequency-domain information to capture complementary aesthetic cues, including line orientations, edge distributions, texture patterns, and structural details that are closely related to emotional expression. Spatial-domain features emphasize global color composition and structural layout, while frequency-domain features highlight local texture variations and brushstroke details. By integrating information from both domains, the proposed feature extraction strategy enables a more comprehensive and discriminative representation of artistic aesthetics that are often ignored by generic visual encoders.On this basis, a progressive adaptive fusion architecture is constructed to dynamically balance the contributions of general visual features learned from large-scale pre-training and domain-specific artistic features extracted from Dongba paintings. Through progressive fusion, the model can gradually adjust feature importance across different layers, ensuring that culturally relevant aesthetic features are effectively emphasized while preserving the robustness of general visual representations. This adaptive strategy allows the model to accommodate stylistic diversity, heterogeneous painting patterns, and variations in artistic expression across different samples.Furthermore, to address the coexistence of multiple emotions under label-imbalanced conditions, object category prior knowledge of painting subjects is incorporated into the learning process. Since emotional expressions in Dongba paintings are often closely associated with depicted symbolic objects, a category-guided cascaded question-answering reasoning mechanism is introduced. This mechanism strengthens logical consistency, reinforces emotion-object associations, and improves classification accuracy, robustness, and reliability in complex multi-emotion scenarios.ResultExtensive experiments conducted on a self-constructed multi-label Dongba painting emotion dataset demonstrate the effectiveness and stability of the proposed method. The model achieves a mean average precision (mAP) of 91.77% and an exact match accuracy of 84.81%, outperforming the second-ranked text as image (TAI++) model by 1.25% and 8.86%, respectively. Comparative experimental results show that explicitly modeling color-brushstroke features and incorporating category-guided reasoning effectively alleviate domain shift, enhance discriminative emotional representations, and produce more reliable and consistent emotion predictions in culturally specific art contexts.In the visualization experiments, the proposed dual-domain color-brushstroke feature extractor demonstrates a strong capability to efficiently capture the distinctive visual characteristics of Dongba paintings. Compared with ResNet-50 and VGG16, whose feature response patterns mainly focus on local texture extraction, the proposed model can effectively identify color schemes and brushstroke styles that are closely associated with specific emotional expressions, thereby validating the task-oriented design and effectiveness of the feature extraction strategy. Moreover, the generated attention heatmaps not only accurately localize emotionally salient regions within the paintings but also provide a quantitative representation of stylistic differences and color semantic connotations across different subject categories, further enhancing the interpretability of the model and its ability to distinguish subtle variations in artistic expression.ConclusionThe proposed method provides an effective and interpretable technical pathway for emotion understanding in ethnic artworks. By integrating artistic aesthetic features with adaptive fusion and semantic reasoning mechanisms, the approach advances culturally aware visual emotion analysis and multi-label classification. Moreover, it offers meaningful technical support for the digital preservation, intelligent analysis, and sustainable inheritance of intangible cultural heritage, highlighting the importance of incorporating cultural and artistic knowledge into intelligent visual understanding systems.
关键词:Dongba paintings;emotion classification;color and brush stroke features;adaptive feature fusion;cascaded reasoning
摘要:ObjectiveRemote-sensing image super-resolution (RSISR) seeks to reconstruct high-resolution (HR) images from low-resolution (LR) observations, thereby enhancing spatial detail while preserving structural integrity across large-scale scenes. Although convolutional neural networks (CNNs) and Transformer-based architectures have achieved impressive progress, existing approaches still face an intrinsic trade-off between reconstruction fidelity and computational efficiency. CNNs capture fine-grained local texture effectively but are limited by small receptive fields, whereas Transformers model long-range dependencies well but incur quadratic complexity, making them expensive for high-resolution remote-sensing imagery. The recently proposed selective state-space model, Mamba, offers linear-time sequence modeling with strong global dependency representation. However, when applied to vision tasks, Mamba still has several shortcomings, including unidirectional information flow, disrupted pixel ordering caused by one-dimensional unfolding, and weak preservation of local semantic structure. These limitations motivate the design of an architecture that can jointly strengthen local-detail representation and global spatial consistency under constrained computation.MethodTo this end, we propose local-aware Mamba for super-resolution (LAMA-SR), a novel state-space-based reconstruction framework for RSISR. Rather than encoding positional information inside the scanning module itself, LAMA-SR first applies an explicit two-dimensional rotary position embedding (RoPE-2D) to input feature maps to encode relative spatial relationships and inter-pixel geometric dependencies along both axes. This design preserves structural continuity between neighboring regions and reduces the spatial distortion introduced by one-dimensional scanning. The position-enhanced features are then processed by a two-dimensional selective scan (SS2D) mechanism, which models long-range dependencies efficiently with linear computational complexity. To strengthen fine-grained spatial representation, a multi-scale mixture-of-experts (MoE) local information aggregator is introduced before the Mamba module. This component divides input features into small-, medium-, and large-receptive-field branches, allowing adaptive extraction of local patterns at different spatial scales. Unlike conventional MoE designs that rely on explicit routing, a lightweight channel-attention-based soft-routing mechanism dynamically adjusts the contribution of each expert according to the input content. This allows the network to emphasize small-scale experts for texture-rich regions and larger-scale experts for structural continuity, thereby enriching local representation and providing stronger input for subsequent state-space propagation. Collectively, these components create a cooperative mechanism that combines efficient global modeling with high local fidelity within a compact Mamba-based backbone.ResultExtensive experiments were conducted on four widely used benchmark datasets—UCMerced, RSSCN7, AID, and WHU-RS19—under upscaling factors of ×2, ×3, and ×4. Quantitative evaluation using peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) shows that LAMA-SR consistently outperforms representative CNN-based, Transformer-based, and Mamba-based methods. For example, on the UCMerced dataset at a ×4 scale, LAMA-SR achieves 29.26 dB PSNR with only 8.10 M parameters and 20.23 G FLOPs, surpassing MambaIR, which uses 20.42 M parameters and 79.70 G FLOPs, while also delivering better reconstruction fidelity. Ablation studies further demonstrate that RoPE-2D and the local information aggregator (LIA) provide complementary benefits: RoPE-2D improves spatial coherence and relative position modeling, while LIA enhances texture recovery and channel selectivity. Visual results and class-wise analysis confirm that LAMA-SR produces sharper object boundaries and more natural textures, particularly in complex scenes such as airports, industrial zones, and vegetation-rich areas, validating its robustness and generalization across diverse remote-sensing scenarios.ConclusionLAMA-SR provides a state-space-based framework for remote-sensing image super-resolution that improves both reconstruction fidelity and computational efficiency. By promoting more consistent spatial information propagation and targeted refinement of local structures, the model better preserves neighborhood continuity and scene layout during reconstruction than conventional Mamba-style baselines. Experiments on multiple benchmark datasets show that LAMA-SR delivers sharper textures, clearer object boundaries, and a more favorable accuracy-efficiency trade-off across different upscaling factors. These results indicate that state-space models can be adapted effectively to high-resolution remote-sensing imagery while maintaining fine-grained spatial semantics, offering a stable and scalable direction for future research on lightweight super-resolution models.
摘要:ObjectiveThe acquisition of high-quality multispectral remote-sensing imagery is fundamental to understanding the physical properties of the Earth’s surface and monitoring environmental dynamics. Multispectral data, especially those including near-infrared (NIR) bands, reveal unique physical characteristics of land surfaces and are therefore critical for vegetation-health assessment, land-cover classification, and complex environmental analysis. Such data provide essential support for regional ecological monitoring and territorial spatial planning. Deep learning has become a major research direction in remote-sensing image interpretation, but these methods rely heavily on large-scale, accurately annotated datasets. Most existing generative studies focus on standard RGB image synthesis to alleviate data scarcity. Although latent diffusion models (LDMs) have achieved strong performance in generation quality and diversity, directly applying RGB-pretrained foundation models to multispectral generation remains inherently limited. In real remote-sensing scenarios, imagery exhibits complex spectral correlations and domain-specific spatial patterns. RGB-pretrained weights alone are therefore insufficient for faithfully modeling physical constraints between specific wavelengths, such as the red and NIR bands, and for uncovering latent geospatial structure. Moreover, because foundation models lack multispectral representations, simple channel expansion is highly susceptible to distribution mismatch and severe generation degradation. Efficiently adapting diffusion models to the multispectral domain has therefore become an important research problem. However, from both computational and structural perspectives, training large multispectral diffusion models from scratch remains prohibitively expensive. Most widely used controllable generation methods still treat spatial conditions merely as shallow structural constraints encoded by simple convolutional modules. As a result, they struggle to model complex geospatial layouts and cannot adequately capture the high-dimensional textural and spectral characteristics of specific land-cover categories, especially in tasks requiring precise semantic alignment.MethodTo address these limitations, this paper proposes a parameter-efficient multispectral controllable diffusion framework designed to provide robust data support for multispectral image-interpretation tasks. The method adapts RGB-based models efficiently to the four-band remote-sensing domain (RGB + NIR) through a decoupled two-stage training strategy that promotes stable convergence without retraining the full network. A parameter-efficient fine-tuning (PEFT) strategy is adopted by inserting lightweight adapters and low-rank adaptation (LoRA) modules into a frozen pretrained backbone. To better capture complex geospatial patterns, multi-scale depthwise convolutions and modulation mechanisms (MSM and MONA) are integrated into the architecture. Crucially, we introduce a spectral-consistency loss based on the normalized difference vegetation index (NDVI) to enforce physically plausible correlations between the red and NIR bands. This loss is constrained by a brightness threshold to suppress noise from low-reflectance regions and improve spectral fidelity. In addition, a text-aware spatial-semantic encoding (TASSE) mechanism is designed to support precise layout control through semantic segmentation masks by explicitly mapping category-specific text embeddings to spatial features. This design improves both the physical plausibility and the practical usability of the synthesized data, thereby supporting research on diverse land-cover categories and downstream segmentation tasks.ResultSystematic experiments on standard datasets, including French land cover from Aerospace Imagry(FLAIR), Five-Billion-Pixels, and IRSAMap, demonstrate the effectiveness of the proposed framework and reveal the usability and challenges of multispectral generation for real-world data augmentation. Under a unified experimental setting, we compare our method with state-of-the-art approaches such as ControlNet, T2I-Adapter, CRS-Diff, and EarthSynth. Using NDVI-RMSE, constrastive language-image pre-training(CLIP) Score, and learned perceptual image patch similarity(LPIPS) as evaluation metrics, we analyze the impact of physical constraints and spatial encoding on the model’s ability to synthesize high-fidelity multi-band imagery. The results show that integrating both physical and spatial information is fundamental to multispectral generation: the proposed framework successfully generates valid NIR bands, achieves an average NDVI-RMSE of 0.317 3, and outperforms the baselines in both spectral fidelity and semantic alignment. Furthermore, when the synthesized multi-band data are used to train open-vocabulary segmentation models, mean intersection over union(mIoU) improves by approximately 0.3% to 1.0%.ConclusionOverall, the proposed framework effectively addresses channel mismatch and physical-feature loss, providing a resource-efficient data foundation for advancing remote-sensing image interpretation from data-scarce settings toward comprehensive, high-quality multispectral modeling. Future work will extend this parameter-efficient adaptation framework to a broader range of remote-sensing modalities, including hyperspectral and synthetic aperture radar (SAR) imagery, in order to capture more diverse physical properties and scattering mechanisms. We also plan to incorporate temporal-consistency constraints to support the generation of continuous remote-sensing image sequences for long-term, stage-wise land-surface change detection. Beyond spatial and spectral enhancement, advanced large language models (LLMs) will be integrated to improve the framework’s understanding of complex geospatial instructions and ultimately enable more fine-grained, interactive, and user-driven remote-sensing image synthesis.
摘要:ObjectiveRemote sensing image captioning (RSIC) is a pivotal cross-modal task that links computer vision and natural language processing, aiming to translate complex visual information in remote sensing images, including key ground objects, spatial scenes, inter-object relationships, and environmental contexts, into human-readable textual descriptions. This technology plays a crucial role in a wide range of practical applications. In disaster emergency response, RSIC enables rapid interpretation of affected areas (e.g., identifying collapsed buildings or flooded regions from post-disaster remote sensing images) to support real-time decision-making. In agricultural monitoring, this technology facilitates automated analysis of crop distribution, growth status, and land-use changes. In environmental change analysis, RSIC supports long-term monitoring of environmental phenomena such as deforestation, glacial melting, and coastal erosion. Unlike single-modal tasks such as image classification or object detection, RSIC delivers more comprehensive and coherent semantic expression, aligning closely with human understanding and lowering the threshold for non-professional users to utilize remote sensing data. However, existing RSIC methods still face two critical challenges that hinder their performance and applicability. First, feature extraction is often incomplete. Many approaches either fail to fully capture multiscale texture features of remote sensing images (resulting in loss of fine-grained details such as surface textures of ground objects) or over-rely on a single type of semantic feature (limiting the capability to represent complex scene information). Second, cross-modal information alignment remains insufficient. Most models focus on processing image features but lack effective mechanisms to bridge the semantic gap between visual features and textual information, leading to inconsistencies between generated captions and image content.MethodAiming to address the aforementioned challenges, this paper proposes a novel RSIC method integrated with multiple functional modules. In the feature extraction stage, two core modules are designed: the multiscale feature fusion module (MSFFM) and the hybrid semantic integration module (HSIM). MSFFM adopts ResNet50 as the backbone and extracts shallow detail features, mid-level main features, and deep global features from the last three convolutional layers. The shallow detail features are processed via depthwise separable convolution to capture fine-grained details and reduce computational cost, along with spatial attention focusing on key textures to optimize main features. The deep global features are enhanced via dilated convolutions to expand the receptive field and channel attention to strengthen the transmission of critical global features. After weighted fusion, frequency-domain enhancement uses FFT and learnable parameters to adaptively select useful bands, thereby boosting texture and structural representation. The HSIM module is designed to enrich image semantic representations by integrating multi-source features. Specifically, Faster R-CNN is employed to extract object-level local features, while the pre-trained CLIP model, which benefits from large-scale image-text pair pre-training, is employed to extract global semantic features. Thus, this module helps address the problem of limited remote sensing data. Before fusion, the global features extracted by CLIP are passed through a fully connected layer to align their dimensionality with that of local and texture features (output from MSFFM). The three types of features (texture, local object, and global semantic) are then concatenated, and a self-attention mechanism is utilized to optimize the final visual feature representation. Unlike traditional self-attention, this mechanism abandons learnable parameters; instead, it calculates similarities in spatial and channel dimensions through matrix transposition and multiplication, followed by SoftMax normalization to generate attention weights that highlight key semantic information. In the text generation stage, a cross-modal alignment module (CMAM) is constructed to achieve deep alignment between visual and textual features. First, the hidden state of the LSTM (containing rich temporal context information) is concatenated with word embedding vectors to form a comprehensive textual semantic representation. The representation serves as a guiding signal for pertinent image feature selection. This textual representation is then used to compute feature similarity with visual features through dot-product operations, and the results are fed into a fully connected layer to generate channel-wise attention weights for the image features. Using the textual semantic representation as queries and the channel-enhanced visual features as keys and values, multi-head attention is applied to achieve precise focus on relevant visual features, ensuring strong alignment between visual content and textual semantics. Aiming to further minimize the semantic gap between image and text features, a cross-modal feature alignment loss based on temperature-scaled cross-entropy is introduced. This loss function calculates the cosine similarity matrix of all image-text pairs in a batch using image features extracted by the encoder and global text features extracted by an additional Transformer encoder. Cross-entropy loss is computed along the image and text dimensions to optimize bidirectional alignment. Furthermore, a temperature parameter τ is introduced to adjust the probability distribution, controlling the “smoothness” of predictions and improving the stability of model training.ResultExperiments are conducted on the UCM-Captions, RSICD, and SAR-Captions datasets, with comparisons against numerous mainstream models. The evaluation metrics include BLEU1-BLEU4, METEOR, ROUGE, and CIDEr. The proposed method consistently outperforms all comparative models across the three datasets. On the UCM-Captions, the proposed model achieves a BLEU4 score of 0.713 0 and a CIDEr score of 3.354 3. The attenuation rate from BLEU1 to BLEU4 is 17.9% (lower than mlat’s 18.7%), indicating superior long-sentence coherence. On the RSICD dataset, the proposed model achieves a BLEU4 score of 0.327 7 and a CIDEr score of 0.922 3, outperforming HC and AoA. On the SAR-Captions dataset, the proposed method achieves a BLEU4 score of 0.857 0 and a CIDEr score of 4.260 6, verifying strong adaptability to SAR images and validating the effectiveness of the dataset. Ablation experiments further show that the cross-modal alignment module substantially boosts BLEU4 and CIDEr, while MSFFM + HSIM improves the performance across all metrics. The full model (integrating all modules) achieves optimal performance, confirming multi-module synergy.ConclusionThis paper proposes a novel RSIC method that integrates multiscale feature fusion, hybrid semantic integration, and cross-modal alignment modules, along with a dedicated SAR image captioning dataset. The MSFFM and HSIM modules effectively address the problem of incomplete feature extraction through the integration of multiscale texture features, local object features, and global semantic features. Meanwhile, the CMAM module optimizes cross-modal information alignment, reducing the semantic gap between images and texts.
摘要:ObjectiveScarce annotated data is a critical bottleneck that affects the performance of remote sensing hyperspectral image classification models. The small-sample condition not only leads to model overfitting but also poses considerable challenges of severely insufficient generalization capability. Aiming to address this issue, this study aims to explore a novel solution that combines data augmentation with semantic enhancement. Pre-trained language models are innovatively utilized to generate diverse semantic prompts, supporting the development of a data augmentation technique that strictly maintains category consistency. Through the generation of training samples that are semantically rich and distributionally diverse, this technique fundamentally enhances the model’s capability to capture the core semantics of categories, rather than merely memorizing a limited set of training samples. Experimental results demonstrate that the proposed method effectively enhances the model’s classification accuracy and generalization capability on the target domain, introducing a new pathway for small-sample hyperspectral image classification.MethodThis study adopts the HyperBlend data augmentation method and the PromptMix mechanism. At the data level, the HyperBlend method performs cropping and blending of hyperspectral images from the same category, followed by the application of random rectangular masks. This process simulates real-world conditions such as occlusion, noise, and illumination variations, generating blended images that maintain semantic consistency while exhibiting visual diversity. The method is easy to implement, allows adjustable mask ratios, and effectively expands the training dataset. At the semantic level, the PromptMix mechanism utilizes a pre-trained bidirectional encoder representations from Transformers(BERT) model to generate three diverse textual prompts for each target domain category, from which semantic features are extracted. During training, different prompt features are randomly selected as class prototypes, and a diversity regularization loss is introduced to limit the discrepancies among these features. This approach prevents the homogenization of semantic representation, enriches class representations, and mitigates overfitting. In terms of loss function design, the framework integrates classification loss, cross-modal alignment loss, target-domain supervised contrastive loss, and the proposed diversity regularization loss, achieving end-to-end optimization.ResultExperiments were conducted on four standard hyperspectral datasets: Indian Pines (IP), Houston (HT), Salinas (SA), and LK (WHU-Hi-LongKou), achieving remarkable improvements. On the IP dataset, compared with the SCFDA method, the overall accuracy (OA), the average accuracy (AA), and the Kappa coefficient (KC) increased by 3.8%, 2.67%, and 4.3%, respectively. On the SA dataset, compared with the momentum-enhanced dual-prototype learning framework (MEDPL) method, the OA, the AA, and the KC demonstrated 1.16%, 0.84%, and 1.28% improvements, respectively. On the LK and HT datasets, the classification performance is comparable to the best results achieved by existing methods such as SCFDA and MEDPL, demonstrating strong competitiveness.ConclusionThis paper introduces HyperBlend, a novel data augmentation method that generates high-quality and diverse training samples through structured image blending and masking operations with minimal computational overhead. Through the strategic fusion of spectral-spatial features from limited labeled data, HyperBlend effectively simulates realistic inter-class variations and boundary conditions, addressing the data scarcity problem in hyperspectral image analysis. Concurrently, PromptMix is a semantic-level augmentation mechanism that leverages learnable prompt embeddings to enhance the model’s semantic discrimination capability. This mechanism operates in the feature space, generating semantically meaningful perturbations that prompt the learning of highly robust feature representations. The synergy between HyperBlend and PromptMix creates a comprehensive augmentation framework that operates at the pixel level (through structured image mixing) and the semantic level (through prompt-based feature manipulation). This combined approach substantially improves model robustness and generalization performance in few-shot learning scenarios. Importantly, both methods feature simple implementation architectures requiring neither complex network designs nor substantial parameter additions. Therefore, the proposed framework offers a practical and efficient solution for hyperspectral image analysis, particularly for tasks with limited labeled data. Extensive experiments demonstrate consistent performance improvements across multiple benchmark datasets, validating the effectiveness of the proposed method in improving model adaptability in data-constrained environments. Key contributions include the following: 1) the HyperBlend method for structured spectral-spatial augmentation; 2) the PromptMix mechanism for semantic-level feature diversification; and 3) a lightweight and easily deployable framework that substantially boosts few-shot learning performance without introducing additional computational burden.
关键词:hyperspectral image(HSI);Cross-Domain Few-Shot Learning;HyperBlend data augmentation method;PromptMix Mechanism;Multimodal Feature Alignment
摘要:ObjectiveAs an active microwave remote sensing technology, synthetic aperture radar (SAR) is capable of acquiring Earth surface data under all-weather and day-night conditions. However, due to its coherent imaging mechanism, SAR systems inevitably introduce substantial speckle noise during backscattered signal processing. Furthermore, SAR imagery primarily reflects the dielectric properties and geometrical structures of targets, rather than the spectral characteristics perceived by the human visual system. This phenomenon leads to fundamental differences between SAR and optical images in terms of imaging mechanisms, physical properties, and image characteristics. Such differences severely limit their joint application and analysis in subsequent tasks. SAR-to-optical image translation is a data processing technique designed to address the modality gap between SAR and optical imagery by converting heterogeneous remote sensing images into data with homogeneous image characteristics. This approach provides crucial technical support for subsequent applications and analyses such as SAR-optical image matching and change detection, making it an important research direction. However, existing SAR-to-optical image translation methods often rely on single-generator structures, which struggle to simultaneously maintain global semantic consistency and local textural-structural details. This limitation often leads to semantic distortions and blurred details, consequently limiting their reliability and practical utility in real-world applications. Aiming to address this issue, this paper proposes a unidirectional knowledge transfer generative adversarial network (UKT-GAN) for dual-fidelity SAR-to-optical image translation. The model is designed to achieve dual fidelity in global and local dimensions of the generated imagery through unidirectional knowledge transfer between its dual-branch network architecture.MethodThe proposed method adopts a dual-branch generation framework that allocates two fundamentally distinct tasks—local textural-structural detail reconstruction and global semantic information preservation—to a detail reconstruction subnetwork and a semantic preservation subnetwork, respectively. This architectural design effectively addresses the inherent limitation of single-branch GANs in simultaneously maintaining global semantic consistency and local textural-structural fidelity. The detail reconstruction branch comprises a generator and a discriminator. The generator employs a U-Net architecture embedded with convolutional atention module (CBAM), utilizing skip connections to achieve multiscale fusion of feature maps between the encoder and decoder, thereby preserving richer spatial details. The discriminator adopts a shallow two-layer convolutional structure to evaluate the authenticity of local textural and structural details, while pixel-level loss constraints are applied to ensure the fidelity of local detail reconstruction. Similarly, the semantic preservation branch also comprises a generator and a discriminator. Its generator utilizes a ResNet framework integrated with PE-Transformer modules, leveraging multihead attention mechanisms and residual blocks within the PE-Transformer to perform deep, multidimensional extraction and integration of semantic features from the imagery. This design ensures the complete and accurate preservation of semantic information throughout the feature extraction and transmission process. The discriminator employs a deeper four-layer convolutional network to evaluate the overall structural rationality and global semantic consistency of the generated images, supplemented by a combination of pixel-level and feature-level loss constraints to maintain global semantic information consistency. Finally, a unidirectional consistency loss is introduced to transfer the detail capture capability from the detail reconstruction branch to the semantic preservation branch. This process optimizes and refines the local textural and structural details of the optical images generated by the semantic preservation subnetwork, ensuring that these generated images can simultaneously maintain global semantic consistency and local textural-structural fidelity, thereby further enhancing the overall quality of the imagery produced by the subnetwork.ResultAiming to validate the effectiveness of UKT-GAN in SAR-to-optical image translation, this paper comparatively analyzes six mainstream image translation methods, including Pix2pix, DCLGAN, Parallel-GAN, Conditional Diffusion, StegoGAN, and HVT-cGAN, on two public datasets: SEN1-2 and WHU-OPT-SAR. The evaluation uses four metrics, namely, peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), learned perceptual image patch similarity (LPIPS), and root mean square error (RMSE), assessing the quality of generated images from four aspects: global quality, structural similarity, deep feature fidelity, and pixel-level accuracy. On the farmland and mountain subsets of the SEN1-2 and the WHU-OPT-SAR datasets, the optical images generated by UKT-GAN achieved optimal results across all four evaluation metrics. In particular, the model revealed notable improvements in LPIPS, outperforming the second-best methods by at least 2.3% (where lower LPIPS values indicate better performance). On the building and forest subsets of SEN1-2, UKT-GAN also achieved optimal results in terms of SSIM and LPIPS. Overall, these findings demonstrate that the proposed UKT-GAN can generate higher-quality optical images compared to six mainstream image translation methods.ConclusionThis paper proposes a UKT-GAN for dual-fidelity SAR-to-optical image translation. Through unidirectional consistency loss, the model effectively transfers the local textural-structural reconstruction capability from the detail reconstruction subnetwork to the semantic preservation subnetwork. This approach enables the generation of optical images that simultaneously maintain global semantic information and local textural-structural details during testing, requiring the deployment of only the single semantic preservation subnetwork. Furthermore, experimental results demonstrate that compared with other methods, the proposed UKT-GAN can generate optical images with clearer structures and superior overall quality when processing different land cover types. Moreover, when confronted with data distribution shifts caused by variations in sensor characteristics and imaging configurations, UKT-GAN maintains stable translation performance, exhibiting strong generalization capability. The open-source code for this paper is available at:https://www.scidb.cn/s/YNjqIf.
摘要:ObjectivePansharpening aims to generate a high-resolution multispectral (HRMS) image by integrating the fine spatial structure of a high-resolution panchromatic (PAN) image with the rich spectral information of a low-resolution multispectral (LRMS) image. Although deep-learning-based approaches have made substantial progress, existing methods still face several key limitations, including insufficient spatial-detail recovery, spectral inconsistency, and degradation during upsampling. These problems arise largely because current frameworks do not model spatial texture, spectral dependency, and scale transition in a unified manner. To address these issues, this paper proposes a detail-preserving pansharpening network based on prism pyramid fusion (PPFusion). The proposed model integrates differential enhancement, Restormer-based long-range dependency modeling, hierarchical attention fusion, and shared upsampling regularization into a single deep-learning architecture. By jointly optimizing spatial and spectral representations, PPFusion aims to achieve a balanced and interpretable fusion process, producing HRMS images with improved fidelity and structural clarity.MethodThe PPFusion framework first upsamples the LRMS image to match the spatial resolution of the PAN image. The upsampled LRMS and PAN images are then concatenated and fed into a dual-path backbone. The first branch uses a differentially enhanced convolution (DEConv) module to capture high-frequency spatial details and edge contours explicitly through directional differential operators. This design strengthens texture gradients and sharp structures that are often lost during fusion. The second branch adopts a Restormer-based transformer module, which uses multi-head self-attention and feed-forward channel aggregation to model long-range spectral dependencies and maintain spectral consistency across channels. The two branches interact through three cross-scale fusion stages, allowing mutual refinement of spatial and spectral representations. This progressive interaction supports hierarchical detail reconstruction and spectral-spatial alignment. After backbone processing, a hierarchical content-guided attention fusion (HCGAF) module performs advanced multi-scale aggregation. HCGAF integrates four complementary inputs—the upsampled LRMS image, the PAN image, the intermediate fusion output, and the fusion features from the previous stage. Within a prism-pyramid structure, multi-level attention maps guide the integration of local fine-grained texture and global spectral semantics, ensuring that the fused image preserves both high-frequency precision and global coherence. To further stabilize training, a dual-branch attention-guided shared upsampling (DASU) module is introduced as a regularization branch rather than part of the main fusion pipeline. The DASU branch independently learns a mapping from LRMS to HRMS, using the PAN image as spatial guidance. During training, a self-consistency loss is imposed between the DASU output and the main-network output, forming a teacher-student-style constraint that regularizes optimization, mitigates overfitting, and improves generalization. The total loss includes three components: an L1 spectral-fidelity term between the fused image and the reference HRMS image, a spatial-consistency term derived from DASU supervision, and a smoothness constraint that promotes stable optimization. Together, these losses encourage balanced optimization of spectral accuracy and spatial sharpness.ResultExperiments were conducted on three benchmark satellite datasets—WorldView-3 (WV3), GaoFen-2 (GF2), and QuickBird (QB)—with twelve state-of-the-art deep-learning pansharpening methods used as baselines. Quantitative results show that PPFusion achieves state-of-the-art performance in spectral angle mapper (SAM) while maintaining competitive PSNR, ERGAS, and SSIM across datasets. Full-resolution evaluation also indicates comparable quality with no reference (QNR) scores, demonstrating robustness under no-reference conditions. Visual inspection shows sharper building edges, more realistic vegetation textures, and more natural color transitions, with fewer spectral shifts and fewer blurred boundaries. Ablation studies confirm the effectiveness of each module: DEConv sharpens edges, Restormer improves spectral-spatial coherence, HCGAF enables effective multi-scale integration, and DASU enhances training stability.ConclusionThe proposed PPFusion network introduces a new framework that jointly leverages differential convolution, Restormer-based long-range modeling, hierarchical content-guided fusion, and shared upsampling regularization. By combining local-detail reconstruction with global spectral modeling, it effectively balances spatial sharpness and spectral fidelity. Extensive experiments verify that PPFusion achieves state-of-the-art performance in both quantitative accuracy and perceptual quality. The DASU regularization branch further stabilizes training and improves adaptability across different satellite sensors. Future work will extend the prism-pyramid fusion paradigm to cross-sensor and multi-temporal pansharpening and explore the integration of physical imaging priors and generative modeling for more interpretable and adaptive remote-sensing image fusion.
摘要:ObjectiveMedical image segmentation is crucial in computer-aided diagnosis, treatment planning, and clinical research. Traditional segmentation methods often rely on handcrafted features and classical image processing algorithms, which often struggle with the complexity, heterogeneity, and low-contrast characteristics commonly found in modern medical imaging data. Although deep learning methods, particularly convolutional neural networks (CNNs) and, more recently, Transformer-based architectures, have advanced the field through automatic feature extraction and state-of-the-art performance, considerable challenges remain. Existing deep learning models often have limited capability to preserve fine boundary details and effectively leverage multiscale contextual information, leading to suboptimal segmentation of irregular or small anatomical structures and pathological regions. Aiming to address these limitations, this study aims to design and validate a novel deep learning framework that synergistically combines the advantages of hierarchical feature learning, long-range dependency modeling, and precise edge delineation to enhance the accuracy, robustness, and generalizability of medical image segmentation across diverse tasks and datasets.MethodThis paper introduces Swin-EdgeNet, a hybrid encoder-decoder architecture for medical image segmentation that innovatively integrates a Swin Transformer backbone with a U-Net style decoder enhanced with specialized modules. The encoder is built upon Swin Transformer V2-B, which serves as a powerful feature extractor. Its shifted-window self-attention mechanism efficiently captures hierarchical feature representations and establishes long-range spatial dependencies across the image, providing a rich global context that is often missing in pure CNN architectures. The corresponding decoder is designed to progressively recover spatial resolution and generate precise segmentation maps. Aiming to specifically address the challenge of boundary ambiguity, a multiscale edge dense module (MS-EDM) is introduced within the decoder. This module employs parallel convolutional pathways with different receptive fields (dilated convolutions) to simultaneously capture and fuse features at multiple scales. Additionally, the dense connections within MS-EDM facilitate feature reuse and strengthen gradient flow, preserving and enhancing fine edge details and local textures throughout the upsampling process. Furthermore, a dual-attention mechanism is incorporated to dynamically focus the model’s capacity on semantically critical and structurally salient regions. This mechanism comprises two components: a detail-enhanced visual attention (detail-enhanced dual attention, DEVA) module and a dual attention (dual attention, DA) module. The DEVA module refines feature maps along the channel dimension by emphasizing interdependent channel maps, while the DA module combines spatial and channel attention to selectively aggregate contextual information from relevant positions and feature channels. This dual strategy helps suppress irrelevant background noise while highlighting subtle structures and pathological areas. Aiming to optimize computational efficiency without compromising performance, standard Conv2d layers in specific parts of the decoder are replaced with KanConv layers, a parameter-efficient alternative that reduces the overall model parameters. The entire network is trained end-to-end using a combined loss function, typically involving Dice loss and cross-entropy loss, to address class imbalance.ResultExtensive experiments and quantitative evaluations were conducted on four public benchmark datasets encompassing distinct medical image segmentation tasks to validate the effectiveness and generalization capability of the proposed Swin-EdgeNet. On the International Skin Imaging Collaboration (ISIC2018) skin lesion segmentation dataset, the proposed method achieved a Dice similarity coefficient (DSC) of 89.87% and an intersection over union (IoU) of 83.97%, outperforming existing state-of-the-art methods such as Rolling-Unet. The high recall rate of 93.44% indicates the model’s strong capability to capture complete lesion areas while minimizing false negatives. For the lung segmentation task, the model also demonstrated exceptional performance, attaining a DSC of 97.62% and an IoU of 95.44%, surpassing strong baselines such as DCSAU-Net (DSC: 97.56%, IoU: 95.35%) and HTC-Net (DSC: 97.48%, IoU: 95.19%). Additionally, the model maintained high precision (97.12%), recall (95.94%), and F1-score (96.44%), indicating a robust balance between accuracy and coverage. In the challenging domain of retinal fundus image vessel segmentation, where detecting thin capillary structures is difficult, the proposed method achieved a notably high sensitivity of 80.19%, substantially exceeding that of models such as U-Net++ (61.56%) and M2SNet (43.46%). This finding highlights its enhanced capability for segmenting low-contrast, fine-grained structures. The model also attained an F1-score of 77.77% and an AUC of 88.63%. For the polyp segmentation task, Swin-EdgeNet achieved a DSC of 93.80% and a recall of 91.93%, representing improvements of 2.12% and 6.58%, respectively, over models such as Trans-UNet. The F1-score for polyp segmentation reached 89.81%. Qualitative visual analysis further corroborated these quantitative findings. As illustrated in the figures, the proposed method produces segmentation masks with noticeably sharper boundaries and more accurate contours for skin lesions. Additionally, the method retains more complete small vessel branches in retinal images and provides more precise coverage for flat and sessile polyps compared with other models, which often suffer from under-segmentation or over-segmentation issues.ConclusionThis paper presents Swin-EdgeNet, a robust and effective deep-learning framework for medical image segmentation. By integrating the global contextual understanding of the Swin Transformer encoder with the detailed, localized feature recovery of a progressively enhanced U-Net decoder, the proposed model effectively bridges the gap between high-level semantics and low-level spatial precision. The introduced MS-EDM and dual-attention mechanism collectively address key challenges in medical imaging, including the preservation of critical edge information, the effective utilization of multiscale contextual information, and the dynamic focus on diagnostically relevant regions. Additionally, the replacement of standard convolutions with KanConv contributes to a more parameter-efficient design. Comprehensive experimental results across four diverse and challenging clinical tasks—skin lesion segmentation, lung segmentation, retinal vessel segmentation, and polyp segmentation—consistently demonstrate that the proposed method achieves superior or highly competitive performance compared with current state-of-the-art approaches. Overall, these results confirm the strong generalization capability of Swin-EdgeNet and highlights its potential for improving the accuracy and reliability of computer-aided diagnosis systems.