最新刊期
-
摘要：ObjectiveHyperspectral image （HSI） classification is an important task in remote sensing image interpretation because HSI data contain continuous spectral responses that are useful for fine-grained land-cover recognition. Existing deep learning methods， especially convolutional neural networks （CNNs）， have achieved high classification accuracy under the closed-set assumption. However， this assumption requires that all test classes have been observed during training， which is often inconsistent with real remote sensing applications. In large-scale and dynamic scenes， a trained model may encounter undefined or newly emerging land-cover categories. Conventional closed-set classifiers usually force such unknown samples into one of the known classes and may produce overconfident predictions. In addition， HSI data are affected by illumination variation， atmospheric scattering， sensor noise， and mixed pixels， resulting in the phenomena of same material with different spectra and different materials with similar spectra. These factors make hyperspectral open-set recognition （OSR） more challenging than ordinary closed-set classification. To address these problems， this paper proposes a frequency knowledge-guided contrastive learning method， called FKCL， for hyperspectral open-set recognition. The main purpose is to improve the rejection ability for unknown land-cover classes while maintaining reliable classification performance for known classes.MethodThe proposed FKCL framework integrates data-driven spatial-spectral representation learning with frequency-domain physical prior knowledge. The whole network consists of two collaborative branches. The first branch is a spatial-spectral feature extraction branch based on a 3D-2D hybrid convolutional neural network. It uses 3D convolution to capture local spectral correlations and then uses 2D convolution to extract spatial contextual information from hyperspectral image patches. The extracted high-level representation is projected into a compact feature space， where class-wise metric learning is performed. The second branch is a frequency-domain physical attribute extraction branch. Instead of using the average spectrum of the whole patch， this branch uses the central pixel spectrum because the category label of a hyperspectral patch usually corresponds to the central pixel. The central spectral sequence is transformed into the frequency domain by real fast Fourier transform （RFFT）， and its magnitude spectrum is used to describe the frequency fluctuation pattern of the ground object. This design preserves the correspondence between the original spectral reflectance curve and its physical frequency representation， and avoids the interference caused by heterogeneous neighboring pixels. To obtain stable class-level frequency representations， an exponential moving average （EMA） mechanism is introduced to update the frequency prototype of each known class during training. The EMA update reduces prototype fluctuation caused by small-batch sampling and gradually accumulates the stable frequency pattern of each known category. For feature-space discrimination， the class anchor clustering （CAC） loss is adopted. The anchor loss pulls samples toward their corresponding class anchors， whereas the tuplet loss pushes samples away from non-corresponding anchors， thereby improving intra-class compactness and inter-class separability. During inference， FKCL uses a dual-distance decision strategy. The semantic feature distance and the frequency-domain prototype distance are fused by a frequency weight selected on the validation set， and the open-set rejection threshold is also determined on the validation set. The test set is used only for final evaluation， so no test information is involved in threshold selection.ResultExperiments were conducted on three public HSI benchmark datasets， namely WHU-Hi-LongKou， Pavia University， and Salinas. A class-holdout protocol was adopted to construct pseudo-unknown classes. In this protocol， selected categories are completely removed from training and appear only during validation and testing for unknown rejection evaluation. For WHU-Hi-LongKou， the Mixed weed class was held out as the pseudo-unknown class. For Pavia University， the Shadows class was held out. For Salinas， Stubble， Lettuce_romaine_6wk， Lettuce_romaine_7wk， Vinyard_untrained， and Vinyard_vertical_trellis were treated as pseudo-unknown classes. The proposed FKCL method was compared with six representative closed-set and open-set methods， including DBDA， SSFTT， CACL， SSMTL， HyperCASR， and SS-MTr. The evaluation metrics included overall accuracy （OA）， average accuracy （AA）， Kappa coefficient， area under the receiver operating characteristic curve （AUROC）， and unknown class accuracy. Experimental results show that FKCL achieves a better balance between known-class classification and unknown-class rejection. On WHU-Hi-LongKou， FKCL achieved an OA of 92.38%， an AA of 92.58%， and a Kappa coefficient of 90.08%， while its unknown class accuracy reached 25.20%， which was higher than most deep closed-set comparison models. On Pavia University， FKCL achieved an OA of 92.02%， an AA of 89.13%， a Kappa coefficient of 89.48%， an AUROC of 99.96%， and an unknown class accuracy of 100.00%， indicating that the frequency-guided dual-distance strategy can effectively identify the held-out Shadows class. On Salinas， the proposed method maintained high classification performance for known classes， but the rejection of pseudo-unknown samples remained more difficult because several unknown classes had very high spectral similarity to known classes. This result indicates that the open-set difficulty is closely related to the physical and spectral separability between known and unknown categories. The ablation analysis further shows that the frequency-domain branch， frequency prototype learning， and dual-distance decision mechanism contribute to the improvement of open-set detection performance. In particular， removing the frequency prototype EMA weakens unknown rejection， demonstrating that stable frequency prototypes are important for modeling class-level physical attributes.ConclusionThe proposed FKCL method introduces frequency-domain knowledge into hyperspectral open-set recognition and constructs a collaborative framework that combines deep spatial-spectral features with physical spectral priors. Compared with single-branch data-driven models， FKCL uses RFFT-based frequency representations and EMA-updated frequency prototypes to provide complementary information for distinguishing spectrally similar but physically different land-cover categories. The CAC-based metric learning module constructs a compact and separable feature space for known classes， and the dual-distance decision strategy further improves the ability to reject samples outside the known-class distribution. Under the class-holdout pseudo-unknown evaluation protocol， FKCL reduces the overconfidence problem of conventional deep models and improves the discrimination of unseen categories. Nevertheless， when unknown classes are extremely similar to known classes in both the original spectral domain and the frequency domain， open-set rejection remains challenging. Future work will focus on adaptive frequency-weight learning， more realistic open-world remote sensing scenarios， cross-region generalization， dynamic newly emerging categories， and open-set recognition with multimodal remote sensing data.关键词：hyperspectral image;open-set recognition;frequency domain guidance;class anchor clustering;spatial-spectral fusion;dual-branch architecture1|0|0更新时间：2026-07-16 -
摘要：ObjectiveVisual imitation learning aims to learn control policies directly from high-dimensional image observations. It is important for robotics tasks， where agents often need to make decisions from visual inputs. However， compared with imitation learning based on low-dimensional proprioceptive states， visual imitation learning still suffers from a clear performance gap. One major reason is that behavior-related differences in pixel observations are often subtle. As a result， the visual encoder may fail to learn sufficiently discriminative state representations. This problem becomes more serious in visual adversarial imitation learning. In this framework， the discriminator needs to distinguish expert samples from agent samples and provide reward signals for policy learning. If the learned visual representation is weak， the discriminator may produce unstable or inaccurate rewards. This further affects policy optimization. Existing visual adversarial imitation learning methods mainly focus on the distinction between expert samples and agent samples. They usually treat this distinction as a static discrimination problem. However， they do not fully exploit the internal structure of agent samples stored in the replay buffer. They also do not explicitly model the dynamic evolution of agent samples during training. In practice， the quality of agent samples gradually improves as the learned policy approaches the expert policy. Ignoring this process may limit the quality of visual representation learning. To address these issues， this paper proposes a visual adversarial imitation learning method based on calibrated contrastive learning.MethodThis paper introduces a calibrated contrastive representation learning method in the visual adversarial imitation learning framework. The main idea is to improve the discriminative ability of the visual encoder by contrastive learning. Specifically， similar states are pulled closer in the feature space， while different states are pushed apart. In this way， the encoder can learn more effective visual representations for adversarial imitation learning. Different from existing methods that mainly focus on the static distinction between expert samples and agent samples， the proposed method further explores the internal sample structure in the replay buffer. It also models the dynamic improvement of agent sample quality during training. In the proposed framework， agent samples are regarded as a mixture of high-quality samples and low-quality samples. High-quality agent samples may be close to expert samples， especially in the later stage of training. Low-quality agent samples are still far from expert behavior and should not be treated as expert-like samples. Based on this observation， this paper introduces a calibrated supervised contrastive loss. This loss adaptively adjusts the contrastive relationship between agent samples and expert samples. It reduces the risk of incorrectly pushing high-quality agent samples away from expert samples. At the same time， it preserves the discrimination between expert behavior and low-quality agent behavior. The proposed method therefore improves both visual representation quality and adversarial training stability. The final objective combines the adversarial imitation learning loss with the proposed contrastive representation learning objective. The whole framework can be optimized in an end-to-end manner.ResultExperiments are conducted on 9 continuous control tasks from the DMControl Suite. These tasks cover different types of control behaviors， including balancing， locomotion， and complex body movement. The experimental results show that the proposed method achieves higher cumulative rewards on multiple tasks. It also shows better sample efficiency in the early training stage. Compared with the representative method PCIL， CAIL improves the average performance by 22.6% at 1M training steps. This result indicates that calibrated contrastive representation learning can effectively improve visual adversarial imitation learning. The ablation study further verifies the effectiveness of the proposed contrastive losses. The unsupervised contrastive loss on agent samples improves the use of the replay buffer. It helps the encoder learn from a large number of dynamically collected visual states. The calibrated supervised contrastive loss further improves performance by modeling the changing quality of agent samples. The results show that these components are complementary. The full CAIL model achieves better performance than the version without calibration. This demonstrates the importance of dynamically modeling agent samples during training. In addition， visualization results show that the discriminator learned by CAIL can focus more accurately on behavior-related regions， such as the joints and body parts of the agent. These regions are important for distinguishing different motion states. The visualization results further support that CAIL learns more meaningful and discriminative visual representations.ConclusionThis paper proposes a calibrated contrastive visual adversarial imitation learning method for high-dimensional visual control tasks. The proposed method makes better use of agent samples in the replay buffer. It also explicitly models the change of agent sample quality during training. By introducing calibrated contrastive representation learning， the method improves the discriminative ability of visual state representations and stabilizes adversarial training. Experimental results on DMControl tasks show that the proposed method achieves better average performance and sample efficiency than representative visual imitation learning methods. The proposed framework provides an effective representation learning strategy for imitation learning from high-dimensional image observations.关键词：reinforcement learning;imitation learning;adversarial imitation learning;contrastive learning;representation learning74|56|0更新时间：2026-07-02 -
摘要：ObjectiveWith the rapid proliferation of unmanned aerial vehicles （UAVs） in civil and commercial fields， the demand for robust low-altitude security， airspace management， and intelligent visual surveillance systems has grown exponentially. Precise and real-time visual detection of UAV targets in continuous video streams is the core link of these systems. However， in practical uncontrolled surveillance scenarios， UAVs are usually located at extreme distances from the camera， occupying only a tiny fraction of pixels on the imaging plane， thus lacking clear morphological outlines， observable textures， and structural details. They are highly susceptible to being obscured by complex and dynamic backgrounds such as moving clouds， fluctuating lighting， and cluttered ground landscapes. Traditional single-frame object detection algorithms segment continuous video streams into isolated static images for frame-by-frame processing， fundamentally ignoring the crucial temporal motion information between frames. This leads to severe representation limitations when facing extremely small targets， with persistently high missed detection and false alarm rates. Existing temporal detection methods based on 3D convolution or Transformer can capture temporal features， but their huge parameter amount and computational overhead make them impractical for deployment on resource-constrained real-time security systems. This paper aims to break through the performance bottleneck of pure spatial detection frameworks and achieve high-precision and real-time detection of tiny UAV targets in complex backgrounds.MethodThis paper proposes a Temporal Motion-Aware Dual-Branch Network （TMAD-Net）. A dual-branch decoupling design is adopted at the data input end to physically separate spatiotemporal multimodal information. Specifically， the spatial semantic extraction branch processes multi-frame stacked raw RGB images to extract rich spatial appearance， local texture， and global contextual semantic features of the target and its surrounding environment. The parallel explicit motion prior branch is dedicated to processing inter-frame difference images between adjacent frames that have undergone absolute value operation and morphological denoising filtering. This branch filters out static background interference， camera jitter， and sensor noise， and generates a high-purity motion saliency map that highlights the high-frequency temporal motion trajectory of the flying UAV. The two data streams independently pass through the shallow network to complete high-dimensional feature extraction， and are then cascaded and concatenated along the channel dimension to maximally preserve multimodal spatiotemporal information. Subsequently， a motion-spatial adaptive fusion module is introduced to perform adaptive weight calibration on the fused features from both motion and spatial dimensions， so as to enhance the feature response of weak target signals and suppress irrelevant background noise. Finally， 1×1 convolution is applied to accomplish cross-channel dimensionality reduction and feature mapping， and the calibrated spatiotemporal features are seamlessly fed into the subsequent backbone network to complete target detection.ResultExtensive comparative tests and ablation experiments were conducted on the public ARD-MAV dataset and two private UAV datasets （Phone and DJI） collected in real-world scenarios， which are characterized by small target scales， varying illumination conditions， and low signal-to-noise ratios. Quantitative experimental results show that， compared with the standard baseline model， the proposed method achieves steady performance improvement on the ARD-MAV dataset： the mean average precision at 50% intersection over union （mAP50） increases from 0.264 to 0.588， and the mean average precision over the intersection over union range of 50% to 95% （mAP50-95） increases from 0.155 to 0.334；on the Phone dataset： the mAP50 increases from 0.802 to 0.887， and the mAP50-95 increases from 0.590 to 0.638.On the highly challenging DJI dataset， the proposed model achieves a remarkable performance improvement： the mAP50 surges from 0.316 of the baseline to 0.822， with a performance gain of over 160%， and the mAP50-95 jumps from 0.083 to 0.266. Compared with mainstream lightweight detection algorithms including YOLOv9s， YOLOv11s， and YOLOv12s， the proposed model achieves optimal detection accuracy on all three datasets. Even compared with the larger-scale RT-DETR-L model based on the Transformer architecture， the proposed model still achieves better overall performance in both detection accuracy and inference speed. Meanwhile， the proposed model has a parameter count of 11.33M， which is basically consistent with the 11.13M of the baseline model. It reaches an inference speed of 122 frames per second （FPS）， maintaining excellent real-time inference capability while realizing a steady improvement in accuracy. Ablation experiments further verify the effectiveness of the dual-branch architecture， spatial semantic extraction branch， and motion-spatial adaptive fusion module respectively， proving the rigor of the design logic of the proposed architecture.ConclusionThe TMAD-Net proposed in this paper successfully compensates for the inherent limitation of insufficient feature extraction capability of traditional single-frame detection algorithms when facing tiny targets. Through explicit separation and in-depth fusion of multi-frame RGB spatial appearance features and computationally optimized motion features， combined with the precise calibration capability of the motion-spatial adaptive fusion module， this method effectively isolates real targets from heavily cluttered and dynamic backgrounds， and effectively alleviates the long-standing performance bottleneck of video small target detection in complex real-world environments. Empirical results confirm that the proposed model significantly improves the detection accuracy and algorithmic robustness of UAV target recognition under non-ideal conditions， and provides a feasible， effective and novel visual detection paradigm for real-time video surveillance， public safety monitoring， and advanced low-altitude defense systems.关键词：video object detection （VOD）;unmanned aerial vehicle （UAV）;tiny object;motion decoupling;temporal motion perception;spatial semantic extraction61|63|0更新时间：2026-07-02 -
摘要：ObjectiveDriven by rapid advances in deep learning， text-to-speech （TTS） and voice conversion （VC） technologies have achieved unprecedented improvements in audio naturalness， fidelity and diversity， enabling large-scale and high-quality audio forgery. However， the widespread abuse of such deepfake technologies has posed severe and persistent threats to personal privacy， social credibility and even national security， triggering urgent demands for effective audio deepfake countermeasures. Existing research in this field mostly focuses on binary deepfake detection， which only judges whether an audio clip is forged but fails to identify the specific generation model or method behind the forgery. In contrast， audio deepfake attribution （ADA） targets tracing the exact source of forged audio by extracting unique model-specific artifacts， offering interpretable evidence for AI content governance， supporting complete evidence chains for judicial forensics， and promoting standardized supervision of the AI speech industry， thus emerging as a pivotal research hotspot in artificial intelligence security. Nevertheless， current ADA approaches are plagued by prominent defects that hinder practical deployment： on one hand， most methods rely on single-layer or shallow handcrafted features， lacking sufficient feature extraction capability to mine deep semantic information and fine-grained differences between diverse forgery models； on the other hand， they exhibit extremely limited generalization ability， especially in low-resource few-shot scenarios with scarce annotated training samples and open-set scenarios involving emerging unknown forgery models， suffering from drastic performance degradation and failing to adapt to the continuous iteration of speech synthesis technologies. Therefore， this study aims to develop a high-performance audio deepfake attribution method suitable for low-resource conditions， with strong discriminative ability for known forgery methods and reliable generalization for unknown forgery scenarios.MethodTo bridge the aforementioned research gaps， this paper proposes a novel Low-resource Audio Deepfake Attribution （LADAR） method dedicated to few-shot learning scenarios， which integrates innovative multi-layer feature fusion and multi-prototype learning mechanisms to break through the bottlenecks of existing methods. The core technical design is refined based on in-depth analysis of current mainstream attribution frameworks and their limitations. First， a multi-layer adaptive feature fusion module is built to leverage the powerful pre-training representation capability of Wav2Vec2-BERT 2.0， a state-of-the-art self-supervised audio model. Instead of using single-layer hidden states as in traditional methods， this module assigns learnable attention weights to aggregate multi-layer hidden features， and introduces a shallow bias factor to enhance the sensitivity to subtle low-level forgery traces that are easily ignored by deep layers； meanwhile， a learnable gating mechanism is adopted to dynamically fuse attention-weighted pooling and average pooling， generating compact and highly discriminative model embedding representations with optimized intra-class compactness and inter-class separability. Second， a K-means clustering driven multi-prototype learning module is designed to address the limitation of traditional single-prototype representation， which cannot cover the complex intra-class feature distribution of each forgery method. By generating multiple representative prototype vectors for each category， this module fully captures the feature variation patterns of the same synthesis model under different audio lengths， speakers and content conditions， strengthening the model’s robustness to sample variability. Finally， the attribution performance of LADAR is systematically evaluated under two critical experimental setups： closed-set scenario with known forgery methods and open-set scenario with unknown forgery methods， to fully verify its effectiveness and generalization.ResultsComprehensive comparative experiments are conducted against several representative state-of-the-art audio deepfake attribution methods， and the quantitative results fully validate the superiority of the proposed LADAR method. In the closed-set scenario targeting known forgery models， the LADAR method achieves significant performance gains， with overall accuracy improved by 35.25%， 26.26%， 9.22% and 5.65% respectively compared with baseline methods， and the F1-score， a comprehensive metric balancing precision and recall， increased by 38.82%， 27.42%， 7.23% and 5.74% correspondingly， reflecting its outstanding ability to accurately distinguish different known forgery methods. In the more challenging open-set scenario involving unknown forgery models， which is highly consistent with real-world application environments， LADAR still maintains stable and efficient performance： its accuracy is enhanced by 15.80%， 37.51%， 10.73% and 10.42% respectively， and the F1-score is elevated by 20.29%， 34.17%， 11.40% and 12.33% respectively compared with contrast methods. The experimental results demonstrate that LADAR effectively alleviates performance deterioration caused by insufficient training samples and inconsistent feature distribution of unknown samples， solving the core pain points of weak generalization and poor fine-grained recognition in existing methods.ConclusionThe systematic experimental analysis confirms that the proposed LADAR method effectively addresses the key technical challenges in audio deepfake attribution， with remarkable advantages in accuracy， robustness and cross-scenario generalization. By optimizing the feature extraction pipeline and innovating the prototype learning mechanism， it overcomes the inherent defects of insufficient feature mining and limited adaptability to unknown samples in traditional attribution methods. This method provides reliable， interpretable and efficient technical support for practical high-demand scenarios including judicial forensics， deepfake content source tracing， network audio supervision and national security defense， and also offers a feasible technical reference for other low-resource audio security tasks. In the context of continuous evolution of deepfake technologies， the LADAR framework can extend to new speech synthesis models， enabling sustainable AI audio security governance.关键词：Audio deepfake attribution;few-shot learning;Multi-layer feature fusion;Multi-prototype network;Anti-spoofing30|44|0更新时间：2026-07-02 -
摘要：ObjectiveMultispectral remote sensing forest image change detection aims to identify temporal variations in land-cover types or surface conditions by comparing multispectral images acquired over the same area at different times. This technique has been widely applied in ecological monitoring and natural resource management， among which forest change detection is of particular importance for tracking forest cover dynamics， assessing ecosystem function changes， and supporting forest resource conservation. Multispectral remote sensing images， acquired through observations across multiple discrete spectral bands， can effectively characterize spectral response differences associated with changes in land-cover types and surface conditions. Meanwhile， they provide large-area coverage and periodic observations with controllable data acquisition costs， thereby offering stable and discriminative data support for long-term and continuous forest change monitoring. Consequently， multispectral remote sensing image–based change detection has become an essential technical approach for forest resource monitoring. In early studies， change detection was mainly conducted using traditional image processing and pattern recognition techniques， where changes were modeled based on manually designed features and predefined thresholds. Although these methods are relatively simple and interpretable， they are highly sensitive to noise， illumination variations， and imaging condition discrepancies， which limits their reliability in complex scenarios. With the rapid advancement of deep learning， convolutional neural network （CNN）–based methods have become the dominant paradigm for remote sensing change detection due to their strong capability in automatic feature representation and end-to-end learning. Siamese architectures further improve detection performance by explicitly modeling feature differences between bi-temporal images. However， the fixed receptive fields of CNNs restrict their ability to capture long-range dependencies and global contextual information. To alleviate this limitation， attention mechanisms have been introduced to emphasize change-relevant features across spatial and channel dimensions. More recently， Transformer-based models have been applied to change detection， benefiting from self-attention mechanisms to model global dependencies and complex change patterns. Despite these advances， deep learning–based methods still face significant challenges in forest change detection. Bi-temporal remote sensing images often exhibit pronounced style discrepancies caused by seasonal variations， illumination conditions， and sensor differences， which introduce substantial pseudo-change interference. Moreover， forest changes are characterized by high spatial heterogeneity， ranging from subtle local variations to large-scale degradation or recovery processes， resulting in complex multi-scale and multi-pattern change characteristics. These factors make it particularly challenging to suppress style-induced pseudo changes while accurately perceiving and localizing change targets at different spatial scales. To address these challenges， this paper proposes a frequency-enhanced hierarchical interaction network for accurate forest change detection.MethodThis paper proposes a frequency-enhanced hierarchical interaction network for forest change detection in remote sensing imagery. Bi-temporal multispectral images are taken as input and first processed by shared-weight spatial–spectral feature extraction modules to obtain multi-level spatial–spectral representations. To optimize the global representation of bi-temporal features， a frequency feature enhancement module （FFEM） is designed to enhance bi-temporal feature representations. This module performs adaptive modulation in the frequency domain to suppress low-frequency style interference and emphasize mid- and high-frequency structural information， thereby alleviating pseudo-change artifacts caused by imaging condition discrepancies. Subsequently， a cross-temporal correlation-guided interaction module （CCIM） is introduced to explicitly model temporal correlations between bi-temporal images and extract discriminative change features. By leveraging feature interaction and correlation guidance， this module captures temporal dependencies， enabling the extraction of stable change representations and improving the network’s sensitivity to subtle forest changes. In addition， a spatial–spectral coupled interaction module （SCIM） is employed to fully exploit the complementary information among change features at different hierarchical levels. This module jointly models spatial and channel-wise dependencies by integrating fine-grained structural details with high-level semantic representations， thereby enhancing the perception of multi-scale forest change regions. Finally， the fused change features are fed into a classification head to generate the change detection results.ResultComparison experiments were conducted on three forest change detection datasets to compare the proposed method with six advanced approaches， including Spectral-Former， CSANet， TriTF， DIEFEN， FrFTML， and AIWSEN. For each dataset， 10% of the samples were randomly selected for training， while the remaining samples were used for testing. Performance was evaluated using five widely adopted metrics for change detection tasks， including overall accuracy （OA）， Kappa coefficient （KC）， precision （Pre）， recall （Rec）， and F1-score （F1）. The proposed method achieves Kappa coefficients of 0.8102， 0.7355， and 0.7635 on the three datasets， respectively. Compared with the second-best method， it yields performance gains of 1.09%， 2.62%， and 1.74%. In addition， ablation studies conducted on all three datasets further verify the effectiveness of the proposed modules. The ablation experiments indicate that the frequency feature enhancement module， the cross-temporal correlation-guided interaction module， and the spatial–spectral coupled interaction module play key roles in improving change detection accuracy by effectively suppressing pseudo-changes and enhancing the perception of subtle and multi-scale forest changes. The proposed method consistently outperforms mainstream approaches across all datasets. These results highlight the advantage of the proposed design in achieving accurate and stable forest change detection.ConclusionThis paper proposes a frequency-enhanced hierarchical interaction network for forest change detection in remote sensing imagery. By integrating bi-temporal frequency-domain feature enhancement， cross-temporal change extraction， and collaborative interaction among multi-level change features， the proposed method effectively improves forest change detection performance. Experimental results show that the proposed approach consistently outperforms existing mainstream methods across multiple forest change detection datasets， demonstrating strong consistency and stability. Benefiting from the synergistic design of frequency enhancement and hierarchical interaction， the method effectively suppresses style interference caused by bi-temporal imaging condition discrepancies， leading to a significant reduction in pseudo-change false alarms. Meanwhile， in complex forest scenarios， the proposed approach exhibits enhanced capability in perceiving and localizing subtle change regions as well as change objects at different spatial scales， enabling accurate identification of both local structural variations and large-scale evolutionary processes. These results indicate that the proposed method achieves strong robustness and applicability under complex remote sensing imaging conditions， providing a reliable solution for forest change monitoring and related applications.关键词：multispectral remote sensing images;forest change detection;frequency features;feature interaction;multi-scale change perception46|45|0更新时间：2026-07-02 -
摘要：ObjectiveWith the rapid advancement and widespread proliferation of facial forgery technologies， social issues such as the dissemination of misinformation and identity fraud have become increasingly severe. Regarding existing forgery detection methods based on face image reconstruction， most employ multi-task learning to reconstruct either real or fake faces in isolation， failing to explicitly magnify the reconstruction discrepancies between the two. Consequently， the detection accuracy and generalization performance of these network models are limited. To address these issues， a method called “differential reconstruction-driven residual guidance for face forgery detection” is proposed.MethodFirstly， to explicitly magnify the differences between real and fake faces during the image reconstruction process， a contrastive differential reconstruction network （CDRNet） is introduced. This method constructs clear and blurred image reconstruction targets for real and fake faces， respectively， guiding the network to focus on the high-frequency regions of faces with obvious forgery traces during reconstruction， thereby enhancing the overall model's ability to distinguish between real and fake faces. Secondly， addressing the issue that existing detection methods fail to fully exploit the guidance information from reconstructed residual maps， a residual dual-domain guiding module （RDDGM） is designed. This module deeply integrates spatial and high-frequency domain features， utilizing the reconstructed residual information to guide the fused dual-domain features， thereby enhancing the network model's capability to capture subtle forgery traces. In addition， in order to enable the model to learn the common forgery features among different forgery methods， a text-aware loss module （TALM） is introduced. Through the guidance of text modal information， the results of contrastive differential reconstruction are further optimized， and the generalization performance of the network model for unknown forgery methods is significantly improved. The main contributions of this paper are summarized as follows： 1） In order to explicitly magnify the differences between real and fake faces before and after reconstruction， the contrastive differential reconstruction network （CDRNet） was designed to construct differential reconstruction targets for real and fake faces respectively. The model's ability to distinguish between real and fake human faces has been enhanced. 2） In order to enhance the ability of the network model to capture subtle forgery traces， the residual dual-domain guidance module （RDDGM） is proposed， which uses the reconstructed residual information to guide the fusion features of the high-frequency domain and the spatial domain of the image. Fully explore the subtle forgery traces in the image domain and the high-frequency domain. 3） In order to enhance the generalization ability of the network model for unknown forgery methods， the text-aware loss module （TALM） is proposed. Text modal information is introduced to further optimize and compare the differential reconstruction results， promoting the model to learn the common forgery features among various forgery methods. The generalization ability of the network model has been significantly enhanced. 4） Experimental results demonstrate that the proposed method achieves highly competitive performance in both in-domain and cross-domain evaluations across multiple datasets， including FaceForensics++ （FF++）， Celeb-DF V1 and V2， DeepFake Detection Challenge （DFDC）， DeepFake Detection Challenge Preview （DFDCP）， and Deepfake Detection （DFD）. Based on the Dlib library， this paper extracts 32 facial frames from each video in the training set and 64 frames from each video in the test set. All images are uniformly resized to 224×224， and their pixel values are normalized to before being fed into the network. In terms of evaluation metrics， following prior research， this study primarily adopts accuracy （ACC） and area under the ROC curve （AUC） to assess network performance. Experimental verification confirms that the AdamW optimizer is used for training the network model， with the initial learning rate and batch size set to 1E-4 and 8， respectively. The weights of the image encoder are initialized using an EfficientNet-B4 model pre-trained on ImageNet. The radius of the filter frequency domain mask in CDRNet is set to 16. During training， multiple data augmentation strategies are employed， including random horizontal flipping， small-angle rotation， random cropping， scaling， and color jittering. The proposed method is implemented based on the PyTorch framework， and the model is trained using a single NVIDIA GeForce RTX 2080Ti GPU.ResultIn intra-domain experiments， compared with the best-performing comparison method， the proposed method improves ACC and AUC by 2.83% and 1.75%， respectively， thereby demonstrating superior performance in identifying forgeries within the same data distribution. In cross-domain experiments， the method was rigorously tested on 5 public datasets and compared with 13 typical methods， achieving a 1.75% improvement in average AUC. Comprehensive ablation studies further demonstrate that the proposed modules， including CDRNet， RDDGM， and TALM， all significantly contribute to enhancing the overall detection performance and generalization of the face forgery detection model.ConclusionThis work introduces a novel integration of contrastive learning and image differential reconstruction for face forgery detection， substantially improving the model's generalization accuracy on unseen forgery methods. Specifically， the CDRNet amplifies the discrepancies between real and forged faces during the reconstruction process to bolster discriminative capability， while the RDDGM leverages reconstruction residual information to guide dual-domain feature fusion， thereby improving the perception of subtle forgery traces. Furthermore， the TALM introduces text modality information to learn generic forgery features across different methods， enhancing generalization to unknown forgeries. Although experimental results demonstrate that the proposed method outperforms existing approaches in both in-domain and cross-domain scenarios， it currently faces challenges regarding inference efficiency due to model complexity， and the use of fixed text prompts limits the full potential of TALM's cross-modal guidance. Consequently， future work will focus on lightweight network optimization and the construction of a dynamic text prompt library to strengthen cross-modal correlations， aiming to further improve the model's efficiency and practicality.关键词：DeepFake detection;face forgery detection;multi-task learning;contrastive differential reconstruction;residual dual-domain guidance67|67|0更新时间：2026-07-02 -
摘要：ObjectiveFew-shot image classification aims to recognize novel visual categories using only a few labeled samples per class. It is an important research topic for improving the generalization ability of visual recognition systems under low-resource conditions. Existing few-shot learning methods have achieved considerable progress through metric learning， meta-learning， data augmentation， and pretrained visual representation transfer. However， when the number of labeled support images is extremely limited， especially in the 1-shot setting， purely visual methods often suffer from unstable prototype estimation and insufficient discrimination between visually similar classes. Recently， image-text collaborative learning and semantic-enhanced few-shot learning have provided new opportunities for addressing this limitation. By introducing class names， attributes， textual prompts， or language model representations， semantic information can serve as an additional prior to compensate for insufficient visual evidence. Nevertheless， most existing semantic-enhanced methods still rely mainly on coarse-grained textual information， such as category names or short prompt templates. Such semantic information usually describes only the global concept of a class and is inadequate for capturing local attributes， appearance details， part-level cues， and subtle inter-class differences. Moreover， when fine-grained textual descriptions are directly generated or introduced， they may contain redundant expressions， ambiguous words， and weakly discriminative descriptions. Another important issue is the modality gap between textual semantics and visual features. Simple feature concatenation or additive fusion is insufficient for fully aligning multi-grained semantic information with visual representations. Therefore， how to construct high-quality class semantics and how to deeply align semantic representations with visual features remain key challenges for semantic-enhanced few-shot image classification. To address these issues， this paper proposes a few-shot image classification framework named text-visual semantic multi-branch alignment network （TSMA-Net）， which aims to improve the robustness and discriminability of class prototypes by combining multi-angle semantic mining， semantic refinement， adaptive semantic fusion， and multi-branch cross-modal alignment.MethodThe proposed TSMA-Net consists of two main components： a text-visual semantic association module and a multi-branch alignment module. The semantic association module first mines fine-grained category descriptions from multiple complementary perspectives and refines them to reduce redundancy and ambiguity. These fine-grained semantics are then adaptively fused with class-name semantics to obtain robust and discriminative multi-grained textual representations. The multi-branch alignment module further projects the fused semantics into multiple subspaces and interacts them with visual features， enabling more sufficient cross-modal alignment than direct additive fusion. A residual recalibration structure is introduced to suppress noisy cross-modal responses， and visual prototypes and semantically enhanced prototypes are jointly used for query classification.ResultExtensive experiments are conducted on four standard few-shot image classification benchmarks， including miniImageNet， tieredImageNet， CIFAR-FS， and FC100. The evaluation follows the widely used 5-way 1-shot and 5-way 5-shot settings. Compared with existing visual-only and semantic-enhanced few-shot learning methods， TSMA-Net achieves competitive or superior performance across different datasets. In particular， compared with SimpleFSL using the same Visformer-Tiny backbone， TSMA-Net improves the classification accuracy by 2.11% and 0.47% on miniImageNet under the 1-shot and 5-shot settings， respectively. On tieredImageNet， the corresponding improvements are 2.35% and 0.49%. On CIFAR-FS， TSMA-Net obtains gains of 1.21% and 0.28%， and on FC100， it improves the accuracy by 0.54% and 0.10%. These results demonstrate that the proposed method can effectively improve few-shot classification performance across datasets with different image distributions， category granularity， and semantic complexity. The performance gains are more pronounced in the 1-shot setting， indicating that the proposed semantic mining and multi-branch alignment strategy is particularly beneficial when visual evidence is extremely scarce. Ablation experiments further confirm the contribution of each component. The text-visual semantic association module improves the quality of class textual representations by introducing multi-angle fine-grained semantic descriptions， while the multi-branch alignment module enhances cross-modal interaction and improves prototype discriminability. The analysis of the fusion factor shows that an appropriate balance between class-name semantics and fine-grained semantics is important for stable semantic representation. The branch number analysis indicates that multiple semantic projection branches can improve performance by decomposing semantic information into different subspaces， whereas too many branches may introduce redundancy and over-fragment the semantic representation. The residual recalibration experiment also shows that recalibrating the fused cross-modal representation helps suppress noise and improve robustness. In addition， visualization results based on t-SNE show that TSMA-Net forms more compact intra-class clusters and clearer inter-class separation than the baseline method， which further verifies the effectiveness of the proposed semantic association and alignment mechanisms.ConclusionThis paper proposes TSMA-Net for semantic-enhanced few-shot image classification. Unlike existing methods that mainly use category names or short prompts as textual priors， TSMA-Net explicitly mines multi-angle fine-grained semantic descriptions from a large language model and refines them into concise， relevant， and discriminative class semantics. By adaptively integrating class-name semantics and fine-grained descriptions， the method constructs a more robust multi-grained textual representation. Furthermore， the proposed multi-branch alignment module projects semantic information into multiple subspaces and performs deeper interaction with visual features， thereby alleviating the modality gap between text and vision. The residual recalibration structure further improves the stability of cross-modal fusion， and the dual-path prototype classification strategy enhances decision robustness. Experimental results on four benchmark datasets demonstrate that TSMA-Net can learn more discriminative class prototypes under limited supervision and consistently improve few-shot classification performance. The advantage is especially evident in the 1-shot scenario， where additional semantic priors are most needed. Overall， the proposed framework provides an effective way to exploit large language model-generated fine-grained semantics for few-shot visual recognition and offers a useful reference for future research on multimodal semantic alignment under data-scarce conditions.关键词：few-shot learning;few-shot image classification;prototype learning;multimodal learning;cross-modal semantic alignment;large language model43|65|0更新时间：2026-07-02 -
摘要：Multimodal detection using unmanned Aerial Vehicles （UAVs） has demonstrated significant practical value in applications such as border surveillance， disaster relief， and urban security. This is primarily due to their advantages of high mobility， rapid deployment， and all-weather observation capabilities. Compared with single-modality perception methods， multimodal object detection can simultaneously utilize the textural information provided by visible （VIS） images and the thermal radiation characteristics captured by infrared （IR） images. This complementary characteristic significantly improves detection performance in complex environments. However， most existing studies are developed under an idealized assumption that IR and VIS images are strictly aligned at the pixel level. Such an assumption rarely holds in real UAV scenarios. In practical applications， factors including sensor installation deviations， viewpoint differences， flight attitude changes， and platform vibrations often introduce nonlinear spatial offsets between modalities. These discrepancies further lead to cross-modal feature misalignment， semantic inconsistency， and severe performance degradation. Existing infrared-visible （IR-VIS） datasets also exhibit several inherent limitations. Most datasets are collected in ground-level or low-altitude scenarios and therefore lack sufficient representation of high-altitude UAV viewpoints. In addition， object-scale distributions are often highly imbalanced， while category settings remain relatively limited. As a result， these datasets cannot adequately reflect the diverse object distributions and complex environmental conditions encountered in real-world UAV tasks. Under such conditions， mainstream multimodal detection methods often suffer from issues such as localization drift， missed detections， and misclassifications when processing real-world UAV data. This is because real-world data typically involves spatial disparities， dynamic scale variations， and complex background interference. These issues severely limit the practical value of existing methods. To address the above limitations， this paper presents DIV， a multimodal UAV object detection dataset specifically designed for real-world weakly aligned scenarios. The dataset was collected using UAV platforms equipped with infrared and visible sensors across diverse real-world environments. Unlike existing strictly aligned datasets， DIV intentionally preserves the spatial inconsistencies caused by systematic errors and dynamic flight factors， thereby providing a more realistic representation of practical deployment conditions. The proposed dataset emphasizes the characteristic of “weak alignment”， enabling more effective evaluation of model robustness and generalization under cross-modal offset conditions. In terms of data content， DIV exhibits both diversity and complexity. The dataset contains objects ranging from extremely small objects with very limited pixel coverage to large-scale objects occupying substantial image regions， allowing comprehensive evaluation of scale adaptability. The category settings include person， non-motorized vehicle， and multiple types of motor vehicles， which improves both task complexity and practical relevance. In addition， the dataset covers a variety of environments， including urban， mountainous， and rural areas. To further simulate real-world conditions， the data are subdivided into daytime， nighttime， and low-light scenarios. To evaluate the challenges and effectiveness of DIV， several mainstream multimodal object detection algorithms were selected for experimental analysis. Experimental results demonstrate that methods achieving strong performance on conventional strictly aligned datasets experience noticeable performance degradation on DIV， particularly in small object detection and complex scene perception tasks. These findings indicate that many existing methods heavily depend on accurate cross-modal alignment and lack sufficient capability to model realistic UAV flight conditions. Furthermore， although some approaches perform well under specific metrics or limited scenarios， their overall perception capability remains insufficient when simultaneously confronted with multi-scale targets， multiple object categories， and complex environmental interference. The results further reveal a coupling relationship among weak multimodal alignment， scale variation， and environmental complexity. These factors jointly affect the quality of multimodal feature representation and fusion. Therefore， optimization at a single level alone is insufficient to comprehensively address these challenges. For practical UAV applications， there is an urgent need to develop a multi-dimensional collaborative optimization framework that improves multimodal modeling and fusion capability at the data， feature， and task levels simultaneously. By breaking the conventional ideal-alignment assumption， the proposed DIV dataset effectively bridges the gap between multimodal algorithm research and real-world UAV deployment scenarios. This work provides a foundational benchmark for future research on robust multimodal perception in realistic environments.关键词：aerial remote sensing images;multimodal image fusion;Cross-Modal Alignment;infrared and visible;object detection22|25|0更新时间：2026-07-02 -
From Knowledge-Driven to Data-Driven： A Comprehensive Review of Remote Sensing Image Registration AI导读
摘要：As a fundamental and critical task in remote sensing image processing， image registration has consistently been a hot research topic among scholars both domestically and internationally， holding significant theoretical and practical value. This task aims to geometrically align two or more images of the same scene that were acquired at different times， by different sensors， or from different viewpoints， thereby providing a reliable data foundation for subsequent applications such as image mosaicking， image fusion， object detection， change detection， and guidance matching. However， remote sensing images often face significant challenges such as geometric distortion， nonlinear radiometric differences， complex object occlusion， and various types of noise interference. These factors interact with one another， making it difficult to accurately extract and match corresponding features between images. This severely hinders the achievement of high-precision， robust registration and poses greater challenges to the adaptability and robustness of existing registration methods in complex scenes. This paper systematically reviews remote sensing image registration algorithms. First， it outlines the basic concepts and technical framework of remote sensing image registration. Second， existing registration methods are categorized into two major groups for systematic analysis： knowledge-driven expert design methods and data-driven deep learning methods. Knowledge-driven， expert-designed methods rely on manually designed features and transformation models. Through explicit feature extraction and iterative optimization， they achieve interpretable registration in traditional scenarios. These methods can be further subdivided into template-based registration methods， feature-based registration methods， and hybrid registration methods. Template-based registration methods select a template window on the reference image， use its center as the point to be matched， and search for matches on the target image based on similarity criteria to identify corresponding points. These points are then used to solve for geometric transformation parameters， thereby completing remote sensing image registration. Feature-based registration methods identify and effectively describe salient features in remote sensing images. These features are used to establish correspondences between them and， subsequently， to construct geometric transformation relationships between the images. Hybrid registration methods leverage the advantages of multiple techniques to perform remote sensing image registration， thereby compensating for the shortcomings of single-method approaches. Most of these methods integrate various types of image information， such as gradient and phase information， to improve upon classical methods—such as scale-invariant feature transformation—in either the spatial domain or the transform domain， or to enhance registration accuracy by integrating multiple registration strategies. However， knowledge-driven， expert-designed methods suffer from issues such as heavy reliance on specialized knowledge， poor adaptability to complex scenes， and significant limitations in feature selection， making it difficult to meet the registration demands of large-scale， diverse remote sensing images. In recent years， with the continuous advancement of deep learning technology， numerous deep learning network architectures have become increasingly prevalent in the field of image processing， presenting new opportunities for remote sensing image registration methods. Consequently， data-driven deep learning registration methods have emerged. These methods can be further categorized into ensemble learning methods， end-to-end learning methods， style transfer methods， and large-model-driven methods. Ensemble learning methods integrate deep learning networks into template-based or feature-based registration frameworks to generate more robust and efficient feature representations or similarity metrics， thereby facilitating subsequent registration. End-to-end learning methods utilize deep learning networks to perform end-to-end learning from image pair inputs to target outputs， employing joint optimization to accomplish feature extraction， matching， or direct prediction of geometric transformation parameters between images. Style transfer methods use deep learning models， such as generative adversarial networks， to perform style transfer on remote sensing images， significantly reducing radiometric or textural differences between multimodal remote sensing images and creating more consistent image characteristics for subsequent registration operations. Large-model-driven methods leverage the powerful feature representation capabilities and cross-modal generalization abilities of pre-trained large models， providing a novel solution path for addressing challenges such as complex geometric distortions and nonlinear radiometric differences in remote sensing images. By learning directly from the data， these methods can solve image matching and registration problems more efficiently. They utilize deep learning networks to automatically learn feature or mapping relationships between image pairs， extract high-level features for registration， or directly predict transformation models end-to-end， thereby effectively addressing complex scenarios such as significant geometric distortion and nonlinear radiometric differences. Furthermore， this paper systematically reviews and analyzes datasets for multimodal image registration tasks. By summarizing representative datasets in the field of multimodal registration and focusing on their modal composition， data scale， spatial resolution， covered scenarios， and applicability， it provides an effective reference for researchers to select appropriate datasets for registration tasks， thereby facilitating systematic analysis and fair comparison of different methods under unified conditions in future research. Finally， based on the aforementioned review of methods in the field of remote sensing image registration， this paper outlines future development trends in the field.关键词：remote sensing image registration;knowledge-driven;data-driven;feature matching;deep learning75|102|0更新时间：2026-06-22 -
摘要：ObjectiveSynthetic Aperture Radar （SAR） to optical image translation （SAR-to-Optical， S2O） is essential for full-time and all-weather earth observation. Optical images provide rich texture information and intuitive visual representations， which are important to a wide range of downstream tasks such as land-cover classification and urban monitoring. However， optical sensors are highly vulnerable to clouds， fog， and illumination changes， limiting continuous data acquisition. In contrast， SAR sensors operate in the microwave spectrum and can acquire images under all-weather and all-time conditions. Consequently， S2O translation can significantly enhance the interpretability of SAR data and facilitate multimodal remote sensing applications. Despite recent progress in cross-modal image translation， achieving high-quality S2O translation remains challenging. First， SAR images inevitably contain multiplicative speckle noise that is strongly coupled with structural information， and together with the substantial semantic gap caused by fundamentally distinct imaging mechanisms between SAR and optical sensors， this leads to insufficient cross-modal feature fusion and representation capability. Second， although diffusion models have recently demonstrated impressive performance in image generation tasks， their high computational overhead and inefficient conditional fusion mechanisms limit their applicability for real-time processing on resource-limited remote sensing platforms. To address these issues， we propose a Spatial-Frequency Strongly-Sparse Guided Diffusion Model （SFSG-Diff） for efficient and high-quality SAR-to-optical image translation.MethodThe proposed SFSG-Diff framework follows the standard diffusion generation paradigm， consisting of a forward diffusion process and a reverse denoising process. In the forward process， Gaussian noise is progressively added to a clean optical image until the data distribution approaches pure noise. During the reverse process， a conditional denoising network iteratively removes the noise to reconstruct the target optical image. To improve both generation quality and computational efficiency， SFSG-Diff introduces spatial-frequency guided feature extraction and a strongly sparse conditional fusion mechanism. To effectively suppress SAR speckle noise and enhance structural representation， a Multi-scale Spatial-Frequency Denoising Encoder （MDE） is designed to extract robust conditional features from the input SAR image. The encoder adopts a dual-branch architecture composed of a spatial feature extraction branch and a frequency feature extraction branch. The spatial branch focuses on capturing local textures and contextual information through multi-scale convolutional operations， while the frequency branch transforms SAR images into the frequency domain to capture global structural information that is less sensitive to speckle noise. Since speckle noise is mainly concentrated in high-frequency components， whereas meaningful structural information is typically distributed in low and mid-frequency regions， the spatial-frequency joint representation enables effective separation of noise and structural features. The outputs of both branches are fused across multiple scales to produce robust conditional representations that guide the diffusion model during image generation. To further improve conditional guidance efficiency， we introduce a lightweight Strong-Sparse Semantic Fusion （SIF） pattern. Instead of densely integrating conditional features into all feature channels， the SIF module performs channel-wise selection and adaptive attention-based modulation to identify and fuse the most informative feature components. This sparse guidance mechanism not only reduces computational overhead but also improves cross-modal semantic alignment by emphasizing the most relevant structural cues. Furthermore， to enhance training stability and generation quality， we adopt a two-stage training strategy， leveraging a joint loss， including a simplified mean squared error loss， perceptual loss， focus frequency loss， and adversarial loss.ResultTo evaluate the performance of the proposed framework， extensive experiments are conducted on three publicly available datasets， including SEN1-2， QXS-SAROPT， and WHU-OPT-SAR. The experimental results demonstrate that SFSG-Diff consistently outperforms several state-of-the-art GAN-based and diffusion-based image translation models. Quantitative evaluation results demonstrate that SFSG-Diff consistently outperforms existing state-of-the-art methods across multiple evaluation metrics. On the SEN1-2 dataset， the proposed SFSG-Diff achieves a PSNR improvement of 0.77 dB over the best competing method， while the Structural Similarity Index （SSIM） improves by 17.8%. In addition， the perceptual quality metrics show substantial improvements， with LPIPS and Fréchet Inception Distance （FID） reduced by 8.0% and 17.6%， respectively. These results indicate that the proposed model not only improves reconstruction fidelity but also produces images with higher perceptual realism. Similar performance gains are observed on the QXS-SAROPT and WHU-OPT-SAR datasets， which include more complex urban structures and higher spatial resolution imagery. The proposed SFSG-Diff demonstrates strong robustness across diverse scenes and maintains consistent advantages in both structural similarity and perceptual quality. Visual comparisons further confirm the superiority of SFSG-Diff. Compared with existing methods， the SFSG-Diff generates optical images with clearer structural boundaries， more realistic textures， and fewer noise artifacts. In particular， the model exhibits improved performance in challenging regions such as dense urban areas and vegetation-rich scenes. In addition to generation quality， the computational efficiency of the proposed framework is also evaluated. Owing to the lightweight SIF pattern and the two-stage training strategy， SFSG-Diff achieves significantly faster inference compared with conventional diffusion models. The average inference time per image is approximately 0.21 seconds， which represents a 69.1% reduction in computation time compared with representative diffusion-based baselines.ConclusionWe presents SFSG-Diff， a novel framework for one-step S2O translation， which addresses key challenges in cross-modal image generation by leveraging spatial-frequency joint feature modeling and lightweight Strong-Sparse conditional guidance. The MDE effectively suppresses speckle noise and enhances structural feature representation， while the SIF pattern improves cross-modal alignment and reduces computational complexity. Extensive experiments on multiple benchmark datasets demonstrate that the proposed method significantly outperforms existing approaches in both quantitative metrics and visual quality. Moreover， the framework achieves substantial improvements in inference efficiency， making it more suitable for practical remote sensing applications with limited computational resources. Overall， the proposed SFSG-Diff framework provides an effective solution for robust and efficient SAR-to-optical image translation and offers new insights into the integration of spatial-frequency modeling and sparse conditional guidance within diffusion-based generative models.关键词：SAR-to-optical image translation;diffusion model;feature fusion;remote sensing imagery;multi-scale encoder;strongly sparse guidance60|142|0更新时间：2026-06-22 -
摘要：ObjectiveVisual saliency prediction aims to simulate the human visual attention mechanism by estimating the spatial probability distribution of gaze points in complex scenes. This task is fundamental to various downstream applications， including autonomous driving， robot navigation， and image compression. While Convolutional Neural Networks （CNNs） have long been the backbone of saliency modeling due to their proficiency in local feature extraction and multi-scale representation， they are inherently limited by their local receptive fields. This restriction often leads to "fixation fragmentation" and background false alarms in cluttered environments. Conversely， Transformer-based models have introduced global dependency modeling via self-attention； however， their quadratic computational complexity becomes a significant bottleneck when processing high-resolution saliency maps. Recently， State Space Models （SSMs）， specifically Mamba， have emerged as a promising alternative， offering linear-time complexity while maintaining long-range context. Nevertheless， standard Mamba architectures are designed for 1D sequences， which inherently disrupts the 2D spatial topological continuity of images. Furthermore， during long-range state propagation， background noise is often indiscriminately amplified alongside salient features. Additionally， the continuous bilinear upsampling typically employed in decoding stages causes a "structural over-smoothing" effect， which diminishes the peak energy of salient regions and blurs boundaries. To address these multi-faceted challenges， this study proposes a novel visual saliency prediction network named Spatio-Spectral Uncertainty-Gated Mamba Network （S²UG-Mamba）. The goal is to achieve an optimal synergy between global context modeling， noise suppression， and high-frequency structural restoration within a linear computational framework.MethodThe proposed S²UG-Mamba follows a "backbone extraction， encoder enhancement， multi-scale decoding， frequency modulation， and prediction" pipeline. We employ an ImageNet-pretrained ConvNeXt-Tiny as the backbone to extract multi-level features. In the encoder enhancement phase， we introduce the Uncertainty-Aware State Space Enhancement Module （UA-SSM）. To preserve 2D spatial continuity， UA-SSM utilizes a bidirectional orthogonal serpentine scanning strategy， which unfolds features in both horizontal and vertical directions to capture omnidirectional contextual interactions. To mitigate the propagation of background noise， we design a dual-view uncertainty proxy within UA-SSM. This proxy consists of a spatial variance component （capturing local texture fluctuations in 3x3 windows） and a channel variance component （measuring semantic activation disagreements）. These proxies are fused to generate a pixel-level confidence gate G， which dynamically modulates the Mamba input sequence， thereby suppressing responses in high-uncertainty background regions.In the decoding stage， to counteract the smoothing effects of upsampling， we propose the Semantic-Guided Dynamic Frequency Modulation （SDFM） module. Unlike conventional static filters， SDFM employs a joint logical routing mechanism that combines deep semantic priors from the encoder with local content features from the decoder. This routing mechanism dynamically synthesizes a set of learnable complex frequency filters. The feature maps are transformed into the frequency domain via a real Fast Fourier Transform （rFFT） after StarReLU activation， multiplied by the adaptive filters， and then mapped back to the spatial domain using an inverse rFFT. This residual frequency compensation explicitly restores high-frequency structural details and tightens the energy concentration of fixation peaks. The entire network is optimized using a joint loss function of Kullback-Leibler （KL） divergence and Linear Correlation Coefficient （CC）.ResultExtensive experiments were conducted on five benchmark datasets： SALICON， MIT300， MIT1003， CAT2000， and TORONTO. On the SALICON test set （LSUN'17）， S²UG-Mamba achieved state-of-the-art performance， reducing the KL divergence to 0.176 while reaching a CC of 0.918 and an NSS of 2.007. On the challenging MIT300 blind test set， our model yielded a CC of 0.829 and a KL of 0.370， outperforming the advanced GSGNet by 2.2% and 9.8%， respectively. To evaluate cross-domain generalization， we performed zero-shot testing on the MIT1003 dataset using weights trained solely on SALICON； S²UG-Mamba attained an NSS of 2.386 and a CC of 0.6849， demonstrating robust learning of universal saliency priors. Ablation studies confirmed the efficacy of each component： the introduction of UA-SSM improved the NSS from 1.9627 to 2.0152， while SDFM significantly reduced the KL divergence. The complexity and efficiency analysis demonstrates that our model achieves favorable performance in terms of computational cost and GPU memory consumption. Meanwhile， it maintains comparable parameter scale and inference speed with state-of-the-art methods， fully validating the efficient modeling capability of the Mamba architecture. Qualitative visualizations further illustrated that S²UG-Mamba effectively suppresses non-salient background clutter in densely textured scenes and produces sharper， more compact salient boundaries compared to existing methods.ConclusionThis study presents S²UG-Mamba， a linear-complexity network that synergistically integrates uncertainty-aware state space modeling and semantic-guided frequency modulation for visual saliency prediction. By employing orthogonal scanning and a dual-view uncertainty gating mechanism， the encoder efficiently captures long-range dependencies while adaptively suppressing noise. Simultaneously， the SDFM leverages high-level semantic guidance to perform adaptive frequency filtering， effectively restoring structural details lost during spatial upsampling. The experimental results across multiple benchmarks validate that S²UG-Mamba provides a superior balance between prediction accuracy， cross-domain robustness， and computational efficiency. Future research will explore hardware-friendly parallel 2D scanning algorithms and low-annotation learning paradigms to further enhance the model’s practical deployment potential.关键词：saliency prediction;State Space Model;Mamba;uncertainty-aware;frequency domain enhancement84|116|0更新时间：2026-06-22 -
Anomaly detection in aerial and ground videos using dual-stream network with memory enhancement AI导读
摘要：ObjectiveVideo anomaly detection （VAD） aims to automatically identify events that deviate from learned normal patterns in video sequences and has become a critical component of intelligent surveillance systems. It plays an important role in public security， intelligent transportation， industrial inspection， and urban management. With the rapid deployment of unmanned aerial vehicles （UAVs）， large-scale aerial video data have become increasingly available， creating new opportunities for anomaly detection. However， compared with conventional fixed-camera surveillance videos， UAV videos exhibit substantial viewpoint variations， severe object-scale changes， dynamic backgrounds， and complex motion patterns， which significantly increase the difficulty of anomaly detection. Although recent deep-learning-based methods have achieved promising performance， most existing approaches focus primarily on appearance modeling while insufficiently exploiting motion information， despite the fact that abnormal events are often characterized by unusual motion patterns such as sudden acceleration， abrupt direction changes， illegal crossings， or abnormal trajectories. Furthermore， reconstruction-based and prediction-based methods commonly rely on generative models to learn normal patterns and identify anomalies through reconstruction or prediction errors. These models often suffer from the over-generalization problem， where abnormal samples can still be reconstructed or predicted with relatively low errors， thereby reducing the discriminative capability between normal and abnormal events. Therefore， developing a unified framework that can effectively model both appearance and motion information while enhancing the representation of diverse normal patterns remains an important challenge for anomaly detection in both ground-based and aerial surveillance scenarios.MethodTo address the above issues， a dual-stream memory-enhanced network for ground-to-air video anomaly detection is proposed. The framework adopts a dual-encoder single-decoder architecture. First， a pretrained FlowNet2 optical-flow estimation network is employed to extract motion information from consecutive video frames. To prevent overfitting and reduce training complexity， the FlowNet2 parameters remain fixed throughout the training process. The original RGB frames and corresponding optical-flow maps are then fed into two independent encoders， namely an appearance encoder and a motion encoder， to extract multi-scale appearance features and motion features， respectively. To establish a stronger correlation between appearance and motion representations， a variance-attention appearance-motion fusion module is designed. Specifically， the spatial variance of motion features is computed to characterize the intensity of motion changes. Regions exhibiting large motion variance， such as sudden acceleration， abrupt direction changes， or unusual object movements， are assigned higher attention weights. These weights are subsequently used to enhance appearance features， enabling the network to focus on potentially anomalous regions while suppressing irrelevant background information. Through this mechanism， appearance and motion features can be deeply fused at multiple scales， resulting in more discriminative feature representations. To alleviate the over-generalization problem and improve the learning of diverse normal patterns， a memory-enhanced module with a dynamic update strategy is further introduced. The memory module consists of multiple learnable memory items that store representative normal feature prototypes. During the memory-reading process， query features extracted from high-level fused representations retrieve the most relevant memory items according to cosine similarity. The retrieved memory features are then combined with the original query features to generate memory-enhanced representations. During memory updating， a regularization score derived from prediction errors is employed to distinguish normal frames from potential abnormal frames. Only features associated with normal frames are allowed to update memory items， thereby preventing abnormal patterns from contaminating the memory bank. This strategy enables the memory module to continuously learn diverse normal behaviors while preserving its discriminative capability. In the decoding stage， multi-scale fused features are delivered to the decoder through skip connections， allowing low-level spatial details and high-level semantic information to be jointly utilized for future-frame prediction. Furthermore， efficient channel attention modules are embedded after the first three upsampling layers to adaptively recalibrate channel-wise feature responses and enhance prediction quality with negligible computational overhead. The entire network is optimized using a joint loss function composed of prediction loss， optical-flow loss， feature compactness loss， and feature separation loss. These losses collaboratively encourage accurate future-frame prediction， motion consistency preservation， compact intra-class representations， and discriminative memory-item distributions.ResultsExtensive experiments were conducted on three benchmark ground-surveillance datasets， namely UCSD Ped2， CUHK Avenue， and ShanghaiTech， as well as the Drone-Anomaly UAV dataset. Experimental results demonstrate that the proposed method achieves state-of-the-art or highly competitive performance across different scenarios. On the UCSD Ped2 dataset， the proposed method achieved an AUC of 98.8%， outperforming representative methods such as MemAE （94.1%）， MNAD （97.0%）， and AMMC （96.6%）. On the CUHK Avenue dataset， an AUC of 89.1% was obtained， slightly surpassing DEDDnet （89.0%） and exceeding most recent prediction-based approaches. On the challenging ShanghaiTech dataset， which contains more than 270，000 training frames， multiple scenes， and diverse anomaly categories， the proposed method achieved an AUC of 74.7%， demonstrating strong robustness under complex environments. To further evaluate the generalization capability in aerial surveillance scenarios， experiments were conducted on the Drone-Anomaly dataset， which contains multiple UAV-captured inspection and traffic-monitoring scenes. The proposed method achieved 94.76% AUC with an EER of 0.09% on Railway Inspection and 91.41% AUC with an EER of 0.10% on Farmland Inspection， significantly outperforming Frame-Pred， MNAD， ANDT， MKD， and SSPCAB. In addition to detection accuracy， computational efficiency was evaluated. The proposed model contains only 20.39M parameters and achieves an inference speed of 62.66 FPS on an NVIDIA RTX4090 GPU， outperforming several existing methods in terms of both efficiency and accuracy. Even when the computational cost of FlowNet2 is included， the end-to-end processing speed remains approximately 39.76 FPS， satisfying the requirements of most real-time surveillance applications. Ablation studies further validate the effectiveness of the proposed components. Additional loss-function ablations confirmed that the combination of prediction loss， optical-flow loss， compactness loss， and separation loss yields the best detection performance.ConclusionThis paper presents a dual-stream memory-enhanced framework for video anomaly detection in both ground-based and UAV surveillance scenarios. By jointly modeling appearance and motion information through a variance-attention fusion mechanism and learning diverse normal patterns via a dynamically updated memory module， the proposed method effectively alleviates the over-generalization problem and improves anomaly discrimination in complex dynamic environments. Extensive experiments on multiple benchmark datasets demonstrate its robustness， generalization capability， computational efficiency， and real-time performance. The proposed framework provides a practical and effective solution for intelligent anomaly detection across diverse surveillance applications and offers a promising foundation for future research on unified ground-to-air video understanding systems.关键词：anomaly detection;convolutional neural networks;memory networks;fusion algorithms;attention mechanisms83|101|0更新时间：2026-06-22 -
摘要：Cross-domain few-shot object detection （CD-FSOD） in aerial remote sensing aims to detect target objects when the target domain contains only a few annotated instances and differs substantially from the source domain. This setting is important for land resource investigation， disaster monitoring， traffic supervision， maritime and airport surveillance， emergency response， and military reconnaissance， because many operational remote sensing systems must be deployed in new regions， seasons， sensors， or platforms before large-scale annotation is available. Conventional object detectors usually assume abundant labeled data and a relatively stable training-test distribution. In contrast， aerial images are affected by platform altitude， satellite or unmanned aerial vehicle imaging geometry， optical or synthetic aperture radar （SAR） sensing mechanisms， spatial resolution， illumination， weather， seasonal background， geographical scene composition， and category occurrence frequency. These factors change target scale， texture， orientation， density， and contextual appearance at the same time. Consequently， a detector trained on a well-annotated source domain may suffer from severe classification confusion， localization drift， false alarms， and missed detections after being transferred to a new target domain. This paper provides a systematic survey of CD-FSOD for aerial remote sensing scenes. First， the task definition and research boundary are clarified. CD-FSOD is distinguished from classic object detection， few-shot object detection （FSOD）， cross-domain object detection， and general remote sensing object detection. Classic object detection emphasizes classification and localization under roughly independent and identically distributed data. FSOD focuses on new-category detection with limited examples but often assumes that the base and novel domains are close. Cross-domain object detection addresses domain shift but usually does not impose an extreme target-domain annotation constraint. Remote sensing object detection describes the application scenario and its characteristic difficulties， including small objects， dense object distributions， arbitrary orientations， and complex backgrounds. CD-FSOD couples the last two difficulties： the target domain is both data-scarce and distributionally shifted. Second， existing methods are organized into five technical routes. Transfer and domain-alignment methods， including domain-adaptive Faster region-based convolutional neural network （DA Faster R-CNN）， strong-weak distribution alignment （SWDA）， spatial attention pyramid network （SAPNet）， and adaptive teacher frameworks， reduce the discrepancy between source and target features by image-level， feature-level， instance-level， or teacher-student constraints. Meta-learning and metric-learning methods， including meta region-based convolutional neural network （Meta R-CNN）， attention region proposal network （Attention-RPN）， few-shot object detection via contrastive proposal encoding （FSCE）， Detect Everything with Vision Transformer （DE-ViT）， and CD-ViTO， improve the use of support samples through episodic learning， class prototypes， relation modeling， and contrastive representation learning. Generative augmentation methods， such as hallucination-based feature synthesis， multi-perspective data augmentation （MPAD）， AeroGen， Control Copy-Paste， and Domain-RAG， expand the target-domain sample distribution by generating features， foreground instances， target layouts， or domain-consistent backgrounds. Vision-language alignment methods， including RegionCLIP， PromptDet， Grounding DINO， and rich-text cross-domain multimodal FSOD， introduce category names， attributes， scene descriptions， and professional textual semantics to compensate for insufficient visual examples. Large-model methods exploit visual foundation models， detection transformer （DETR） variants， and multimodal large models such as Rex-Omni to provide broader open-vocabulary recognition and stronger initial representations. Third， the paper summarizes datasets， task protocols， and evaluation metrics. The reviewed benchmarks include multi-domain few-shot object detection （MoFSOD）， CD-FSOD， object detection in optical remote sensing images （DIOR）， dataset for object detection in aerial images （DOTA）， xView， vehicle detection in aerial imagery （VEDAI）， and the Northwestern Polytechnical University very-high-resolution 10-class dataset （NWPU VHR-10）. Their differences involve image number， category number， bounding-box type， scene coverage， object scale， and whether the protocol stresses a balanced K-shot instance setting. Common metrics include precision （P）， recall （R）， average precision （AP）， mean average precision （mAP）， AP50， AP75， mAP@50：95， novel-class mAP， and intersection over union （IoU）. Instead of listing formulae for routine metrics， this survey explains their meaning and discusses how they should be used to evaluate target-domain generalization， shot sensitivity， robustness across domains， and localization quality for small dense objects. Fourth， this review identifies several open problems. Complex domain shift should be modeled at global-scene， local-object， and category-conditional levels rather than by a single overall feature alignment. Few-shot robustness requires support-sample quality assessment， stable prototype learning， multi-scale representation， uncertainty estimation， and joint optimization of classification and localization. Current datasets and evaluation protocols remain partly disconnected from real remote sensing applications， where categories are long-tailed， annotations are incomplete， sensors are heterogeneous， and deployment constraints are strict. Multisource adaptation across optical， infrared， multispectral， and SAR data is still underexplored. Foundation models and multimodal large models offer stronger semantic priors， but they also introduce high computational cost， insufficient remote sensing domain knowledge， and limited performance on small， dense， rotated， or sensor-specific targets. To improve reuse and citation of the survey materials， a curated performance summary of representative algorithms under the aerial remote sensing CD-FSOD setting is provided in the manuscript and will be maintained through the open resource page：https：//github.com/Farenweh/CD-FSOD-Links（Github）orhttps：//gitee.com/cby1241385936/cd-fsod-links（Gitee）. Overall， CD-FSOD in aerial remote sensing is moving from isolated fine-tuning or domain-alignment strategies toward integrated frameworks that combine domain adaptation， few-shot discrimination， data generation， vision-language semantics， and foundation-model transfer. In terms of practical use， the paper also emphasizes that CD-FSOD should not be evaluated only by a single accuracy score obtained under a fixed split. A more informative protocol should report the stability of each method under different shots， different target domains， different object scales， and different sensor conditions， because an algorithm that performs well on one aerial dataset may fail when the dominant background， ground sampling distance， or object density changes. The survey therefore recommends that future benchmark construction record the source of domain shift explicitly and separate the effects of category novelty， image style， spatial resolution， and annotation scarcity. Such analysis is particularly necessary for aerial remote sensing， where missed small targets and inaccurate localization may be more damaging than a modest decrease in overall mAP. Future progress depends on remote-sensing-specific pretraining resources， reliable benchmark construction， efficient adaptation for high-resolution imagery， and stronger integration between precise detectors and general-purpose foundation models.关键词：aerial remote sensing;object detection;cross-domain few-shot object detection;vision-language alignment;Large models41|100|0更新时间：2026-06-22 -
摘要：Airborne spectral imaging， including hyperspectral （HSI） and multispectral （MSI） data from UAVs and manned aircraft， has become a cornerstone of modern remote sensing. Compared to satellites， airborne platforms offer ultra-high spatial resolution， operational flexibility， short revisit cycles， and customizable sensors， making them indispensable for precision agriculture， environmental monitoring， military reconnaissance， urban planning， geological exploration， and cultural heritage preservation. However， airborne spectral data present unique challenges： high dimensionality （especially for HSI with hundreds of bands）， strong spectral-spatial correlations， complex noise， geometric distortions from platform motion， variable illumination， and the mixed-pixel problem. Traditional methods （e.g.， SVM， random forests， physical models） struggle to fully exploit these data due to limited feature representation， the curse of dimensionality， and poor generalization across sensors or flight conditions.This paper systematically reviews intelligent processing techniques for airborne spectral images， focusing on deep learning and advanced machine learning. We first highlight the complementary roles of HSI and MSI： HSI provides rich continuous spectra for fine material identification and sub-pixel analysis， while MSI offers higher signal-to-noise ratio， lower data volume， and better real-time performance. Their synergistic use， often in a layered strategy （MSI-based coarse detection followed by HSI-based fine identification）， is a key trend. We then detail the preprocessing pipeline—radiometric calibration， atmospheric correction， geometric registration， denoising， and resolution enhancement—noting the shift from purely physical models to hybrid data-driven and physics-informed deep learning. The core of the review covers six intelligent processing tasks. For spectral-spatial feature extraction， we trace the evolution from 2D/3D CNNs and hybrid CNN-Transformer models to state-space models like Mamba， which achieve linear complexity while capturing long-range spectral dependencies， balancing accuracy and efficiency for real-time scenarios. For classification and segmentation， we discuss supervised， semi-supervised， few-shot， and self-supervised methods （e.g.， HybridSN， Spectral-Former， masked spectral modeling）； for MSI， lightweight networks （MobileNet， EfficientNet） and transfer learning dominate， with cross-domain few-shot learning addressing label scarcity. For target detection， HSI exploits subtle spectral differences for camouflaged targets and chemical agents using anomaly detection or Transformer-based detectors （e.g.， SpecDETR）， while MSI leverages YOLO and Faster R-CNN for real-time object detection； multimodal fusion （visible， thermal， spectral） further improves robustness. For change detection， Siamese networks， Transformers （BIT， ChangeFormer）， and Mamba hybrids are applied to multi-temporal images， with challenges in distinguishing real changes from illumination/atmospheric variations and incorporating physical constraints. For spectral unmixing， autoencoders learn nonlinear mixtures but lack physical constraints （non-negativity， sum-to-one， energy conservation）； recent efforts embed linear mixture or radiative transfer models into deep networks for better interpretability. For multi-source fusion， we cover heterogeneous fusion （HSI+LiDAR， HSI+SAR） using multimodal Transformers or GNNs， and spectral-spectral fusion （HSI+MSI） to generate high-resolution hyperspectral data， outlining four collaborative strategies： HSI-guided band selection， MSI-assisted dimensionality reduction， layered cascaded processing， and end-to-end joint modeling. A dedicated section addresses real-time processing and lightweight deployment on airborne edge devices （e.g.， NVIDIA Jetson）. We review model compression （pruning， quantization， knowledge distillation， neural architecture search） and evaluate backbones （MobileNet， HybridSN， Mamba， efficient Transformers） for onboard inference. Real-time HSI classification and MSI detection have been demonstrated， but real-time unmixing and change detection remain challenging. The paper surveys typical applications， emphasizing HSI/MSI complementarity： precision agriculture （early disease detection， nutrient/water stress， fruit counting）； environmental and disaster monitoring （oil spills， red tides， forest fires， air pollution， mining）； military reconnaissance （camouflage target detection， chemical agents）； urban and infrastructure （land cover classification， building change detection， traffic， infrastructure health）； geological exploration （mineral mapping， alteration zones， oil/gas microseepage）； and archaeology （subsurface relics， surface disease analysis）. We then identify five core challenges： scarcity of high-quality labeled data， insufficient model generalization， real-time constraints， lack of interpretability （black-box models ignoring physical mechanisms）， and difficulties in air-space-ground collaboration （heterogeneous data， limited bandwidth， cross-platform transfer）. Finally， we outline future directions： foundation models with self-supervised learning for spectral-spatial representation； lightweight edge intelligence via hardware-software co-design； physics-informed deep models embedding radiative transfer or mixture constraints； robust and trustworthy AI with adversarial defense and uncertainty quantification； and integrated air-space-ground collaborative observation systems for hierarchical， intelligent remote sensing. This review provides a systematic reference for researchers and engineers advancing airborne spectral image intelligent processing. The datasets mentioned in this paper have been compiled and are available at The datasets mentioned in this paper have been compiled and are available athttps：//github.com/zhaoxb2025/Airb-spe-Img.关键词：Airborne Spectral Images;hyperspectral images;multispectral images;Intelligent Processing;object detection;change detection;classification42|119|0更新时间：2026-06-22 -
摘要：ObjectiveIn real-world surveillance scenarios， the types and degrees of occlusion exhibit significant uncertainty， and different occlusion patterns damage the structural integrity and temporal continuity of gait silhouettes to varying degrees. Recognition methods that extract discriminative identity features solely from the visible regions of incomplete sequences inherently suffer from severe information limitations caused by heavy reliance on visible areas. In contrast， although gait inpainting methods can partially restore damaged silhouettes， existing generative models are mostly restricted to frame-wise 2D inpainting and suffer from blind restoration issues， resulting in unnatural trajectory breaks and temporal jitter in reconstructed sequences. To address these problems， this chapter proposes an occlusion gait inpainting and recognition method based on visibility-aware spatial-temporal diffusion. It aims to guide a 3D diffusion model for targeted inpainting via visibility-aware occlusion perception， preserving spatial-temporal coherence between consecutive frames while restoring the spatial information of each gait frame， and better extracting the structural integrity and texture authenticity of gait silhouettes in the feature space， thereby recovering highly discriminative identity representations under complex occlusion conditions.MethodA visibility-aware spatial-temporal diffusion network based on visibility perception and a 3D diffusion model is employed to restore gait sequences， maintaining spatial-temporal coherence between frames while repairing the spatial information of each gait frame. First， the occluded gait sequence is fed into a pre-trained and frozen region score estimator to perform fine-grained estimation of the visibility of each body region in each frame， yielding corresponding visibility scores. This score is then injected as a spatial guidance signal into the visibility-aware diffusion inpainting module to drive the network to capture spatial-temporal dependencies and conduct targeted spatial-temporal denoising and structural completion on occluded regions， generating inpainted sequences with complete structures and coherent dynamics. Next， the original occluded sequence and the inpainted sequence are fed into separate 3D convolutional branches for feature extraction， followed by adaptive fusion in an attention fusion module to comprehensively utilize the authentic identity information in the original sequence and the structural completion information in the inpainted sequence. Meanwhile， a visibility spatial attention mechanism is constructed based on visibility scores and applied to the backbone network via residual connections， further enhancing the model’s attention to high-confidence human body regions.ResultIn this study， the OccCASIA-B benchmark dataset is generated and extended based on the existing OccSilGait framework， covering three occlusion patterns： static object occlusion， directional occlusion， and dynamic crowd occlusion. Static occluding objects are synthesized using real-world obstacles from the Mapillary-Vistas street view dataset. Directional occlusion simulates body part missing caused by camera view limitations or pedestrian detection box truncation. Crowd occlusion is simulated by dynamic interference from other pedestrians crossing the subject’s path. Experiments on OccCASIA-B demonstrate that the proposed VAST-DRNet maintains a Rank-1 recognition accuracy of 50.6% under extremely severe occlusion with occlusion area exceeding 50%， outperforming the GaitGL model by 3.8%. In static object occlusion scenarios， the method achieves a Rank-1 accuracy of 83.1%， leading GaitGL by 11.2%. For dynamic crowd occlusion， the Rank-1 accuracy reaches 78.1%， surpassing GaitGL by 7.6%. In addition， the region score estimator achieves average MAE， RMSE， and coefficient of determination of 0.0305， 0.0543， and 0.8800 respectively on the visibility regression task， indicating its ability to accurately characterize the occlusion degree of different body parts. Quantitative evaluation of inpainting quality confirms the effectiveness of the 3D diffusion model approach. The introduction of RSE guidance improves the edge F1-score to 0.7021 and the structural similarity index （SSIM） to 88.3%， demonstrating that the model successfully recovers sharp and realistic silhouette boundaries and effectively alleviates the common "flickering" artifacts in frame-wise inpainting.ConclusionThe proposed gait recognition method in this study effectively leverages the visibility-aware 3D diffusion process to achieve high-quality， spatially-temporally consistent targeted inpainting. By bridging occlusion perception and generative inpainting， the confidence-guided fusion mechanism effectively reduces the interference of generative noise. The significant performance improvement on the OccCASIA-B dataset highlights the robustness of the method and its potential for practical deployment in uncontrolled surveillance environments. Future work will explore latent space diffusion to reduce computational costs.关键词：gait recognition;spatio-temporal diffusion model;visibility awareness;gait silhouette restoration;three-dimensional convolutional neural network32|71|0更新时间：2026-06-22 -
摘要：Aerial image restoration is an important low-level vision task for improving the quality and reliability of remote sensing and aerospace imagery. Compared with general natural image restoration， aerial image restoration faces more complex imaging conditions caused by long-distance observation， platform motion， atmospheric interference， illumination variation， and sensor limitations. These factors often lead to diverse and coupled degradations， which not only reduce visual quality but also affect downstream tasks such as object detection， semantic segmentation， change detection， scene understanding， and autonomous perception. Therefore， robust aerial image restoration has become a key problem in both computer vision and aerospace information processing. In recent years， image restoration has evolved from task-specific models for individual degradations to unified restoration frameworks and large model-driven paradigms. Existing surveys mainly focus on general natural image restoration， specific low-level vision tasks， or individual degradation problems. In contrast， this survey focuses on aerial image restoration from the perspective of large model empowerment. It emphasizes the unique imaging characteristics and practical requirements of aerial scenarios， and systematically reviews the technical evolution from single-degradation restoration to unified modeling， and further to large model-driven intelligent restoration. In particular， this paper highlights how large-scale pretraining， multimodal representation learning， physical priors， and agent-based decision mechanisms can contribute to robust aerial image restoration under complex degradation conditions. Specifically， this paper first reviews representative restoration methods for typical single-degradation tasks and summarizes their basic modeling ideas， advantages， and limitations. Traditional methods usually rely on handcrafted priors， optimization models， and physical assumptions， which provide clear interpretability but are often limited in complex real-world scenarios. Deep learning-based methods improve restoration performance by learning nonlinear mappings from degraded images to clean images， but many of them are still designed for specific degradation types and may lack generalization ability when facing unknown or mixed degradations in aerial scenes. Then， this paper discusses unified image restoration methods for complex aerial imaging scenarios. In practical applications， aerial image degradation is rarely caused by a single factor. Different degradation factors may coexist and interact with each other， resulting in complex coupled degradation and spatially non-uniform degradation distributions. Unified restoration methods aim to handle multiple degradations within a single framework. Existing studies mainly explore multi-degradation modeling， prompt-based restoration， and dynamic routing or expert-based frameworks. These methods improve the adaptability of restoration models by learning shared representations， introducing degradation-aware prompts， or dynamically selecting suitable restoration paths. However， they still face challenges in explicitly modeling degradation mechanisms， distinguishing different degradation patterns， and dealing with spatially varying degradations. On this basis， this survey focuses on large model-driven aerial image restoration. Vision foundation models provide transferable visual representations and structural priors through large-scale pretraining， which can improve restoration robustness under data-limited and cross-scene conditions. Multimodal large models introduce semantic and cross-modal information into the restoration process， enabling models to better understand complex scenes and provide high-level guidance for structure recovery. The combination of physical models and large models further provides a promising hybrid paradigm， where physical priors derived from imaging mechanisms constrain the restoration process， while large models provide strong representation and generation capabilities. Agent-based restoration methods extend image restoration from a static mapping process to a dynamic decision-making process， in which degradation analysis， strategy planning， model selection， feedback evaluation， and iterative optimization can be integrated into a unified restoration pipeline. Furthermore， this paper analyzes the main challenges faced by existing aerial image restoration methods. First， the modeling of coupled degradations remains insufficient， especially when multiple degradation factors interact in complex imaging environments. Second， the collaboration between semantic priors and low-level visual features is still limited， making it difficult to achieve reliable structure recovery under severe degradation. Third， the integration of physical knowledge and data-driven large models remains underexplored， and balancing physical consistency with generative flexibility is still an open problem. Fourth， most existing methods lack dynamic adaptive mechanisms and cannot adjust restoration strategies according to image content， degradation state， or task requirements. Finally， the high computational cost of large model-driven methods restricts their deployment on airborne platforms， edge devices， and real-time remote sensing systems. Overall， aerial image restoration is moving from degradation-specific and pixel-level processing toward unified modeling， multimodal collaboration， physical prior-guided restoration， and intelligent decision-making. This survey aims to provide a systematic reference for researchers by clarifying the relationship between traditional restoration methods， unified restoration frameworks， and large model-driven paradigms. It also discusses future research directions， including multi-source data construction， unified representation learning， multimodal and physical prior-guided restoration， adaptive decision-making， and efficient deployment， with the goal of promoting robust， interpretable， and intelligent aerial image restoration systems.关键词：Aerial Image Restoration;computer vision;Prompt-based Learning;Unified Modeling;Multi-source Fusion;Agent-based Framework36|95|0更新时间：2026-06-22 -
摘要：Robust matching of multi-temporal and multi-modal remote sensing images is a fundamental prerequisite for image registration， change detection， target localization， image fusion， and three-dimensional reconstruction in aerial and satellite remote sensing applications. However， stable correspondence estimation remains highly challenging when the matching process is simultaneously affected by significant spectral discrepancy， complex seasonal variation， repetitive textures， weakly textured regions， and large appearance inconsistency across acquisition times. In visible-to-near-infrared remote sensing scenarios， the same geographic area may present markedly different reflectance patterns， local structures， and texture distributions under different seasons and spectral bands， making traditional hand-crafted descriptors and many existing deep matching methods unreliable when they depend mainly on local photometric similarity or low-level geometric consistency. To address these issues， this paper studies the problem from the joint perspectives of benchmark construction and semantically enhanced matching design. A standardized benchmark is constructed based on the public Seasonal Contrast （SeCo） dataset by organizing cross-temporal visible and near-infrared image pairs and introducing random homography perturbations to establish reliable geometric supervision under a unified evaluation protocol. The resulting benchmark contains 15，000 image pairs， including 12，000 training pairs and 3，000 testing pairs， and provides a reproducible experimental setting for quantitative comparison under severe temporal and spectral changes. In addition， the protocol evaluates models using homography reprojection error and multi-threshold Matching Homography Accuracy （MHA）， enabling a unified assessment of geometric precision and matching robustness. Building upon this benchmark， a semantically enhanced cross-modal feature matching framework is proposed based on XoFTR， a hierarchical coarse-to-fine architecture for cross-modal correspondence estimation. The proposed method injects dense semantic priors extracted by a frozen DINOv3 model into the coarse matching stage of XoFTR through a lightweight Semantic Boosting Module， enabling joint modeling of high-level semantic consistency and geometric correspondence. For each modality branch， the DINOv3 features are aligned to the coarse feature resolution and fused with geometric tokens， allowing the model to emphasize semantically stable regions while preserving local geometric sensitivity. This design is particularly important because the coarse stage determines the candidate regions explored by subsequent fine matching； once the global correspondence field is biased by unstable appearance cues， later local refinement is unlikely to recover correct matches. The overall pipeline follows a progressive coarse-to-fine paradigm in which multi-scale geometric features are first extracted from visible and near-infrared inputs， coarse tokens are enhanced by DINOv3-derived semantic priors， global correspondences are established through transformer-based context interaction， and local fine-level refinement is subsequently performed to improve correspondence precision. Training is conducted in a single-stage in-domain optimization manner， with the DINOv3 encoder kept frozen throughout optimization and the XoFTR backbone together with the semantic boosting components jointly learned. This strategy allows the method to leverage semantic priors from large vision models in a lightweight and reproducible manner without introducing excessive trainable parameters or unstable large-model adaptation behavior. Extensive experiments under the unified SeCo evaluation protocol demonstrate that the proposed method outperforms the XoFTR baselines across all MHA thresholds. On the constructed benchmark， the proposed model achieves 33.03%， 37.03%， and 39.17% in MHA@3， MHA@5， and MHA@7， respectively， outperforming the original pretrained XoFTR model by 12.91， 7.68， and 4.62 percentage points and surpassing the direct in-domain fine-tuning baseline by 14.64， 15.85， and 16.61 percentage points. Qualitative comparisons further show that the proposed framework suppresses cross-region mismatches and yields denser correct correspondences in scenes with strong temporal appearance shifts and cross-modal ambiguity. These results indicate that merely adapting a geometric matching network to the target domain is insufficient for robust multi-temporal multi-modal remote sensing matching， whereas integrating high-level semantic priors from a vision foundation model can significantly enhance the stability of global correspondence establishment and improve final geometric estimation accuracy. The study shows that high-level semantic representations learned by large-scale vision models can effectively alleviate the representation discrepancy caused by temporal evolution and spectral inconsistency， and can be incorporated into a hierarchical cross-modal matching framework in a computationally efficient and practically effective manner， thereby providing a lightweight and reproducible semantic injection route for large-model-empowered high-precision remote sensing image matching. The source code is publicly available athttps：//github.com/heng-shan/Dino_ft.关键词：multi-temporal remote sensing;multi-modal image matching;vision foundation models;semantic enhancement;XoFTR;DINO35|97|0更新时间：2026-06-22 -
摘要：Unmanned aerial vehicle vision-language navigation stands as a core research direction of aerial embodied intelligence， which combines computer vision， natural language processing， robot control and unmanned system technology. It endows unmanned aerial vehicles with the ability to understand human natural language instructions and perform autonomous navigation， path planning and task execution in unstructured， GPS-denied and dynamically changing three-dimensional environments. Different from traditional unmanned aerial vehicle navigation methods that rely on satellite positioning， inertial navigation or manual waypoint setting， unmanned aerial vehicle vision-language navigation establishes a direct mapping from high-level semantic instructions to low-level continuous flight actions， so that unmanned aerial vehicles can complete complex tasks such as target search， obstacle avoidance， long-distance cruising and scene reasoning only through visual observation and language understanding. This technology effectively breaks the limitations of traditional navigation in semantic interaction and environmental adaptability， and has important theoretical value and engineering application prospects in low-altitude economy， urban security， emergency rescue， industrial inspection and precision agriculture. In recent years， with the rapid development of multimodal large models， transformer architectures and embodied artificial intelligence， unmanned aerial vehicle vision-language navigation has gradually evolved from early modular pipelines to end-to-end learning frameworks， and then to highly interpretable reasoning systems driven by large language models and vision-language models. However， compared with ground robot vision-language navigation， unmanned aerial vehicle vision-language navigation still faces many unique challenges caused by aerial movement characteristics. First， unmanned aerial vehicles move with six degrees of freedom， and frequent changes in height， pitch， yaw and roll lead to unstable visual input， large differences in object scales and serious geometric distortion， which increases the difficulty of cross-modal alignment between vision and language. Second， three-dimensional spatial topology is more complex， and spatial prepositions such as “above”， “between”， “along” and “across” require stronger geometric reasoning and spatial awareness， which cannot be satisfied by two-dimensional image matching alone. Third， long-range navigation tasks bring problems such as massive visual information， cumulative positioning errors and semantic forgetting， which put forward higher requirements for efficient memory mechanism and environmental representation. Fourth， the discrete semantic decision space is difficult to match with the continuous physical control space of unmanned aerial vehicles， resulting in a large gap between simulation training and real-world deployment. Aiming at the above problems， this paper systematically summarizes the research progress of unmanned aerial vehicle vision-language navigation from the perspective of embodied intelligence. First， three types of simulation platforms are reviewed： general robot simulation platforms， simulation platforms based on real scene reconstruction， and large-scale virtual simulation platforms built by game engines. These platforms provide safe， low-cost and high-efficiency test environments for algorithm verification， data generation and model training， and effectively narrow the domain gap between simulation and reality. Second， mainstream datasets are compared and analyzed from the dimensions of scene scale， instruction complexity， action space definition and environmental authenticity. The development of datasets reflects the trend of unmanned aerial vehicle vision-language navigation moving from indoor structured scenes to outdoor large-scale urban scenes， from short simple instructions to long sequential reasoning instructions， and from discrete actions to continuous six-degree-of-freedom control. Third， the core technical framework is divided into four modules： perception representation， reasoning paradigm， memory storage and embodied control. The perception representation focuses on visual feature extraction， geometric alignment， multimodal fusion and world model construction. The reasoning paradigm mainly includes cross-modal attention mechanism， transformer-based pretraining and large-model-driven chain-of-thought reasoning. The memory storage evolves from implicit temporal memory to explicit semantic maps and topological graphs. The embodied control develops from modular trajectory planning to end-to-end vision-language-action generation with safety constraints. In terms of practical applications， unmanned aerial vehicle vision-language navigation has been widely explored in urban infrastructure inspection， disaster emergency search and rescue， intelligent logistics distribution and precision agricultural monitoring. It can reduce manual participation， improve operation efficiency and enhance safety in high-risk， high-complexity and high-efficiency-demanding scenarios. However， there are still key bottlenecks restricting large-scale deployment， such as simulation-to-real transfer， generalization in unseen environments， real-time performance of onboard computing， safety and stability in dynamic environments， and collaborative navigation of multiple unmanned aerial vehicles. Finally， this paper prospects the future development trends of unmanned aerial vehicle vision-language navigation， including stronger world model and predictive reasoning， better generalization and robustness based on multimodal large models， safer and more reliable physical control， more efficient human-machine interaction and collaborative intelligence， as well as deeper integration with low-altitude digital economy and unmanned system ecology. This review aims to provide a complete and clear technical route for researchers in the fields of embodied intelligence， computer vision， natural language processing and unmanned aerial vehicle systems， promote the breakthrough of key technologies， and accelerate the practical and industrialization process of unmanned aerial vehicle vision-language navigation.关键词：unmanned aerial vehicle;vision-language navigation;embodied intelligence;Cross-Modal Alignment;large model67|98|0更新时间：2026-06-22 -
摘要：With the proliferation of multi-source sensors and the rapid progress of multimodal data fusion and intelligent interpretation techniques， aerial remote sensing is evolving from traditional single-modal perception toward multimodal perception and understanding. This progress provides significant potential for intelligent monitoring and decision-making in precision agriculture， urban environmental monitoring， ecological protection， and natural disaster assessment. With the rapid advancement of Earth observation capabilities， multi-source remote sensing data—including optical imagery， synthetic aperture radar （SAR）， and hyperspectral imagery—are increasingly characterized by high spatial resolution， multi-temporal coverage， and multi-dimensional richness， offering a solid foundation for fine-grained land-cover identification and dynamic change analysis. Nevertheless， traditional remote sensing interpretation methods， which rely primarily on single-source data and single-task learning models， struggle to cope with the complexities of real-world scenarios， where object scales vary dramatically， semantic layers are rich and entangled， and spatiotemporal heterogeneity is strong. Such methods cannot adequately balance holistic scene semantics with local fine-grained details， limiting their capacity for high-level semantic understanding and comprehensive decision-making. Compared with satellite remote sensing， aerial platforms—typically employing unmanned aerial vehicles （UAVs） and low-altitude aircraft—offer higher spatial resolution and greater observational flexibility， enabling tasks such as fine urban modeling， small-object recognition， and emergency monitoring. However， these very advantages introduce severe scale variations， highly complex backgrounds， and inconsistent imaging conditions， which make it extremely difficult for conventional models to capture both global scene semantics and subtle local textures simultaneously. Recent advances in deep learning and artificial intelligence have transformed remote sensing interpretation from manual inspection toward automation and intelligence， achieving notable results in scene classification， object detection， and semantic segmentation. Yet single-task and single-modality paradigms remain inadequate for fully exploiting complementary information in heterogeneous multi-source data， falling short of supporting the high-level semantic comprehension and integrated decision-making required for complex applications. The emergence of multimodal large models provides a transformative pathway for intelligent aerial remote sensing interpretation. By integrating visual encoders with large language models through instruction tuning and cross-modal alignment， these models can jointly perform image understanding and linguistic reasoning， achieving breakthroughs in visual-language fusion， cross-modal reasoning， and task-instruction-guided analysis. Within the remote sensing community， researchers have begun to construct large-scale image-text datasets to train vision-language models and employ linguistic guidance to enhance the comprehension of complex land-cover semantics， marking a critical shift from purely visual modeling toward semantically augmented modeling and laying the groundwork for the subsequent development of multimodal large models. A systematic review reveals that the evolution of remote sensing large models has followed a clear paradigm progression： from early approaches that relied solely on single visual modalities， through vision-language models that introduced textual semantics but were still confined to specific tasks， to the current stage in which unified multimodal large models enable cross-task collaboration and complex reasoning. Despite these promising advances， applying multimodal large models to aerial remote sensing still encounters a series of formidable bottlenecks. The substantial disparities in spatiotemporal references and radiometric properties among heterogeneous data sources such as optical， SAR， and hyperspectral imagery render cross-modal alignment extremely difficult， often giving rise to semantic misalignment and information loss. The fine-grained spatial structures and multi-scale objects inherent in aerial imagery impose stringent demands on spatial cognition and multi-scale reasoning， yet general-purpose large models frequently lack sufficient domain-specific spatial priors to accurately capture geometric and structural relationships. High-resolution scenes impose massive computational and storage burdens， making real-time inference on edge devices highly challenging and failing to meet the stringent timeliness and generalization requirements of time-sensitive tasks such as disaster response and UAV-based inspection. Moreover， the black-box nature of model decision-making leads to insufficient interpretability and trustworthiness， hindering deployment in safety-critical scenarios， while data privacy concerns have become increasingly prominent within distributed collaborative learning frameworks. In response to the outlined challenges， this paper systematically reviews recent advances in multimodal data fusion and large model technologies for aerial remote sensing， tracing the shift from single-source to multimodal integration. Driven by large models， remote sensing interpretation has gradually evolved from low-level perception to cross-modal reasoning and semantic understanding. Nevertheless， substantial challenges still remain in multimodal alignment， spatial cognition， task reliability， and practical deployment in real-world scenarios. We analyze cross-sensor fusion， highlighting that disparities in geometry， radiometry， and acquisition timing among optical， SAR， and LiDAR data severely impede cross-modal semantic alignment； despite advances in vision-language pretraining， high resolution and complex backgrounds often degrade alignment accuracy， necessitating fine-grained multimodal representations through geometric correction， radiometric normalization， and spatiotemporal consistency modeling. For multi-scale spatial reasoning， we emphasize that complex tasks require holistic understanding of structural relations and spatial distributions； existing models partially strengthen spatial reasoning via region-based interaction and spatial question answering， but a unified framework for global spatial relation modeling is needed to elevate task awareness and multilevel reasoning. Generative AI for few-shot interpretation shows potential to alleviate annotation scarcity and support time-critical missions such as disaster response. Regarding trustworthiness and interpretability， hallucination in large vision-language models is exacerbated in remote sensing by peculiar data distributions and insufficient instruction data， thus requiring domain knowledge， task constraints， and dedicated evaluation systems to enforce reasoning consistency and reliable high-stakes decisions. For efficient edge deployment and privacy， high-resolution large-format data impose heavy computational and storage burdens； lightweight networks continue to emerge， yet trade-offs among real-time performance， stability， and complex scene adaptability remain， making model compression， inference optimization， and hardware-software co-design critical， while federated learning offers a privacy-preserving mechanism for multi-source data integration. In summary， addressing these challenges is pivotal not only for advancing theoretical understanding of cross-modal semantic reasoning and multi-scale spatial cognition but also for enabling the practical deployment of intelligent aerial remote sensing systems in high-stakes， real-world applications.关键词：aerial remote sensing;multimodal fusion;visual-language model;large model;semantic comprehension63|96|0更新时间：2026-06-16 -
摘要：The rational and effective utilization of multi-source information will expand the spatial and temporal coverage of measurements， fully excavate the target feature information contained in various sensors， greatly reduce the ambiguity of information， and improve detection performance. Faced with multi-modal target data from different types， characteristics， and perception methods， for example， in airborne multi-sensor target tracking systems， various types of sensors with different working attributes such as visible light images， infrared images， optical sensors， microwave radar， and lidar are often involved， presenting measurement information of the target from different perspectives. The point cloud information obtained by lidar can not only achieve high-precision ranging but also provide more accurate spatial information due to different reflectivities of different objects. However， it is susceptible to noise generated by complex environments and lacks semantic information. Compared to point cloud information， target image information provides rich surface textures and contextual semantic information that can more accurately restore the appearance and structure of objects， but lacks depth information. With the rapid development of sensor networks， information perception methods， and big data processing technologies， faced with information perception from different types， characteristics， and methods， as well as its characteristics of multi-domain， multi-modal， ambiguity， and incomplete data association， multimodal information fusion technology has received increasing attention due to its advantages of human-like automatic perception and powerful comprehensive reasoning capabilities. The paper first introduces the significance and necessity of multi-modal information fusion. Furthermore， at the fusion architecture level， it outlines the evolution from early fusion， late fusion， to hybrid fusion， revealing the trade-off relationship between information retention and computational efficiency in each architecture. At the same time， it elaborates on three cutting-edge multi-modal fusion methods： attention-based interaction modeling， semantic alignment based on contrastive learning， and generative fusion with large language models as the carrier. Then， it introduces typical multi-modal datasets such as the COCO dataset， LAION-400M dataset， Visual Genome dataset， MS COCO Captions dataset， Conceptual Captions dataset， Flickr30k dataset， and aviation-related datasets， as well as their corresponding application fields. Moreover， it further explores the typical military applications of multimodal fusion， including battlefield situation awareness， multi-source intelligence analysis， and unmanned system collaboration. In addition， it points out that with the maturity of contrastive learning and multi-modal pre-training， high-quality single-modal representations are no longer a bottleneck. The research focus has shifted from how to map heterogeneous modalities to a unified space in the early stage to how to design interaction mechanisms that can capture complex， dynamic， and even contradictory relationships between modalities based on representation alignment. Based on fusion architecture， fusion models， and computational costs， it proposes three development directions for multimodal information fusion. The choice of multimodal fusion architecture directly impacts the degree of information retention， computational efficiency， and model interpretability. Based on the stage at which information fusion occurs， existing methods can be categorized into early fusion， late fusion， hybrid fusion. Early fusion， also known as data-level fusion， involves the integration of multi-source information prior to or at the shallowest level of modality-specific feature extraction. This process preserves the richest original information and multimodal interaction details. Typical implementations include multimodal concatenation and multi-view encoding. Taking the visual-language model as an example， early fusion concatenates image patch embeddings and text word embeddings into a unified sequence， which is then fed into a joint encoder for processing. The advantage of this architecture lies in the ability to establish multimodal associations at a shallow level， facilitating the capture of fine-grained modality interactions. However， early fusion faces a severe dimensionality catastrophe problem. When the number of modalities increases or the feature dimensions of each modality are too high， the joint representation space grows exponentially. At the same time， due to the different noise characteristics of different modalities， the noise of some modalities may be amplified during early fusion， resulting in poor representation quality. Late fusion adopts a diametrically opposite strategy： each modality independently performs feature extraction and task prediction， with integration， weighted average， and meta-learner only at the final decision-making layer. In this "divide and conquer" design， the modal branches can be trained in parallel， achieving high computational efficiency； meanwhile， overfitting or noise in one modality is less likely to affect other modalities； in addition， in scenarios where some modalities are missing， the remaining branches can still work normally， demonstrating strong system robustness. However， late fusion may lead to higher consumption of computational resources as it requires training independent models for each modality. At the same time， independent models for each modality struggle to capture low-level interactions between them， making it difficult to model simple fusion at the decision-making level. Furthermore， data from different modalities may face alignment issues in time or space， such as the synchronization of video frames and audio signals. Hybrid fusion introduces cross-modal interaction in the middle layers of the network while maintaining modality-specific processing paths， aiming to combine the strengths of both. Its typical application is to use modality-independent encoders at the bottom layer， introduce a cross-attention module in the middle layer to achieve feature-level interaction， and separate again at the top layer to preserve modality-specific information. A deeper evolutionary direction is dynamic fusion， which allows the network to autonomously decide where and with what intensity to fuse information from various modalities. For example， a gated mechanism-based visual image and LiDAR multimodal fusion network dynamically adjusts the weights of information from each modality based on input data. Specifically， it places more trust in visual images under strong lighting conditions， while giving higher weight to LiDAR point cloud information under low-light conditions. The choice of fusion architecture does not have an "optimal solution"， but rather depends on the function of task characteristics and resource constraints. Generally speaking， when there are fine-grained， location-related interactions between modalities （such as image-text alignment）， early fusion is better； when each modality can complete predictions independently and some modalities are prone to missing， late fusion is more robust； dynamic fusion strikes a good balance between performance and robustness. Looking back at the development of multimodal fusion， the bottleneck of fusion is shifting from "representation" to "interaction". With the maturity of contrastive learning and multimodal pre-training， high-quality single-modality representation is no longer a bottleneck. The research focus has shifted from how to map heterogeneous modalities into a unified space in the early stage to how to design interaction mechanisms that can capture complex， dynamic， and even contradictory relationships between modalities on the basis of representation alignment. The fusion architecture is evolving from "static design" to "dynamic adaptation". In the real world， the correlation and reliability of modalities change dynamically with the environment， and the limitations of fixed fusion strategies are becoming increasingly evident. In military confrontations， electronic jamming may render specific sensors inoperative. Therefore， it is necessary to dynamically adjust the activation state of modalities and fusion weights based on input content and task context to enhance the robustness of the fusion system. The causal fusion model holds promise for breaking through the current limitations of relational learning. Most existing methods focus on learning statistical correlations between modalities， rather than causal relationships. This leads to two issues： firstly， the model is prone to learning spurious correlations， and secondly， it struggles to generalize to environments beyond the training distribution. Therefore， introducing causal inference tools can enhance the fusion model's adversarial robustness and environmental transferability. In addition， a new multimodal fusion mechanism is designed. Redundant information from each modality may cause computational waste， and the corresponding noise information may affect fusion performance. How to utilize information theory techniques to identify and retain complementary information while suppressing redundancy and noise is a new development direction for multimodal fusion technology.关键词：multimodal information fusion;multimodal large language model;attention mechanism;contrastive learning;generative fusion75|99|0更新时间：2026-06-16
- Postal code：100190
- Tel：010-58887035/58887030/58887418 Email：jig@aircas.ac.cn
- Technical support is provided by Beijing Founder electronics co., LTD 京ICP备05080539号-4
京公网安备11010802024621 - It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
- Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
0