最新刊期
-
Construction and evaluation of a dataset for completing missing nighttime satellite band information AI导读
摘要：ObjectiveGeostationary meteorological satellites provide observations of clouds， the atmosphere， and Earth’s surface， but their visible and near-infrared （VIS–NIR） reflectance channels cannot provide usable measurements at night because reflected solar shortwave radiation is unavailable. Consequently， true-color imagery （TCI） and other products derived from these channels have gaps that hinder continuous monitoring of clouds， fog， and severe weather. Missing reflectance observations also affect products describing aerosols， cloud type， cloud optical thickness， and shortwave radiation， extending the importance of reconstruction beyond visual interpretation. Existing nighttime image-generation studies have generally targeted only one or a few visible bands， while public paired datasets spanning multiple reflective bands and diverse scenes remain limited. This study develops a large-scale benchmark， VIS-NIR-70K， for learning the relationship between infrared observations and missing VIS–NIR information. It also evaluates whether representative generative models can reconstruct six reflective bands from inputs available at night， thereby establishing a basis for comparing cross-modal generation methods and examining their potential for nighttime satellite applications.MethodsVIS-NIR-70K was constructed from the Advanced Himawari Imager aboard the Himawari-8/9 satellites. Daytime observations provide simultaneous multi-band infrared brightness temperatures （MIBT） and measured VIS–NIR reflectances， which form paired training and evaluation samples even though nighttime reflectance ground truth is unavailable. The six prediction targets are AHI albedo bands 1~6， spanning visible， near-infrared， and shortwave-infrared wavelengths. Conditional inputs consist of brightness-temperature bands 7~16： one mid-wave infrared band， three water-vapor bands， and six additional thermal infrared bands. Satellite zenith angle （SAZ） and a land-sea mask （Landsea） supply observation-geometry and underlying-surface information. Solar zenith angle （SOZ） is used during preprocessing to exclude poorly illuminated regions with SOZ greater than 85°， rather than serving as a model input. Band-specific extreme values are clipped using the first and 99th percentiles， and coregistered observations are divided into nonoverlapping 512×512-pixel patches； 256×256-pixel versions are used for model training and inference. Samples are separated by more than one hour. The training set contains 62，290 paired patches derived from 690 observation times in 2016~2024， whereas the independent test set contains 5，446 pairs from 80 observation times in December 2025~January 2026. In total， 67，736 paired patches yield 1，219，248 individual multi-band images. Cloudy patches account for approximately 23% of the dataset， and both land and ocean are represented. The year-separated split reduces leakage from temporally adjacent scenes， although the test period is concentrated in winter. A linear baseline， CycleGAN， Pix2Pix， three DiT variants， a DDPM， and RefDiff are evaluated under a shared data split. Assessment combines pixel and radiometric errors， including mean squared error （MSE）， root mean squared error， and mean bias error， with peak signal-to-noise ratio， structural similarity index measure （SSIM）， correlation coefficient， spectral angle mapper， Fréchet Inception Distance， and Kernel Inception Distance. Visual comparisons examine individual bands， TCI， clouds， sea fog， a typhoon， and nighttime full-disk reconstructions.ResultsThe paired-data experiments show that MIBT contains useful information for estimating VIS–NIR reflectance， although performance varies across target bands and scene types. Diffusion-based approaches generally outperform the adversarial models on structural similarity and perceptual distribution measures. The unpaired CycleGAN has difficulty preserving local structures， while paired Pix2Pix improves structural fidelity but tends to smooth complex cloud textures. DiT variants represent large-scale patterns and offer faster inference than the image-space diffusion approaches， yet their results retain less fine detail； increasing DiT model size produces only modest changes in reconstruction quality. DDPM and RefDiff， which perform diffusion in the original image space， recover cloud boundaries， spatial patterns， and subtle brightness variation more effectively. Across all six predicted bands， DDPM attains an average SSIM of 0.864， an MSE of 0.003， an FID of 11.178， and a KID of 0.007. RefDiff achieves a slightly higher average peak signal-to-noise ratio of 29.143 and a lower spectral angle mapper value of 0.0579， illustrating that no single model leads on every criterion. These differences show why pixel accuracy， feature-space similarity， and multi-band spectral agreement should be considered together when comparing generators for remote sensing. For TCI assembled from the three visible channels， DDPM obtains an average SSIM of 0.881 and an MSE of 0.002. Qualitative results show that the original-image-space diffusion models better preserve low-cloud textures and the organization of the typhoon-eye region. Experiments across cloudy and clear skies， land， ocean， and sea-fog scenes also indicate broad scene coverage. Nevertheless， albedo bands 4 and 5 remain harder to reconstruct than the visible channels， reflecting information about surface reflectance and cloud microphysics that infrared brightness temperatures do not fully determine. Input ablations identify the complete set of bands 7~16 as the main source of performance improvement； adding SAZ and Landsea together provides complementary constraints and improves overall stability. A nighttime full-disk example illustrates spatially continuous generated bands and a plausible TCI across the day–night boundary. An approximate scaling calculation suggests that DDPM could generate a full disk in about 57 seconds， within AHI’s ten-minute observation interval， although this is not a measured deployment time. Because concurrent nighttime VIS-NIR measurements are unavailable， this example supports qualitative interpretation rather than direct quantitative validation.ConclusionVIS-NIR-70K provides a multi-band， multi-scene resource for evaluating nighttime missing-band completion using daytime paired observations. The benchmark reveals the value of complete infrared inputs and the complementary roles of observation geometry and surface-type information. Original-image-space diffusion yields strong structural and spectral reconstruction， especially for complex clouds， while latent-space approaches offer a useful quality-efficiency tradeoff. The generated nighttime imagery could support continuous meteorological analysis， but daytime-to-nighttime domain shifts， limited independent seasonal testing， and the lack of nighttime reflectance ground truth constrain present claims about operational accuracy. Future work should incorporate physical knowledge of radiation and clouds， develop task-specific evaluation measures， and reduce diffusion sampling costs to improve the reliability and practicality of multi-band generation. Dataset access link： （ScienceDB：https：//doi.org/10.57760/sciencedb.j00240.00227）.关键词：multiband satellite imagery;missing-band completion;multiband dataset;VIS-NIR generation;generative algorithms;Himawari-8/964|123|0更新时间：2026-11-02 -
摘要：ObjectiveAccurate and efficient disparity estimation is essential for three-dimensional scene understanding， autonomous driving， robotic perception， and image-based reconstruction. Stereo matching estimates dense correspondences between rectified left and right images， but reliable disparity estimation remains difficult when local appearance provides weak or ambiguous evidence. High-accuracy stereo networks often rely on high-dimensional cost volumes， computationally intensive aggregation， or large recurrent states. These designs improve the representation of candidate matches but also increase memory use and inference time. Compact stereo networks reduce computation by lowering feature dimensionality and simplifying cost aggregation. Excessive compression， however， can weaken the discrimination of similar disparity hypotheses， especially in weakly textured regions， occlusions， and areas near object boundaries. Moreover， when an iterative estimator repeatedly consults an unchanged matching representation， early disparity errors can affect later updates. To balance estimation quality and computational efficiency， this paper proposes FLISNet， a fast iterative stereo matching network with low-dimensional cost volume feedback correction. The method combines compact matching evidence， disparity-guided recursive correction， and multi-scale structural compensation so that intermediate predictions can contribute to subsequent cost refinement.MethodFLISNet takes a rectified stereo pair as input and uses a shared MobileNetV2 backbone with a U-Net-style reconstruction structure to extract multi-scale matching features. A separate context branch provides features at full， half， and quarter input resolutions. The compact cost representation combines low-dimensional feature projection， channel weighting， and group-wise correlation. Specifically， the left and right matching features are projected into a lower-dimensional space， and learnable channel-wise weights shared across spatial positions emphasize informative responses before correlation is computed within feature groups. This procedure limits cost volume size while retaining discriminative correspondence cues. Recursive feedback correction on the compact cost representation operates at one-quarter of the input resolution. The initial disparity at this resolution is set to zero. A convolutional gated recurrent unit predicts successive disparity residuals， and each intermediate estimate is encoded into a spatial feedback feature. A gate generated from this feature modulates the low-dimensional cost responses through a residual connection. The updated cost responses are then used for subsequent recurrent updates， allowing the current disparity estimate to influence the matching representation itself. The standard FLISNet performs eight feedback updates， while FLISNet+ performs eighteen updates for higher accuracy. Following the final update， the gating features are resized to match the spatial resolutions of the structural compensation stages. The network combines the final quarter-resolution disparity estimate with context and gating features， then progressively restores details at one-quarter， one-half， and full resolution relative to the input image. Three output stages are jointly supervised with loss weights of 0.5， 0.7， and 1.0. This coarse-to-fine compensation process is designed to improve local structures and disparity discontinuities without introducing a large cost volume.ResultExperiments on Scene Flow， KITTI 2012， KITTI 2015， ETH3D， and Middlebury examine matching accuracy， inference efficiency， and performance across different scene types. On Scene Flow， the standard FLISNet achieves an endpoint error （EPE） of 0.68 pixels with an inference time of 32 ms. CoEx obtains 0.69 pixels and 33 ms in the same comparison. FLISNet+ reduces the Scene Flow EPE to 0.56 pixels with a runtime of 56 ms. This EPE is 5.1% and 6.7% lower than those of Fast-ACVNet+ and LightStereo， respectively， in the same comparison. Increasing the recursive update count from six to twenty reduces EPE from 0.73 to 0.55 pixels， while inference time increases from 27 to 62 ms. The additional EPE reduction from eighteen to twenty updates is only 0.01 pixels， supporting eighteen updates as the accuracy-enhanced setting. On KITTI 2012， FLISNet+ obtains a 3-all error of 1.77% （all pixels with valid ground truth， 3-pixel threshold）. Compared with the 2.41% reported for LBSM-L-Refine in the same table， this represents a relative reduction of 26.6%. On ETH3D， FLISNet+ achieves a Bad 2.0 error of 0.54%， the lowest among the methods listed in the comparison. In the zero-shot setting， the standard FLISNet is trained only on Scene Flow and evaluated directly on KITTI 2012， KITTI 2015， and Middlebury. The zero-shot comparison indicates that FLISNet can be applied to unseen datasets without target-domain fine-tuning and still produce relatively stable disparity estimates. Ablation experiments evaluate the role of each component under the same training and evaluation conditions. Replacing the compact cost representation with a standard correlation cost volume increases Scene Flow EPE from 0.68 to 0.72 pixels. Replacing residual feedback modulation with a two-dimensional lookup raises the EPE to 0.74 pixels， while replacing multi-scale compensation with direct upsampling raises it to 0.79 pixels. Using lower-resolution update scales increases the EPE to 0.78 pixels and the inference time to 37 ms. Qualitative comparisons show clearer object boundaries and improved continuity in weakly textured regions. Inference times in this study were measured on a single NVIDIA RTX 3090 GPU with batch size 1， 384 × 1248 inputs， and FP32 computation. Each measured value is the average over 200 runs after 50 warm-up runs； runtimes quoted from original papers are marked separately.ConclusionFLISNet combines low-dimensional matching features with disparity-guided residual modulation and multi-scale structural compensation. The experiments show that the standard configuration maintains competitive disparity accuracy with modest inference time， while increasing the update count improves accuracy when a higher runtime is acceptable. The ablation results support the contributions of compact cost representation， feedback correction， and coarse-to-fine compensation. Future work will investigate model compression， knowledge distillation， and low-cost feature adaptation， along with adaptive matching methods for visually degraded environments.关键词：stereo matching;lightweight network;low-dimensional cost volume;iterative optimization;disparity refinement2|0|0更新时间：2026-11-02 -
摘要：ObjectiveDiffusion inversion super-resolution （InvSR） reconstructs images with only a few reverse sampling steps by estimating the noise required for intermediate diffusion states. However， the original noise prediction network lacks explicit preservation of local edges and fine-grained textures during high-resolution feature compression. In addition， the bottleneck layer does not jointly model cross-region structural context and local texture continuity， while the fixed sampling path is difficult to adapt to inputs with different degradation characteristics. To improve the detail preservation capability and sampling-path adaptability of diffusion inversion super-resolution， a detail-preserving noise prediction and adaptive sampling-path scheduling method for InvSR is proposed.MethodThe proposed framework jointly modifies the noise-prediction network and the sampling policy. First， a high-frequency preserving downsampling module （HPDM） replaces the attention-heavy first downsampling stage. A time-conditioned residual block retains the modulation of feature responses by the diffusion timestep. Three depthwise-convolution branches with 3×3， 5×5， and dilated 3×3 kernels encode local texture， slightly wider edge context， and expanded spatial context， respectively. Their outputs are concatenated， fused by a 1×1 convolution， and added to the input feature. PixelUnshuffle followed by a 1×1 projection then rearranges each 2×2 neighborhood into the channel dimension before compression， reducing irreversible loss of local ordering information. A linear spatial-attention block is retained only in the lower-resolution stage， where long-range relations can be modeled at lower cost. Second， a structure-texture dual-branch fusion module （STDFM） is inserted into the semantic bottleneck. Its structure branch uses linear spatial attention to capture global contours and cross-region dependencies， whereas its texture branch uses multi-scale depthwise convolutions to represent local gradients， orientation， and neighborhood continuity. A time-aware gate predicts the relative contribution of the two branches from the input content and timestep embedding， enabling the bottleneck to strengthen structural constraints when global evidence is weak and to emphasize local texture when fine detail is deficient. Third， a sampling path prediction module （SPPM） converts path selection into a sixteen-class degradation-aware classification problem. The candidate set contains sixteen timestep combinations covering one to five-step sampling. During offline label construction， every candidate is evaluated for each training sample using direction-aligned and normalized fidelity， structural， perceptual， and no-reference quality measures. Their weighted restoration utility determines the supervisory path label. A tiny depthwise convolutional network （TinyDWConvNet） with progressively expanded depthwise-separable convolutional stages， global average pooling， dropout， and a sixteen-way classifier learns to infer that label directly from a 128×128 low-quality input. The classifier contains 107，217 parameters and is evaluated only once at deployment； it does not require a ground-truth image， repeated reconstruction with all candidate paths， or online computation of image-quality measures. The three components are deliberately placed at different processing scales rather than stacked at a single resolution. HPDM protects high-frequency evidence before irreversible spatial contraction； STDFM operates after compression， when global relationships can be estimated efficiently； and SPPM acts outside the denoiser to allocate the reverse trajectory. This separation keeps each component aligned with a distinct source of error and avoids changing the pretrained diffusion prior. During reconstruction， the selected path determines only the sequence of reverse timesteps， while the enhanced noise predictor supplies the state-dependent noise estimate required at those timesteps. Consequently， the same pretrained diffusion backbone and interface remain available for one to five-step inference. The structural network and the path classifier use separate objectives， so the classifier can be extended without redesigning the main diffusion backbone.ResultTraining uses the LSDIR（A Large Scale Dataset for Image Restoration） dataset together with a 20，000-image subset of FFHQ（Flickr-Faces-HQ）. Low-quality inputs are synthesized with the Real-ESRGAN（Real-world Enhanced Super-Resolution Generative Adversarial Network） degradation pipeline， and the scale factor is ×4 with 512×512 high-resolution crops and 128×128 low-resolution inputs. Evaluation is conducted on ImageNet-Test， the paired RealSR（Real-World Super-Resolution） dataset， and the unpaired RealSet80 dataset. Full-reference assessment uses peak signal-to-noise ratio （PSNR）， structural similarity （SSIM）， and learned perceptual image patch similarity （LPIPS）； no-reference assessment uses the Natural Image Quality Evaluator （NIQE）， Perceptual Index （PI）， Multi-scale Image Quality Transformer （MUSIQ）， and contrastive language-image pre-training based image quality assessment（CLIP-IQA）. On ImageNet-Test， compared with one-step InvSR， the complete model decreases PSNR from 24.14 dB to 23.65 dB， while increasing SSIM from 0.6789 to 0.6808. It lowers NIQE from 4.3815 to 4.0004 and PI from 3.0866 to 2.9565， while increasing CLIP-IQA from 0.7093 to 0.7274. The parameter count decreases from 33.84 million to 28.47 million， a reduction of approximately 15.9%. The HPDM-plus-STDFM configuration provides the strongest fidelity-oriented ablation result， reaching 24.59 dB PSNR and 0.6902 SSIM with 28.37 million parameters， whereas adding SPPM shifts the balance toward input-dependent perceptual quality and produces the best CLIP-IQA score. On RealSR， the complete model decreases NIQE from 4.2189 to 3.9566 and PI from 3.7779 to 3.6148 relative to InvSR-1， although its full-reference and some semantic-perceptual scores do not all improve. On RealSet80， NIQE decreases from 4.0284 to 3.8938 and PI from 3.4666 to 3.4479， while CLIP-IQA and MUSIQ remain below the baseline. Visual comparisons show clearer local edges after HPDM and more continuous structural boundaries after STDFM.ConclusionThe results indicate that high-resolution detail-preserving compression and bottleneck-level structure-texture coordination are complementary： the former retains evidence that would otherwise be lost before deep encoding， and the latter reorganizes that evidence under a broader structural context.Degradation-aware path prediction further changes reverse diffusion from a fixed schedule into an input-dependent decision without altering the pretrained diffusion backbone. The proposed method is therefore an effective joint structural and scheduling extension for low-step diffusion-inversion super-resolution. Its advantages are most consistent for ImageNet-Test and for no-reference naturalness on real degradations. The remaining variation among fidelity， semantic， and perceptual measures also exposes the current limitations： offline utility labels depend on a finite candidate set， one image-level path cannot accommodate different degradations within the same image， and path decisions are not optimized end to end with reconstruction. Future work will investigate larger and continuous timestep spaces， region-level or patch-level scheduling， joint optimization of the classifier and super-resolution model， and validation across additional degradation types， scale factors， and application domains.关键词：single image super-resolution;diffusion model;diffusion inversion;multi-scale feature enhancement;noise prediction network;sampling path prediction;real-world image restoration1|0|0更新时间：2026-11-02 -
Vision foundation model-guided learning for label-efficient remote sensing salient object detection AI导读
摘要：ObjectiveSalient object detection is a fundamental task in remote sensing image understanding. However， most existing approaches are developed under a fully supervised learning paradigm， where a large number of pixel-level annotations are required for model training. This strong dependence on dense annotations limits the scalability and practical deployment of current deep models in label-scarce remote sensing scenarios. Nevertheless， existing semi-supervised methods still suffer from noisy pseudo-labels， error accumulation， insufficient utilization of unlabeled samples， and a considerable performance gap compared with fully supervised methods. To address these issues， this paper proposes a foundation model-guided learning method for label-efficient remote sensing salient object detection， aiming to improve the utilization of unlabeled remote sensing images and enhance detection accuracy under limited annotation conditions.MethodThe proposed framework consists of two closely related models： ST-SAM3 and DistillMatch. First， we introduce the SAM3 visual foundation model into semi-supervised remote sensing salient object detection and construct a progressive self-training framework named ST-SAM3. Instead of directly using unreliable pseudo-labels generated from a model trained with only a few labeled samples， ST-SAM3 exploits the prompt-based mask prediction capability of SAM3 to refine and evaluate pseudo-labels in an automatic manner. Specifically， a lightweight adapter is fine-tuned on a small labeled subset to adapt SAM3 to remote sensing salient object detection. The fine-tuned model then generates initial pseudo-labels for unlabeled images. Based on these pseudo-labels， foreground and background prompts are automatically extracted from connected components and background regions. These prompts are fed back into SAM3 to perform iterative prompt-based mask refinement， thereby improving the spatial completeness and boundary quality of pseudo-labels. To further assess pseudo-label reliability， we design a quality assessment strategy that jointly evaluates geometric consistency among the original pseudo-label， the prompt-refined mask， and the mask predicted from the horizontally flipped image. High-quality unlabeled samples are selected according to this reliability score， and consensus pseudo-labels are generated through voting among the initial prediction， prompt-refined prediction， and augmentation-consistent prediction. This procedure progressively expands the labeled training set without requiring additional human intervention. On the basis of the reliable pseudo-labeled samples selected by ST-SAM3， we further propose DistillMatch to exploit the remaining unlabeled data more effectively. The reliable pseudo-labeled samples are treated as an extended labeled set， while the remaining unlabeled samples are used for consistency regularization. Different from conventional consistency learning that mainly relies on photometric perturbations， DistillMatch explicitly introduces geometric perturbations， including scale transformation and rotation transformation， to enforce geometric consistency between teacher supervision and student prediction. This design helps the student model learn robust salient object representations under spatial variations that frequently appear in remote sensing images. In addition， we design a high-order statistical distillation strategy to align multi-level feature statistics between teacher and student models within multiscale salient features， enabling the student model to inherit not only pixel-level pseudo-supervision but also higher-level feature distribution knowledge from the foundation-model-based teacher.ResultExtensive experiments are conducted on three remote sensing salient object detection datasets， namely ORSSD， EORSSD， and ORSI-4199， under 5% and 10% labeled-data settings. Experimental results show that ST-SAM3 achieves superior performance across different datasets and annotation ratios， demonstrating the effectiveness of progressive self-training and reliable pseudo-label selection. Meanwhile， DistillMatch retains competitive detection accuracy with a substantially lighter model architecture. The proposed frameworks consistently outperform existing state-of-the-art semi-supervised methods.ConclusionThis study demonstrates that visual foundation models can provide powerful priors for label-efficient remote sensing salient object detection. By introducing SAM3 into a progressive self-training framework， pseudo-labels are automatically refined and subsequently selected according to their reliability， without requiring additional manual annotation. By further combining reliable pseudo-label supervision with geometric consistency learning and high-order statistical information distillation， the proposed DistillMatch enables a lightweight student model to effectively learn from both labeled and unlabeled remote sensing images. Overall， ST-SAM3 and DistillMatch alleviate the dependence on large-scale pixel-level annotations and provide an effective and practical solution for semi-supervised remote sensing salient object detection.关键词：remote sensing salient object detection;semi-supervised learning;visual foundation model;progressive self-training;geometric consistency learning;statistical information distillation2|0|0更新时间：2026-11-02 -
摘要：Sparse-view 3D reconstruction aims to recover the structure and appearance of a scene or object from a limited number of observed images. Insufficient viewpoint coverage， limited image overlap， and occlusion make the problem underconstrained， often leading to incomplete geometry， blurred textures， floating artifacts， and inconsistent renderings. Although neural radiance fields （NeRFs） and 3D Gaussian Splatting （3DGS） improve scene representation and rendering， they do not eliminate the ambiguity caused by missing observations. Diffusion models learn regularities of appearance and structure from large-scale image datasets， providing generative priors for recovering poorly observed regions. However， plausible images may still contain structures or textures that conflict with the actual scene. The central challenge is to incorporate generative information while maintaining fidelity to observed content and consistency across viewpoints. This review focuses on static 3D reconstruction from a small number of observed images， considering both known and unknown camera poses. Single-image 3D generation and conditional novel-view generation are discussed as supporting techniques for viewpoint control， prior construction， and observation completion. Video diffusion is included where generated viewpoint sequences assist the recovery of a static scene； dynamic-scene reconstruction is outside the scope. The review introduces conditional diffusion， score functions， Score Distillation Sampling （SDS）， NeRFs， 3DGS， and differentiable rendering to explain how information learned in image space can influence three-dimensional recovery. The methodological discussion is organized into three sections with distinct purposes. The first two examine diffusion-prior constraints and generated-observation-driven reconstruction as mechanisms for using diffusion-derived information. The third investigates feedback between generation and reconstruction as a separate interaction dimension. These sections therefore do not define three mutually exclusive method categories. Individual methods are described through their optimization objectives， intermediate information， and data flow at each stage. When a framework combines mechanisms or supports alternative reconstruction modules， its different configurations are identified. Diffusion-prior constraints convert score estimates， denoising residuals， or distribution-matching objectives into optimization signals for a three-dimensional representation. These signals are propagated through differentiable rendering to update radiance fields， Gaussian primitives， or implicit surfaces， while reconstruction losses preserve agreement with observed images. Some methods obtain target estimates through multiple denoising steps and use the differences between these estimates and current renderings to guide optimization. Such intermediate images must be interpreted according to their role in the objective and update procedure. The effectiveness of this mechanism depends on prior accuracy， conditioning information， camera poses， and the balance between observation losses and generative guidance. Excessive prior influence can introduce unsupported content， whereas insufficient influence leaves poorly observed regions inadequately constrained. Generated-observation-driven reconstruction uses synthesized or restored RGB images， depth maps， surface normals， viewpoint sequences， or their features as auxiliary supervision or reconstruction inputs. Generated information can support scene-specific optimization or be processed by pretrained reconstruction modules. The recovery procedure may consequently involve iterative fitting， direct parameter prediction， or conditional three-dimensional generation. Feedforward prediction reduces the need for per-scene parameter optimization， although preceding view generation may still require multiple diffusion steps. Nevertheless， errors in shape， texture， visibility， depth， or scale can propagate into the final representation through supervision losses or network inputs. Increasing the number of generated views does not by itself ensure stronger or more reliable geometric constraints. Generation–reconstruction feedback concerns the interaction between generative processing and the current three-dimensional state. Rendered images， depth estimates， visibility information， or auxiliary geometry can guide restoration， and the resulting information can subsequently update the scene representation. A single restoration stage followed by reconstruction from fixed observations is distinguished from repeated feedback， in which the updated state influences later generation. Feedback may operate through prior distillation， generated-image supervision， or learned parameter updates. Image-level interaction focuses on selected viewpoints， whereas sequence-level interaction organizes neighboring views along camera trajectories. These distinctions clarify where feedback occurs and which representation is actually updated. Feedback allows generation to adapt to the reconstruction state， but it also creates additional paths for error propagation. Incorrect geometry can produce unreliable depth， visibility， or spatial correspondences that bias restoration. Subsequent reconstruction may absorb these errors and reinforce them in later rounds. Agreement between restored images and current renderings is therefore insufficient evidence of accuracy. Reliable feedback requires support from real observations and cross-view geometry， together with mechanisms for rejecting inconsistent information， adjusting its influence， and controlling update frequency. Representative methods are compared in terms of supervision， three-dimensional representation， recovery procedure， and interaction pattern. The review also summarizes commonly used object-level and scene-level datasets and discusses image quality， perceptual similarity， and geometric accuracy. Evaluation measures include Peak Signal-to-Noise Ratio （PSNR）， Structural Similarity Index Measure （SSIM）， Learned Perceptual Image Patch Similarity （LPIPS）， Chamfer Distance， and F-score. Directly generated novel views and views rendered from a reconstructed representation are treated as different evaluation outputs. Differences in input-view selection， pose assumptions， scene coverage， test splits， and access to geometric ground truth limit comparisons across studies. Improved rendering quality should therefore be examined alongside geometric accuracy and cross-view consistency， rather than interpreted as sufficient evidence of successful reconstruction. Future research should strengthen the coordination between diffusion priors， observed images， and geometric constraints. Prior strength and generated supervision should account for observation coverage， uncertainty， and reconstruction progress. Feedback should be evaluated against observational and geometric evidence that is independent of the current restoration result. Robustness to pose errors， complex topology， reflective or transparent materials， and distribution shifts remains essential for practical applications. More informative evaluation settings should distinguish scene fitting from generalization and report the costs of diffusion sampling， three-dimensional recovery， and repeated feedback separately. Efficient sampling， diffusion distillation， selective generation， and feedforward reconstruction offer complementary approaches to reducing runtime and memory consumption. Progress along these directions can improve geometric reliability while balancing reconstruction quality， generalization， and computational efficiency.关键词：sparse-view 3D reconstruction;Diffusion models;generative priors;generated observations;diffusion feedback2|0|0更新时间：2026-11-02 -
摘要：ObjectivePerson re-identification （ReID） aims to retrieve the same pedestrian across non-overlapping cameras and is a key technique for intelligent video surveillance， smart cities， and public security. In real deployments， however， cameras， illumination， backgrounds， and even the identities to be recognized change continually， so a ReID model must keep learning from sequentially arriving domains without forgetting the knowledge it has already acquired. Lifelong person re-identification （LReID） is formulated for this purpose， and its central difficulty is catastrophic forgetting. Because the long-term storage of pedestrian images conflicts with privacy protection， storage cost， and regulatory compliance， exemplar-free LReID， in which no historical raw image can be accessed after a training stage ends， has become an important and practical research direction. Existing prototype-based exemplar-free methods compress historical identities into prototypes and reuse them to preserve old decision boundaries or to guide current feature learning. However， they usually treat all historical prototypes as static and equally trustworthy knowledge sources. As learning stages accumulate， historical prototypes differ markedly in domain relevance， statistical reliability， and boundary risk， and feature-space drift further misaligns the prototypes extracted by earlier models with the current representation space. Transferring these prototypes uniformly injects irrelevant， noisy， and boundary-ambiguous supervision into current training， which weakens old-domain retention and disturbs new-domain adaptation at the same time. We therefore argue that historical prototypes should not only be constructed but also governed， and we propose a governed prototype knowledge transfer （GPKT） framework that selects， calibrates， and trust-weights historical prototypes before they are allowed to influence the current model.MethodGPKT follows a “select–calibrate–govern–distill” pipeline built on a standard ReID backbone. First， a hierarchical distribution prototype memory is constructed at the end of each stage without storing any historical image. It records an identity-level distribution prototype， namely the feature mean and per-dimension variance， for every historical identity， a domain-level descriptor consisting of the global feature mean and variance for every historical domain， and sample-support counts that indicate how reliably each prototype is estimated. Second， relevance-guided prototype selection and calibration performs coarse-to-fine retrieval. The most relevant historical domains are retrieved according to the similarity of feature centers and the consistency of log-variances between each domain descriptor and the current mini-batch， and the identity prototypes closest to the current identity centers are then kept within the selected domains. The selected prototypes are calibrated by residual moment matching， which preserves the relative position of each prototype inside its source domain while shifting and rescaling it to the current mini-batch statistics； a midpoint update and variance clipping further prevent over-correction caused by noisy batch statistics. Third， reliability- and risk-aware prototype governance assigns a continuous governance weight to each calibrated prototype. The reliability score increases with sample support and penalizes abnormal variance， the boundary-safety score combines a proximity gate and a margin gate to suppress prototypes that lie close to several current identity centers， and a softmax-normalized domain-level prior controls the overall contribution of each selected domain. Finally， governed relation distillation injects the governance weights as a logarithmic prior into the sample-to-prototype assignment， builds a prototype-induced relation distribution among current samples as the teacher， and aligns the relation distribution of current features with this teacher through the Kullback-Leibler （KL） divergence. Because relation structure， rather than identity-level alignment， is transferred， no explicit identity correspondence across domains is required. All prototype-related operations are performed only during training， so GPKT remains a standard ReID feature extractor at inference time.ResultExperiments are conducted on the standard LReID benchmark， which contains 12 person ReID datasets， including 5 seen domains for lifelong training and 7 unseen domains for generalization evaluation， under two training orders， and GPKT is compared with 9 recent representative exemplar-free methods. With the same 50-layer residual network （ResNet-50） backbone， GPKT achieves the best seen-domain and unseen-domain average performance under both orders. Compared with distribution-aware knowledge aligning and prototyping （DKP++）， the strongest competitor on average， GPKT improves the seen-domain average mean average precision （mAP）/Rank-1 accuracy by 3.1/2.7 and 3.0/2.9 percentage points （pp）， and improves the unseen-domain average mAP/Rank-1 （R1） accuracy by 3.2/2.6 and 3.3/2.6 pp under Order-1 and Order-2， respectively. Replacing ResNet-50 with a base-size vision Transformer using 16×16 patches （ViT-B/16） further raises the seen-domain average mAP by 1.5/1.5 pp and the unseen-domain average mAP by 1.8/1.9 pp. Stage-wise evaluation shows that GPKT reduces average forgetting by about 2 pp relative to DKP++ and raises learning accuracy by 1.4 to 1.5 pp； the decomposition of the final performance further indicates that the final gain stems jointly from old-domain retention and new-domain adaptation， with retention contributing slightly more. A full-factorial ablation verifies that prototype selection， residual calibration， and governed distillation are complementary， and the complexity analysis shows that GPKT introduces no extra inference overhead， while its additional training computation is bounded by the fixed numbers of selected domains and prototypes and its prototype memory occupies only 137.5 mebibytes （MB） after five stages.ConclusionBy combining relevance-， reliability-， and risk-aware prototype governance with relation distillation， GPKT turns the use of historical prototypes from indiscriminate reuse into selective， calibrated， and trust-weighted knowledge transfer. Without storing any historical exemplar， it effectively balances old-knowledge retention， new-domain adaptation， and cross-domain generalization， providing a practical solution for privacy-constrained lifelong person re-identification.关键词：catastrophic forgetting;exemplar-free lifelong learning;person re-identification （ReID）;prototype governance;prototype memory1|0|0更新时间：2026-11-02 -
摘要：ObjectiveInfrared small target detection （IRSTD） aims to accurately identify and segment dim targets from complex infrared backgrounds and is essential for maritime search and rescue， early-warning surveillance， and intelligent transportation. This task remains challenging because distant targets usually occupy only a few pixels， contain limited texture and contour information， and exhibit weak contrast against their surroundings. Meanwhile， sea-surface highlights， cloud edges， isolated thermal sources， and directional ground textures can produce target-like responses， leading to missed detections and false alarms. U-Net-based models have become a dominant framework for IRSTD because their encoder–decoder structures support multi-scale representation learning and their skip connections preserve high-resolution details. However， two limitations remain insufficiently addressed. First， repeated convolution and downsampling may progressively weaken the already sparse responses of small targets， whereas conventional multi-scale structures do not always preserve fine localization cues while acquiring sufficiently discriminative context. Second， shallow encoder features contain detailed spatial information but also substantial background clutter， whereas deep decoder features provide stronger semantics but lose precise target boundaries. Direct concatenation or addition treats these heterogeneous features indiscriminately， enlarges the semantic gap across layers， and propagates background noise into the reconstruction path. Most existing methods improve either backbone feature extraction or skip-connection fusion but do not jointly regulate target representation， bidirectional cross-layer interaction， and structured clutter suppression. To address these issues， this paper proposes a dual-branch interactive guidance network （DIGNet）， which integrates dual-branch multi-scale representation， cross-layer interactive guidance， and direction-aware background suppression within a unified hierarchical encoder–decoder framework.MethodDIGNet contains four principal components： residual U-shaped blocks （RUB） and their dilated variants （RUBD）， a dual-branch interactive fusion （DIF） module， a grouped cross-spatial interactive attention （GCIA） module， and multi-scale deep supervision. RUB is used in shallow and intermediate stages to construct an internal lightweight encoder–decoder path within each block. This design aggregates features at different resolutions while retaining a residual shortcut， thereby preserving local structures and stabilizing the transmission of weak target information. In deeper stages， RUBD replaces selected standard convolutions with dilated convolutions. It enlarges the effective receptive field without introducing additional downsampling， enabling the network to acquire broader contextual evidence while limiting spatial-detail loss. The hierarchical use of RUB and RUBD therefore supports fine-detail preservation in early layers and context-aware discrimination in deeper layers. At each decoding stage， the current skip-connected feature is first recalibrated by GCIA and then fused with the upsampled decoder feature through DIF. Unlike static skip fusion， DIF explicitly establishes bidirectional guidance between the two inputs through complementary channel- and spatial-interaction branches. In the channel-interaction branch， projected decoder and skip features are correlated to generate a cross-branch weighting matrix. Decoder semantics consequently select target-relevant channels from the high-resolution skip feature and suppress channels dominated by clutter. In the spatial-interaction branch， mapped and normalized skip information generates a spatial gate that constrains the decoder response， allowing detailed positional cues to correct coarse semantic reconstruction. The outputs of the two branches are adaptively combined using learnable coefficients and further refined with coordinate attention， which preserves direction-sensitive positional information. This reciprocal design does not merely transfer encoder details to the decoder； instead， it uses decoder semantics to filter skip information and skip details to regularize decoder localization， thereby reducing cross-layer semantic inconsistency and background-noise propagation. GCIA is applied before and after cross-layer fusion to improve the distinction between true targets and structured background responses. It divides the input channels into groups and performs attention modeling within each group， reducing computational redundancy while improving the stability of local statistics. A direction-aware branch aggregates features along the horizontal and vertical axes to characterize directional patterns such as sea streaks， cloud bands， and elongated ground structures. In parallel， a within-group interaction branch combines local convolution and global aggregation to estimate spatial saliency. The two branches jointly produce recalibration weights that enhance compact target responses and suppress structured clutter. Finally， multiple decoder outputs are supervised using a combination of Soft-IoU and binary cross-entropy losses. Deep supervision provides direct gradient signals to intermediate reconstruction stages， while the joint loss promotes an appropriate balance among target-region overlap， pixel-level classification， and background suppression.ResultTo verify the effectiveness of the proposed DIGNet， extensive experiments were conducted on three widely used IRSTD datasets： Nanjing University of Aeronautics and Astronautics single-frame infrared small target （NUAA-SIRST）， infrared small target detection 1K （IRSTD-1K）， and National University of Defense Technology single-frame infrared small target （NUDT-SIRST）. DIGNet was compared with six traditional methods （Top-Hat， Max-Median， WSLCM， TLLCM， IPI， and NOLC） and eight deep learning-based methods （ACM， RDIAN， DNANet， UIU-Net， RPCANet， MSHNet， SCTransNet， and L²SKNet）. Three standard metrics were employed for objective evaluation： intersection over union （IoU）， probability of detection （ -
摘要：ObjectiveExisting functional brain network-based methods for the auxiliary diagnosis of autism spectrum disorder （ASD） commonly rely on predefined statistical measures to construct inter-regional adjacency matrices and often model the brain as a flat， non-hierarchical graph， which limits their ability to learn diagnosis-relevant connectivity patterns and characterize multi-level abnormalities across brain regions and functional communities.MethodsTo address these limitations， this study proposes a prior-guided region-community hierarchical graph network （PGRC-HGN） for ASD auxiliary diagnosis using resting-state functional magnetic resonance imaging （rs-fMRI）. The proposed framework consists of two major components： a salient temporal-aware graph constructor （STAGC） and a prior-guided hierarchical graph network （PG-HGN）. Given blood-oxygen-level-dependent （BOLD） signals extracted from predefined regions of interest （ROIs）， STAGC first adopts multi-scale temporal convolutional networks to encode regional time-series features under different temporal receptive fields， thereby capturing temporal dependencies at multiple scales. A shared temporal self-attention mechanism is then employed to estimate the diagnostic importance of each time point and select key temporal segments that are more closely associated with ASD-related functional abnormalities. Based on these key-time features， a spatial attention module learns a task-relevant adjacency matrix to describe functional interactions between ROIs. In this way， the graph construction process is no longer solely dependent on predefined correlation measures， but is optimized under the supervision of the downstream classification objective. Meanwhile， the Pearson correlation coefficient （PCC） matrix computed from the original BOLD time series is used as the initial node feature representation rather than as a fixed adjacency structure， allowing the model to preserve useful functional connectivity information while learning a more discriminative graph topology. The learned brain graph is subsequently fed into PG-HGN for hierarchical graph representation learning and classification. Unlike conventional graph neural network （GNN）-based brain network models that mainly perform message passing at the ROI level， PG-HGN jointly models brain functional interactions at both the ROI level and the functional community level. Specifically， functional community priors are introduced to provide biologically meaningful constraints for community-level representation learning. In this study， the CC200 atlas is used for ROI parcellation， and the community prior is derived by mapping ROIs to large-scale functional systems based on canonical functional network organization， including the default mode network （DMN）， frontoparietal network （FPN）， limbic network （LN）， ventral attention network （VAN）， dorsal attention network （DAN）， somatomotor network （SMN）， visual network （VN）， and cerebellar and subcortical structures （CB&SC）. Instead of assigning each ROI to a single fixed community， PG-HGN employs a learnable prototype-based attention mechanism to dynamically estimate the affiliation probabilities between ROIs and functional communities. This design enables each ROI to participate in multiple communities with different weights， thereby improving the model’s ability to characterize individual variability and overlapping community organization while maintaining the semantic constraints provided by functional community priors. At the ROI level， graph convolution is used to learn local functional interaction representations among brain regions. At the community level， the dynamically estimated ROI-to-community affiliation matrix is used to aggregate regional features into community-level representations. A cross-level interaction mechanism is then introduced to enable information exchange between the ROI layer and the community layer. Specifically， community-level reasoning is performed to model interactions among large-scale functional systems， and the refined community representations are further propagated back to the ROI layer， allowing local regional features to be updated with global community-level contextual information. Finally， the ROI-level and community-level representations are fused to form a graph-level feature representation， which is fed into a multilayer perceptron for ASD classification. Through this design， PGRC-HGN integrates key-time-aware task-driven graph structure learning with prior-guided hierarchical community modeling， enabling joint characterization of local ROI-level abnormalities and system-level community coordination disruptions.ResultsExperiments were conducted on the preprocessed Autism Brain Imaging Data Exchange I （ABIDE I） dataset. Subjects with fewer than 100 time points were excluded， and all remaining rs-fMRI sequences were truncated to the first 100 frames to ensure consistent temporal length. A total of 1，009 subjects， including 493 individuals with ASD and 516 typical controls， were used for evaluation. The CC200 atlas was adopted to parcellate the brain into 200 ROIs. Five-fold cross-validation was employed to assess the classification performance of the proposed model. Experimental results show that PGRC-HGN achieved an accuracy （ACC） of 75.27%， an area under the receiver operating characteristic curve （AUC） of 79.63%， a sensitivity （SEN） of 75.06%， and a specificity （SPE） of 76.62%. Compared with representative competing methods， including BrainGNN， BrainNetTF， Com-BTF， FBNetGen， RGTNet， and DHGFormer， the proposed method achieved superior overall diagnostic performance. In particular， compared with DHGFormer， the best-performing baseline method in terms of comprehensive performance， PGRC-HGN improved ACC， AUC， SEN， and SPE by 2.76， 2.88， 3.47， and 3.28 percentage points， respectively. These results indicate that the proposed method can more effectively learn discriminative functional connectivity patterns from rs-fMRI data and improve ASD classification performance.ConclusionsThe proposed PGRC-HGN combines task-driven graph structure learning with prior-guided region-community hierarchical modeling for ASD auxiliary diagnosis. By explicitly selecting diagnostically informative time points， STAGC reduces the interference of irrelevant temporal segments and enhances the learning of task-relevant dynamic functional connectivity patterns. By introducing functional community priors and dynamically estimating ROI-to-community affiliations， PG-HGN balances biological interpretability and individual-specific flexibility in hierarchical brain network modeling. The experimental results on ABIDE I demonstrate that PGRC-HGN achieves superior classification performance and provides more interpretable hierarchical representations of ASD-related functional abnormalities. Therefore， the proposed method may provide useful imaging-based support for ASD auxiliary diagnosis and contribute to the analysis of abnormal functional organization in ASD.关键词：autism spectrum disorder;resting-state functional magnetic resonance imaging;Graph neural network;functional connectivity;temporal convolutional network1|0|0更新时间：2026-11-02 -
摘要：ObjectiveHeterogeneous remote sensing images acquired by different sensors exhibit substantial differences in imaging mechanisms， radiometric responses， spatial textures， and noise characteristics. Direct comparison of such images can generate numerous modality-induced pseudo changes. Moreover， existing unsupervised cross-modal learning methods often apply alignment constraints to the entire image without distinguishing changed pixels from unchanged pixels. Because genuinely changed pixels do not satisfy the stable cross-modal correspondence of unchanged regions， their participation in global alignment may disturb the learning of reliable modality mappings and reduce the separability between changed and unchanged regions. In addition， conventional transformation models have limited ability to represent complex and asymmetric nonlinear relationships between heterogeneous modalities. To address these problems， a dendritic neuron model-based cross-modal reconstruction network， termed DNM-CMR， is proposed for unsupervised heterogeneous remote sensing image change detectionMethodDNM-CMR transforms the direct comparison of heterogeneous images into reconstruction-error analysis in the target-modality domain. First， a dual-branch autoencoder architecture with a multiscale convolutional encoding module is constructed to extract hierarchical spatial features from the source-modality image. Local neighborhoods centered on individual pixels are then unfolded into contextual feature vectors containing central-pixel responses， surrounding textures， boundaries， and spatial information. These vectors are fed into a dendritic neuron model to reconstruct the corresponding target-modality responses.The DNM consists of synaptic， dendritic， membrane， and somatic layers. Multiple dendritic branches perform nonlinear modulation and multiplicative aggregation of local feature components， enabling the model to represent high-order interactions among different spatial features and establish complex nonlinear mappings between heterogeneous modalities. The responses of different dendritic branches are subsequently integrated to generate the target-modality reconstruction. Because unchanged regions generally preserve relatively stable relationships between the source and target modalities， they can be reconstructed with relatively small errors. In contrast， genuine changes disrupt the learned cross-modal correspondence and usually produce larger reconstruction residuals. Therefore， the absolute difference between the reconstructed image and the real target image is used to generate the change-intensity map.A dynamic region-mask mechanism is introduced to reduce the interference of genuinely changed pixels during network training. Reconstruction deviation is combined with the channelwise variance of encoded features to estimate the potential change degree of each pixel. After normalization and weighted fusion， K-means clustering divides the pixels into changed and unchanged candidate regions. An exponential moving average is then used to update the masks periodically， reducing fluctuations caused by unstable reconstruction and individual clustering results.The network is jointly optimized using reconstruction loss， structural similarity loss， adversarial loss， and feature consistency loss. The reconstruction and structural similarity losses preserve pixel responses and local spatial structures. The adversarial loss improves the distribution consistency between the reconstructed image and the real target image， whereas the feature consistency loss reduces cross-modal feature differences in unchanged regions. The dynamically estimated unchanged-region mask is applied to the adversarial and feature consistency terms， ensuring that these constraints mainly act on reliable unchanged regions and preventing genuinely changed pixels from being forcibly aligned. During the initial training stage， the network learns a basic cross-modal mapping through reconstruction and structural constraints. The dynamic region mask and consistency constraints are subsequently introduced to refine the mapping and improve change separability.ResultExperiments were conducted on four publicly available heterogeneous remote sensing datasets， namely Italy， Gloucester， California， and Texas， covering lake expansion， flooding， and forest-fire events. The proposed method was compared with five representative unsupervised methods： SRGCAE， SCASC， RIEM， CDRL， and CFRL. The overall accuracies achieved by DNM-CMR on the four datasets were 0.973 4， 0.957 1， 0.924 5， and 0.989 0， respectively. DNM-CMR also obtained the highest Kappa coefficient and F1 score among the comparison methods on all four datasets. Visual results demonstrate that the proposed method effectively suppresses extensive background false alarms while preserving the principal structures of changed regions. Although several comparison methods achieved better individual AUC， false-positive， or false-negative values on specific datasets， DNM-CMR generally provided a more balanced tradeoff between false alarms and missed detections.Ablation experiments on the four datasets further verified the effectiveness of the proposed components. Applying adversarial and feature consistency losses to the entire image caused genuinely changed pixels to interfere with cross-modal consistency learning. Introducing an unchanged-region mask substantially improved the detection results， while dynamically updating the mask further enhanced its adaptability to the current reconstruction state. Replacing the DNM reconstruction module with a conventional convolutional decoder resulted in performance degradation on all four datasets， demonstrating the effectiveness of the DNM in representing complex nonlinear cross-modal relationships.ConclusionDNM-CMR establishes a nonlinear mapping between heterogeneous image modalities and extracts changes through target-domain reconstruction residuals rather than direct heterogeneous comparison. The dendritic neuron model enhances the representation of complex local feature relationships， while the dynamic region mask reduces the adverse influence of changed pixels on network optimization. Experimental results demonstrate that the proposed method can effectively suppress modality-induced pseudo changes and achieve accurate and balanced change detection without requiring manually labeled samples. The complete implementation code of the algorithm described in this paper has been made publicly available， and the open-source address is：https：//cstr.cn/10441.11.sciencedb.j00240.0009x关键词：heterogeneous remote sensing images;unsupervised change detection;cross-modal reconstruction;dendritic neuron model;dynamic region mask2|0|0更新时间：2026-11-02 -
摘要：ObjectiveExisting pig behaviour video datasets remain limited in their coverage of behaviour categories， annotation of welfare-related behaviours and annotation granularity. These limitations hinder the training and evaluation of models for multi-class， fine-grained pig behaviour recognition in complex group-housing environments. Furthermore， some datasets are not publicly available， further limiting fair comparisons between methods and the reproducibility of experimental results. Meanwhile， background interference， occlusion between pigs and frequent interactions in complex group-housing environments make it more difficult to extract behaviour-relevant local regions and temporal dynamics. This hinders fine-grained behaviour recognition， particularly for behaviours that rely on local interaction cues， such as fighting and nose touching， and those involving continuous postural transitions， such as standing up and lying down. To address these issues， this study aims to develop a video dataset of pig behaviour covering multiple basic behaviour classes， fine-grained postures and movements， and welfare-related behaviours. It also proposes a spatiotemporal feature-modelling method for complex group-housing environments to improve fine-grained pig behaviour recognition.MethodThe 13-class Pig Behaviour Video Dataset （PBVD-13） is constructed from publicly available red， green and blue （RGB） videos of pigs in group-housing environments， with individual behaviours annotated at one-second intervals. It comprises 13 behaviour classes， covering basic activities， welfare-related behaviours such as exploration， toy interaction， fighting and nose touching， and fine-grained postural and movement behaviours such as standing up and lying down. The dataset provides a basis for multi-class pig behaviour recognition in complex group-housing environments. To address the difficulty of adequately representing local interactions and continuous motion changes in complex behaviours， a spatiotemporal perception and enhancement network is proposed. It employs a low-frame-rate spatial branch and a high-frame-rate motion branch to extract behaviour-related spatial features and local dynamics， respectively. The spatial branch characterises relatively stable behaviour-related regions， while the motion branch captures local motion changes between consecutive frames. The network further integrates spatial attention， motion feature enhancement and long short-term memory （LSTM） temporal modelling to fuse features across different temporal scales and achieve fine-grained pig behaviour classification. The source footage comprises publicly available recordings of eight pigs housed at a research pig unit at Scotland’s Rural College. Twenty videos are selected for fine-grained behaviour annotation. Each complete video serves as the smallest unit for dataset partitioning， and the data are divided into training， validation and test sets in a ratio of 7：2：1 to reduce the risk of information leakage caused by adjacent video frames being assigned to different sets. Behaviour recognition performance is evaluated using average precision （AP） and mean average precision （mAP）. Dataset benchmarking， model comparisons， ablation studies and external validation on the independent HotPig dataset are conducted to assess， respectively， the dataset’s suitability for different technical approaches， the recognition performance of the proposed method， the contributions of individual modules and the method’s applicability across datasets.ResultPBVD-13 comprises 13 behaviour classes and 13，500 behaviour instances. The four welfare-related behaviour classes comprise 3，513 instances， accounting for 26.02% of all annotated instances. Benchmark experiments show that recognition performance varies considerably across behaviour classes. Complex behaviours， particularly fighting， nose touching， standing up and lying down， remain challenging to recognise， indicating that the dataset supports model training while presenting challenges for behaviour classification. The proposed method achieves a mAP of 75.92% on PBVD-13， outperforming the best-performing comparison model， SlowFast， by 9.41 percentage points. Its AP values for fighting， standing up and lying down reach 60.75%， 52.37% and 41.98%， respectively. Ablation studies show that removing either spatial-attention module， the motion-enhancement module or the LSTM module reduces mAP， indicating that spatial feature enhancement， local motion modelling and temporal information extraction contribute to overall recognition performance. Furthermore， the proposed method achieves an mAP of 76.64% on the independent HotPig dataset， providing preliminary evidence of its applicability across different datasets. Compared with existing pig behaviour datasets examined in this study， PBVD-13 extends the number of behaviour classes to 13. It retains basic activity behaviours while incorporating welfare-related behaviours， including exploration， toy interaction， fighting and nose touching， as well as fine-grained postural and movement behaviours， such as standing up and lying down. The publicly available dataset also supports reproducibility and comparative evaluation in future research. The proposed method also achieves higher AP values than the comparison models for complex behaviours， including fighting， standing up and lying down. These results indicate that dual-branch spatiotemporal feature modelling contributes to the recognition of the 13 pig behaviour classes considered in this study.ConclusionThis study constructs and publicly releases PBVD-13， a fine-grained video dataset covering 13 pig behaviour classes， and proposes a spatiotemporal perception and enhancement network. The network integrates a low-frame-rate spatial branch， a high-frame-rate motion branch and subsequent temporal modelling to extract behaviour-related spatial features and dynamic information. Benchmarking， model comparison， ablation and external validation experiments show that PBVD-13 supports the training and evaluation of multiclass pig behaviour recognition methods. The proposed network achieves promising recognition performance under the experimental conditions and demonstrates applicability to the independent HotPig dataset. PBVD-13 provides a data resource for multiclass pig behaviour recognition in complex group-housing environments， welfare-related behaviour analysis and fair comparison of related algorithms. The proposed network provides a methodological basis for jointly modelling local interaction features and continuous postural changes. As the current dataset includes only eight pigs and a limited number of videos， future work should expand the dataset to include more pigs， farming environments and behaviour instances. The model’s generalisation should also be evaluated under varying husbandry conditions， individual differences and levels of occlusion to improve its practical applicability. The dataset and source code are available at：https：//github.com/qiqiqi7777/PBVD-13-dataset.关键词：Pig;video dataset;behavioural recognition;spatio-temporal features;attention mechanism3|0|0更新时间：2026-11-02 -
摘要：ObjectiveWith the rapid development of intelligent perception and biometric recognition technologies, cross-modal identity recognition has become an important solution for contactless authentication in aviation applications, such as intelligent access control, security inspection, video surveillance, and attendance management. Among various biometric traits, face and gait provide complementary identity information. Face recognition offers strong discriminative capability under favorable imaging conditions, whereas gait recognition enables long-distance and non-contact identification. However, practical aviation environments are often affected by low-quality video, motion blur, long-distance observation, occlusion, and appearance variations, which significantly reduce the performance of existing methods. Moreover, most studies optimize face restoration or gait recognition independently, while the complementary information between heterogeneous modalities and the temporal consistency in video sequences remain insufficiently exploited. To address these challenges, this paper proposes a unified cross-modal identity recognition framework incorporating temporal contextual information to improve accuracy and robustness in complex aviation scenarios.MethodsThe proposed framework consists of three progressive modules. First, a Dynamic Spatial-Temporal-Channel (DSTC) model is developed for low-quality video face restoration. By jointly modeling spatial structures, temporal consistency among adjacent frames, and channel-wise feature dependencies, the proposed model effectively compensates for missing facial details caused by image degradation while preserving identity-related characteristics. The restored facial representations provide reliable identity information for subsequent recognition tasks. Second, a lightweight gait feature extraction network is designed by integrating Spatial and Channel Reconstruction Convolution (SCConv) with a Multi-Scale Dynamic Augmentation Module (MS-DAM). SCConv reconstructs spatial and channel features to reduce feature redundancy and improve computational efficiency, while MS-DAM captures gait information at different temporal and spatial scales to enhance dynamic feature representation. The combination of these two modules improves the robustness of gait recognition under viewpoint variation, carrying conditions, and clothing changes. Finally, a Context-aware Temporal (CoT) fusion strategy is proposed to integrate heterogeneous face and gait features. Instead of directly concatenating multimodal representations, the proposed fusion module exploits temporal contextual relationships to adaptively combine complementary identity information from the two modalities. The three modules are connected in a progressive manner, where high-quality face restoration strengthens facial feature representation, enhanced gait modeling provides discriminative motion features, and context-aware fusion further constructs a unified identity representation for robust cross-modal recognition.ResultsExtensive experiments were conducted to evaluate the proposed framework from three aspects: low-quality video face restoration, gait recognition, and cross-modal identity recognition. For face restoration, experiments were performed on the public LFW dataset and a self-collected contactless attendance dataset. Three widely used image quality assessment metrics, namely Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS), were adopted for quantitative evaluation. The proposed DSTC model achieved PSNR values of 27.43 dB and 26.22 dB, SSIM values of 0.813 and 0.801, and LPIPS scores of 0.189 and 0.197 on the two datasets, respectively, demonstrating its ability to recover degraded facial details while preserving identity-related information. For gait recognition, experiments on the CASIA-B dataset under the normal walking (NM), carrying bag (BG), and clothing variation (CL) protocols yielded Rank-1 accuracies of 97.6%, 94.6%, and 84.9%, respectively. These results indicate that the proposed SCConv-MS-DAM network effectively improves multi-scale dynamic feature representation and maintains robust recognition performance under challenging walking conditions. Furthermore, the proposed Context-aware Temporal fusion algorithm was evaluated on the CASIA-B face-gait dataset. By jointly exploiting restored facial features and discriminative gait representations, the proposed cross-modal framework achieved Rank-1 recognition accuracies of 97.8%, 95.4%, and 86.9% under the NM, BG, and CL scenarios, respectively. The experimental results consistently demonstrate that the proposed method outperforms representative mainstream approaches under different evaluation protocols and achieves more stable recognition performance in complex environments.ConclusionA A unified cross-modal identity recognition framework incorporating temporal context is proposed for intelligent aviation. It progressively integrates low-quality video face restoration, lightweight gait feature extraction, and context-aware multimodal fusion to improve identity representation reliability. The DSTC module enhances facial quality via spatial, temporal, and channel correlations, while SCConv-MS-DAM strengthens gait features through spatial-channel reconstruction and multi-scale dynamic modeling. Based on these enhanced unimodal representations, the CoT fusion strategy exploits complementary face and gait characteristics to generate more discriminative identity features. Experiments on public and self-collected datasets demonstrate superior performance in face restoration, gait recognition, and cross-modal identity recognition with a lightweight architecture. These findings verify the framework's effectiveness and its potential for practical deployment in aviation-oriented contactless authentication systems under complex real-world conditions.关键词：cross-modal person identification;face restoration;face recognition;gait recognition;temporal context fusion8|6|0更新时间：2026-11-02 -
摘要：ObjectiveZero-shot artistic font generation synthesizes target artistic font images from independent reference samples and supports graphic design and multimedia content creation. The task requires a model to identify， disentangle， and recombine three heterogeneous attributes： character identity， glyph structure， and visual text effect. Character identity determines semantic correctness and topology， glyph structure controls strokes and connectivity， and visual text effect defines texture， shading， color， and background. Because these attributes correspond to discrete semantics， topological structure， and high-frequency visual patterns， they differ substantially in representation and can interfere with one another during unified generation. Existing methods often model artistic font style as a single holistic factor， depend on adversarial training or attribute-wise supervision， or rely on static masks and predefined templates. Although such methods may perform well on familiar pairings observed during training， they tend to produce missing strokes， structural drift， texture leakage， color contamination， and style deviation when characters， glyphs， and effects are recombined in unseen ways. To address these problems， a multi-condition diffusion method is proposed for controllable zero-shot artistic font generation. The objective is to model character， glyph， and effect information as independent but coordinated conditions for unseen attribute combinations.MethodA compositional multi-condition diffusion framework， is developed with three condition-specific encoders and a conditional U-Net backbone. Independent character， glyph， and visual-effect reference images are encoded into multi-scale latent features and injected into the denoising process as separate guidance signals. This design allows semantic identity， structural design， and appearance style to jointly control the reverse diffusion trajectory without being compressed into a unified style code. The framework contains three major components. First， a cross-domain feature aggregation module uses character features as structural anchors and introduces channel recalibration to preserve global topology after repeated downsampling. It further applies perpendicular-component attention to remove redundant or interfering components from the joint glyph-effect feature space. This operation reduces the propagation of irrelevant textures from glyph references and unwanted structural patterns from effect references. Second， adaptive instance normalization-based skip connections transfer multi-scale structural features and style statistics from the encoder to the decoder. Means and standard deviations extracted from fused glyph and effect features are used to modulate normalized content features at corresponding decoding stages， enabling the model to retain clear stroke boundaries while reconstructing target colors， textures， and backgrounds. Third， compositional contrastive learning is introduced to improve sensitivity to attribute mismatch. Character， glyph， and effect are treated as separately alignable dimensions. For each target sample， matched combinations are used as positive references， while three negative groups are generated by replacing one attribute at a time. An InfoNCE-based objective pulls generated representations toward the correct composition and pushes them away from combinations containing an incorrect character， glyph， or effect. The final training objective combines denoising loss， perceptual loss， and compositional contrastive loss. To address the limited attribute diversity， structural complexity， bilingual coverage， and combination completeness of existing datasets， the SSAFv2 dataset is constructed for systematic evaluation of unseen combinations. It contains 921，600 artistic font images， 1，024 Chinese and English characters， 30 glyph styles， and 30 visual effects， forming 900 complete glyph-effect categories. The glyph library includes standard printing， creative printing， handwritten styles， and calligraphic styles， while the effect library covers fine textures inspired by natural and industrial materials. SSAFv2 is divided into unseen-character/seen-style， seen-character/unseen-style， and unseen-character/unseen-style subsets to evaluate semantic generalization， compositional style generalization， and their joint difficulty.ResultExperiments are conducted on the SSAFv2， TE141K-EN， and Fonts datasets under a unified evaluation protocol. Performance is measured using L1 loss， Fréchet Inception Distance， and structural similarity， supplemented by qualitative comparison， ablation studies， latent-feature analysis， contrastive-score visualization， and a user study. On the unseen-character/seen-style， seen-character/unseen-style， and unseen-character/unseen-style subsets of SSAFv2， ArtFontDiff obtains L1 losses of 0.1754， 0.1417， and 0.1759； Fréchet Inception Distance scores of 34.58， 30.19， and 52.73； and structural similarity scores of 0.6351， 0.8600， and 0.4870， respectively. These results are overall superior to those of the compared zero-shot artistic font synthesis， image-to-image translation， and diffusion-based font generation methods. On Fonts， TE141K-EN， and SSAFv2-EN， the proposed framework achieves L1 losses of 0.1030， 0.0753， and 0.0380； Fréchet Inception Distance scores of 11.37， 24.21， and 62.27； and structural similarity scores of 0.8897， 0.9194， and 0.9101. Qualitative comparisons show that the proposed model produces clearer structural alignment and more accurate local appearance details than the baselines， particularly for complex unseen Chinese characters， creative glyphs， calligraphic forms， and difficult glyph-effect pairings. Ablation experiments further verify the contribution of each component. Removing channel recalibration causes stroke loss and structural deformation； removing perpendicular-component attention allows irrelevant reference patterns to contaminate the output； removing adaptive instance normalization-based skip connections introduces artifacts and weakens texture fidelity； and removing compositional contrastive learning leads to structural drift and inconsistent rendering. The complete model achieves an L1 loss of 0.1409， a Fréchet Inception Distance of 62.01， and a structural similarity score of 0.6855 in the ablation setting， outperforming all reduced variants. Latent-space analyses show that compositional contrastive learning produces more compact effect clusters， clearer glyph separation， and more consistent organization by character identity. In the user study， the proposed model receives the highest average scores for both Chinese and English samples. The scores range from 4.36 to 4.50 across the four evaluation dimensions for Chinese results and from 4.66 to 4.78 for English results， confirming improvements in character correctness， glyph similarity， effect similarity， and overall visual quality.ConclusionThe results support the central argument that controllable zero-shot artistic font generation benefits from independently modeling character identity， glyph structure， and visual effect while coordinating them through cross-domain feature aggregation， multi-scale style modulation， and compositional supervision. This design alleviates feature interference and erroneous coupling among heterogeneous attributes， strengthens the model’s sensitivity to attribute differences and composition relationships， and improves structural fidelity and style consistency for unseen combinations. The source code and ssafv2 dataset of this paper are publicly available on Science Data Bank athttps：//doi.org/10.57760/sciencedb.013hc.关键词：font generation;Diffusion models;artistic font dataset;zero-shot generation;compositional contrastive learning101|120|0更新时间：2026-10-19 -
摘要：ObjectiveExisting point cloud geometry compression methods generally suffer from two major limitations. On the one hand， most of them rely heavily on spatial-domain feature extraction and pay insufficient attention to frequency-domain information， so the potential of frequency-domain cues in describing global geometric structures has not been fully exploited. On the other hand， they find it difficult to strike a good balance between receptive field size and computational complexity. Convolution-based methods are often constrained by limited receptive fields， especially in hierarchical octree structures， where the effective receptive field decreases significantly as the resolution increases. Transformer-based methods can capture long-range dependencies， but they introduce considerable computational and memory costs， making them hard to apply to large-scale point cloud compression. To address these problems， this paper proposes a point cloud geometry compression method that integrates frequency-domain global features with spatial-domain local features， aiming to improve compression performance while keeping computational complexity low.MethodFirst， a fast Fourier transform is introduced into the point cloud compression network. The input node features are transformed into the frequency domain based on the Morton order of octree nodes， and the resulting frequency components are analyzed and reconstructed. Low-frequency components provide global structural information of point cloud regions， while high-frequency components preserve local geometric details and boundary variations. In this way， the frequency-domain information of the point cloud is exploited to construct global features， which effectively enlarges the receptive field and avoids the high complexity of attention mechanisms. Second， an octree-based convolution is designed. By leveraging the hierarchical structure and compact representation of the octree， this convolution efficiently extracts multi-scale local geometric features of the point cloud， overcomes the high computational and memory overhead of tensor-based sparse convolution， and avoids redundant computation on empty regions. Third， a channel-shift convolution is proposed. Using channel shift operations along the octree node sequence， this design further enlarges the receptive field without changing the size of the octree convolution kernel or introducing significant additional computational overhead， thereby effectively alleviating the problem that the receptive field in an octree becomes increasingly limited as the hierarchy grows deeper. Finally， a multi-stage occupancy prediction strategy is adopted to progressively estimate child-node occupancy probabilities， and the predicted probabilities are combined with context-based entropy coding to generate compact bitstreams. Within each octree level， the prediction is refined stage by stage， so that the network can make full use of the available geometric context and produce more reliable occupancy estimates. Through the collaboration of frequency-domain global modeling and spatial-domain local enhancement， the proposed framework achieves more accurate occupancy probability estimation and more efficient geometric compression.ResultExperiments were conducted on several public point cloud datasets， including 8iVFB （8i voxelized full bodies）， Owlii （Owlii dynamic human textured mesh sequence dataset）， and MVUB （Microsoft voxelized upper bodies）. Bits per point （bpp）， encoding and decoding time， and memory consumption were used as evaluation metrics. The proposed method was compared with the traditional geometry-based point cloud compression method G-PCC and the learning-based SparsePCGC method under both cross-dataset and within-dataset evaluation scenarios. Experimental results show that the proposed method achieves better compression performance than existing methods in both scenarios. In the cross-dataset scenario， the average bpp of the proposed method reaches 0.5625， which is reduced by 43.00% and 5.91% compared with G-PCC and SparsePCGC， respectively. Meanwhile， compared with SparsePCGC， the proposed method significantly reduces encoding and decoding time， achieving approximately 30% acceleration in compression efficiency， and reduces memory consumption by more than 70% on both the 8iVFB and Owlii datasets. The model size is also reduced， demonstrating the compactness and efficiency of the proposed architecture. Ablation studies further verify the effectiveness of each proposed component. The frequency-domain feature aggregation module improves global structural modeling， while the spatial feature aggregation module enhances local geometric representation. Combining these two modules achieves the best compression performance， confirming their complementary advantages. These results indicate that frequency-domain and spatial-domain features play complementary roles.ConclusionBy integrating frequency-domain global feature modeling with spatial-domain local structure enhancement， the proposed method effectively improves the representation capability of point cloud geometric structures and the accuracy of occupancy probability prediction. The designed octree convolution and channel-shift convolution expand the feature modeling capability while maintaining low computational complexity. Extensive experiments demonstrate that the proposed method achieves superior compression performance， lower computational cost， and reduced memory consumption compared with existing approaches. The proposed framework provides an effective solution for efficient point cloud geometry compression and has potential application value in large-scale three-dimensional data storage and transmission scenarios.关键词：point cloud geometry compression;octree convolution;frequency perception;context-based entropy coding;occupancy probability prediction78|121|0更新时间：2026-10-19 -
摘要：ObjectiveSynthetic aperture radar （SAR） is crucial for all-weather， all-day high-resolution Earth observation. However， accurate ship detection in SAR images faces two inherent physical and computational dilemmas. First， the coherent imaging mechanism generates severe multiplicative speckle noise， blurring target boundaries and inducing high false alarm rates in complex ocean clutter. Traditional methods and conventional convolutional neural networks （CNNs） struggle to filter this noise without losing structural information. Second， large artificial targets like ships often appear as discrete scattering centers， requiring models with global receptive fields to extract long-range structural dependencies. While Vision Transformers （ViTs） introduce global modeling， their quadratic computational complexity restricts deployment on resource-constrained edge computing platforms. To overcome these limitations， this study proposes the Frequency-Spatial Mamba Network （FSM-Net）， a novel lightweight architecture harmonizing global long-range modeling with robust speckle noise resistance.MethodsFSM-Net deeply couples frequency-domain priors with selective state space models （SSMs） through three core components. First， a learnable frequency-guided activation （LFGA） mechanism is introduced at the frontend to prevent non-linear speckle noise amplification. LFGA utilizes a two-dimensional fast Fourier transform （2D-FFT） to map features into the frequency domain. Using a dynamic gating mask， it adaptively filters out high-frequency speckle disturbances while retaining low-frequency structural components， providing a purified initial hidden state prior via an inverse FFT. Second， a Depthwise Separable Convolution Mamba （DSCMamba） block serves as the core detection backbone. It replaces standard convolutions with depthwise separable convolutions （DSConv） to reduce parameters and employs a two-dimensional multi-directional continuous scanning strategy. This flattens feature maps across four directions， allowing state transition equations to traverse physical topologies with linear time complexity to associate discrete scattering centers. Third， an adaptive cross-stream fusion （ACF） mechanism is devised to compensate for local spatial details lost during sequence flattening. ACF integrates a local sliding window parameter-free attention （LSWA） branch， evaluating an analytical energy function based on local neuron variance and mean to precisely activate high-frequency edges. High-dimensional features are then compressed via a structural re-parameterization technique， enabling zero-latency inference. Finally， a spatial-frequency adaptive feature fusion module merges the global Mamba topological features with the ACF local high-frequency edge features.ResultsExtensive comparative experiments on the SAR Ship Detection Dataset （SSDD） and High-Resolution SAR Images Dataset （HRSID） demonstrate that FSM-Net establishes a new state-of-the-art baseline. On SSDD， FSM-Net achieves 98.2% mean average precision （mAP） and 73.0% mAP50-95， surpassing the Vision Mamba （VMamba） baseline by 1.9% and 1.5%. On the complex HRSID dataset containing dense inshore clusters， FSM-Net yields 93.1% mAP and 68.4% mAP50-95， improving upon VMamba by 1.5% and 1.1%. Crucially， the model maintains excellent computational efficiency， requiring only 8.72 million （M） parameters and 21.35 giga floating-point operations （GFLOPs）， while reaching an inference speed of 151 frames per second （FPS）. Statistical evaluation reveals a remarkably low log-average miss rate （LAMR） of 11.8%， significantly outperforming Faster R-CNN （29.5%） and YOLOv11s （13.5%）. Feature activation heatmaps qualitatively confirm that FSM-Net effectively suppresses coastal noise and flawlessly connects discrete scattering points， mitigating target fragmentation and missed detections in dense scenarios.ConclusionFSM-Net effectively bridges the gap between frequency-domain speckle suppression priors and spatial-domain long-range evolution. By innovatively integrating the LFGA mechanism， a multi-directional scanning DSCMamba backbone， and the zero-latency ACF module， it completely resolves the dilemma between coherent speckle interference and the long-range association of discrete scattering centers. Adhering to a highly lightweight computational budget， FSM-Net provides a theoretically sound and empirically validated new paradigm for high-precision， real-time SAR target detection in complex maritime scenarios， exhibiting immense potential for future deployment on spaceborne and airborne edge computing platforms.关键词：SAR ship detection;selective state space model;frequency-spatial synergy;depthwise separable convolution;parameter-free attention78|132|0更新时间：2026-10-19 -
摘要：ObjectiveHyperspectral multi-class change detection aims to identify specific types of changes in land cover at different times in the same area， which has important application value in fields such as ecological monitoring， land use analysis， and resource surveys. In recent years， deep learning-based methods have been widely used for change detection in remote sensing images due to their powerful feature extraction capabilities， and have achieved good detection results when labels are sufficient.. However， the hyperspectral image annotation process relies heavily on manual intervention， and the cost of obtaining annotations is high. This makes it difficult for existing deep learning methods to achieve ideal detection results when the labels are limited， which is one of the important challenges currently facing hyperspectral change detection. Meanwhile， dual-temporal hyperspectral images are easily affected by factors such as lighting conditions， climate change， and differences in imaging environment during the imaging process， resulting in pseudo changes， which makes it difficult for the model to distinguish between real changed and unchanged areas accurately. Furthermore， for cross-scene change detection tasks， the common differences in domain distribution across different scenes further increase the difficulty of hyperspectral change detection under small sample conditions. How to make full use of the knowledge in existing scenes and reduce the distribution differences between different domains is one of the key problems that need to be solved in cross-scene small sample hyperspectral image multi-class change detection. To address the aforementioned issues， this paper proposes a bidirectional contrastive and spatial-frequency synergetic Transformer network （BCSFST） for cross-scene hyperspectral multi-class change detection in small samples. The BCSFST based on a Transformer network， effectively integrates few-shot learning， frequency-domain prior modeling and contrastive learning， constructing a unified learning mechanism from dual-temporal feature enhancement to cross-scene knowledge transfer.MethodFirstly， a frequency aware enhancement module is designed to map the features of the dual- temporal image to the frequency domain space through the Fourier transform， and extract the amplitude and phase information respectively. Subsequently， a dynamic correction mechanism is constructed using the amplitude and phase features to adaptively optimize the original spatial features， and the correction weights are adaptively adjusted according to the feature differences at different positions. Based on this， the corrected frequency domain information is mapped back to the spatial domain to further construct the frequency domain prior information. Subsequently， the frequency domain prior and spatial features are input into the Transformer， and the correlation between frequency information and spatial semantic features is established through a spatial-frequency domain collaborative attention mechanism. This enables collaborative modeling of the two types of complementary information， thereby enhancing the ability to discriminate changed features and suppressing pseudo change interference. Secondly， to address the significant distributional differences between the source and target domains in cross-scene few-shot learning， this paper proposes an adaptive cross-scene association contrastive learning strategy. This method generates adaptive neighbor weights by calculating the bidirectional consistency probability between samples from two domains， and mines potential sample matching relationships between different domains under the premise of limited target domain labeled information. Then， soft positive samples are constructed using the generated weights to establish cross-scene contrastive relationships， enhancing the feature compactness among similar samples. By using the bidirectional consistency constraints， the reliability of the feature association between the source and target domains is further improved. The feature distribution differences between different domains are reduced， and the cross-scene knowledge transfer capability and generalization performance of the model under the finite labeling conditions are enhanced. The synergistic effect of the two modules mentioned above enables the model to simultaneously mitigate the problems of pseudo change interference and domain shift from two aspects： frequency domain feature enhancement and cross-scene feature alignment， providing a more stable feature representation for the hyperspectral multi-class change detection in small labeled samples.ResultExperiments are conducted on three hyperspectral multi-class change detection datasets. Under the condition of using only five labeled samples for training in each class， the BCSFST proposed in this paper achieved overall accuracies of 90.80%， 83.67%， and 85.78% on the Irrigated Area， HangZhouWan， and China datasets， respectively， all of which are superior to existing state-of-the-art algorithms. This demonstrates that the proposed method can still uncover discriminative features across different classes using a very small number of labeled samples， exhibiting good cross-scene generalization ability. Based on the quantitative results， the visualization of experimental results can further demonstrate the effectiveness of the proposed method. Compared with other comparison algorithms， the proposed BCSFST can more accurately identify the boundaries of changed regions and effectively reduce the false detection rate of background regions. Meanwhile， ablation experiments further validated the effectiveness of the frequency-aware enhancement module and the adaptive cross-scene association contrastive learning strategy designed in this paper in improving model performance. After removing either of the two modules or the adaptive weights， the detection performance of the model decreased to varying degrees， indicating that the design of the two modules played a positive role in the overall model. In terms of computational time and the number of parameters， the BCSFST achieves both high detection accuracy and computational efficiency， demonstrating excellent overall performance. Finally， experiments with different numbers of labeled samples show that the proposed BCSFST achieves optimal performance in all cases， demonstrating that the proposed method has good adaptability and robustness for hyperspectral multi-class change detection tasks.ConclusionThe BCSFST model proposed in this paper effectively mitigates pseudo change interference and cross-scene domain shift problems through spectral transformation and contrastive learning. Experimental results show that the proposed method can fully combine the advantages of Transformer network and few-shot learning， and significantly improve the accuracy of hyperspectral multi-class change detection in the case of few labeled samples. In the future， this paper will further explore hyperspectral multi-class change detection methods in unsupervised scenes， reduce the model's dependence on labeled samples and improve its generalization ability and practical application value in complex remote sensing scenes.关键词：hyperspectral image;change detection;Transformer network;Fourier transform;contrastive learning;few-shot learning77|105|0更新时间：2026-10-19 -
摘要：ObjectiveColorectal polyp segmentation is an important task in colonoscopic image analysis because accurate delineation of polyp regions can provide useful support for lesion localization and subsequent computer-aided diagnosis. However， colonoscopic images often contain low-contrast lesions， specular reflections， mucosal texture interference， and irregular boundary shapes. These factors make the separation between polyp regions and surrounding tissues difficult， especially for small， flat， or weak-boundary polyps. Existing medical image segmentation methods have achieved considerable progress with convolutional neural networks， Transformer-based architectures， and boundary-enhancement strategies. Nevertheless， many of them still focus mainly on spatial-domain feature learning. During hierarchical encoding， downsampling， and skip-connection fusion， high-frequency structural cues related to polyp edges and fine textures may be weakened. In addition， common boundary-enhancement methods usually strengthen boundary responses or introduce auxiliary edge supervision， but they are less effective in modeling geometric continuity. To address these issues， this study proposes a colorectal polyp medical image segmentation network that integrates frequency-domain enhancement and geometry-guided boundary modeling.MethodThe proposed network follows an encoder-decoder architecture and introduces frequency-domain and geometric priors into different stages of feature representation and boundary reconstruction. In the encoding stage， a Spectral-enhanced Transformer Block is designed to improve the joint modeling of global structures and local details. Specifically， fast Fourier transform is embedded into the multi-head attention and feed-forward processes， allowing the network to enhance frequency-sensitive responses while preserving the semantic representation learned in the spatial domain. This design helps the encoder capture structural information that is closely related to weak polyp boundaries， subtle texture changes， and local contour variations. In the skip-connection stage， a Dual-band Feature Fusion Module is constructed to reduce the loss of boundary details during cross-layer feature transmission. The module decomposes encoder features into low-frequency and high-frequency components through a radial frequency-domain mask. The low-frequency branch is used to preserve region-level contour consistency， whereas the high-frequency branch emphasizes boundary transitions and fine-grained texture responses. These complementary components are then aligned and fused with decoder features， which enables the decoder to recover both global polyp shape and local boundary details more effectively. In the decoding stage， a Dual-stage Geometry-guided Module is introduced to incorporate signed distance and curvature information into boundary reconstruction. The module first estimates a coarse signed distance field and derives curvature-related cues from the distance representation. Narrow-band weighting and curvature-variation weighting are then used as geometric priors to guide the recovery of ambiguous boundary regions and locally curved contours. Furthermore， a Curvature-aware Geometric Supervision Loss is designed to jointly constrain the final segmentation output， boundary localization， signed distance field regression， and local curvature consistency. Through this joint optimization， the proposed network learns not only region-level overlap but also geometric consistency around complex polyp boundaries.ResultExperiments were conducted on five public colorectal polyp segmentation datasets， including Kvasir， ClinicDB， ColonDB， EndoScene， and ETIS. The proposed method was compared with 19 representative medical image segmentation methods， including general segmentation networks and polyp-oriented segmentation models. IoU and Dice were used as quantitative evaluation metrics because they can reflect region-level consistency and segmentation overlap. On the Kvasir dataset， the proposed method achieved an IoU of 85.78% and a Dice score of 91.04%. On the ClinicDB dataset， the IoU and Dice score reached 86.18% and 91.21%， respectively. On the ColonDB dataset， the proposed method achieved 70.62% IoU and 78.03% Dice， indicating its effectiveness under more challenging cross-dataset conditions. On the EndoScene dataset， the IoU and Dice score reached 83.37% and 89.65%， respectively. On the ETIS dataset， which contains many low-contrast and ambiguous polyp cases， the proposed method achieved 66.79% IoU and 75.76% Dice. The average IoU and average Dice over the five datasets were 78.55% and 85.14%， respectively， which were 0.83 and 1.37 percentage points higher than existing state-of-the-art methods. Qualitative visualization results also showed that our model produced fewer missing regions and fewer boundary errors in low-contrast， small-target， and complex-contour cases.ConclusionThe proposed frequency-domain enhanced and geometry-guided segmentation network improves colorectal polyp segmentation by combining spectral detail modeling， dual-band feature transmission， signed distance representation， and curvature-aware supervision. The experimental results demonstrate that the method achieves high segmentation accuracy on five public datasets and shows promising cross-dataset generalization. The method is particularly effective for weak-boundary， low-contrast， and complex-shaped polyps， where both high-frequency detail preservation and geometric boundary modeling are important for accurate segmentation.关键词：colonoscopic image;polyp segmentation;frequency-domain enhancement;geometric guidance;curvature awareness77|101|0更新时间：2026-10-19 -
摘要：ObjectiveZooplankton are important components of aquatic ecosystems， and changes in their abundance， species composition， and morphological characteristics can provide useful information for ecological observation and environmental assessment. Accurate analysis of zooplankton microscopic images is therefore important for species identification， individual counting， and quantitative morphological measurement. At present， microscopic examination still depends heavily on trained specialists and manual operation. This process is time-consuming， subjective， and difficult to apply efficiently when large numbers of samples must be processed. Instance segmentation can simultaneously locate individual organisms and provide their pixel-level masks， making it an important technical basis for automatic zooplankton analysis. However， reliable instance segmentation of zooplankton microscopic images remains challenging. Many zooplankton species have transparent or semi-transparent bodies， resulting in weak boundaries and low contrast with the surrounding medium. Suspended particles， biological debris， uneven illumination， imaging noise， and defocus blur may produce background patterns similar to biological structures. In addition， multiple organisms may be densely distributed， overlapped， or partially occluded， which can cause ambiguous boundaries， mask merging， and incorrect instance confusion. The limited availability of large-scale datasets with pixel-level instance annotations further restricts model training and quantitative evaluation in this field. Although the Segment Anything Model （SAM） has demonstrated strong general segmentation capability， directly applying it to zooplankton microscopic images remains difficult because of the feature distribution gap between natural-image pretraining and microscopic imaging， its dependence on externally provided prompts， and its limited instance discrimination in crowded scenes. To address these problems， this study constructs a zooplankton microscopic image instance segmentation dataset and develops an automatic SAM-based framework for accurate segmentation of individual zooplankton.MethodFirst， we construct a zooplankton microscopic image instance segmentation dataset （ZooMIIS5K）， a large-scale zooplankton microscopic image instance segmentation dataset containing 5，332 images from 48 zooplankton categories. Each visible organism is annotated with a fine-grained pixel-level instance mask， providing a reliable benchmark for model development and evaluation. Based on ZooMIIS5K， we propose a zooplankton microscopic image instance segmentation model （ZooMIIS-SAM）， an end-to-end automatic instance segmentation framework built upon SAM. The framework contains three principal components： a dual-domain fusion vision transformer （DDF-ViT）， a multi-scale fusion prompt module （MSFPM）， and a target-aware perception module （TAPM）. DDF-ViT is designed to reduce the feature distribution gap between the natural-image pretraining domain and the zooplankton microscopic-image domain. It combines spatial-domain adaptation and frequency-domain feature enhancement to strengthen foreground responses and supplement structural and boundary information. A spatial-gated low-rank adaptation branch generates task-related feature increments with limited additional parameters， while a frequency-domain feature enhancement module improves representations of weak boundaries and fine-grained structures. Bottleneck adapters are further introduced for feature alignment and contextual compensation. DDF-ViT is deployed only in selected encoder blocks to balance domain adaptation capability and parameter cost. MSFPM is designed to replace manually specified prompts with automatically generated visual prompts. It integrates multi-level encoder features through multi-scale strip convolution， channel recalibration， and cross-level fusion. The resulting feature is used to construct a feature pyramid and generate candidate regions. Region-of-interest features are then extracted and mapped into sparse soft prompt embeddings compatible with the SAM mask decoder. TAPM is introduced before mask decoding to strengthen instance-level discrimination. It uses candidate-region features to modulate global image representations， highlighting information related to the current target and reducing interference from surrounding backgrounds and neighboring instances. The enhanced image features and automatically generated visual prompts are jointly provided to the SAM mask decoder to predict individual instance masks.ResultExperiments on ZooMIIS5K show that ZooMIIS-SAM achieves the best performance among the compared convolutional neural network-based and visual foundation model-based methods， with a mask mean average precision （ -
摘要：ObjectiveIn digital human dance performance， camera movement quality directly impacts the artistic expression of choreography and the audience's viewing experience. Camera trajectory design serves multiple functions： guiding viewer attention toward key performers， conveying emotional atmosphere through framing and motion， and enhancing the visual impact of dance movements through coordinated camera dynamics. However， manually designing camera movements for a given dance sequence is labor-intensive and demands both cinematic expertise and deep understanding of choreographic structure. Consequently， automatic dance camera movement synthesis has emerged as an important research direction in cross-modal content generation. Existing music- and dance-driven camera synthesis methods， including DanceCamera3D， DanceCamAnimator， and TemMEGA， focus on generating camera trajectories aligned with basic rhythm and motion patterns of the input performance. While these methods have advanced automation， they share a critical limitation： they do not support explicit user control over the generated results. In professional filmmaking workflows， practitioners require fine-grained editorial control along two independent dimensions. The first is style controllability—the capacity to specify the overall artistic style of camera work， such as K-pop， Choreography， or Jazz. The second is trajectory controllability—the ability to input camera parameters at specific keyframes， including position， orientation， and field of view， to guide the generated trajectory toward a desired creative intent. No existing dance camera synthesis method has explicitly modeled either form of user control， leaving a significant gap between research prototypes and practical production tools. This work addresses the problem of controllable multi-style dance camera movement synthesis， targeting dual-dimensional user control over both the artistic style and the spatial trajectory of synthesized camera motion.MethodWe propose CtrlDanceCam， a framework that， for the first time in dance camera synthesis， enables users to interactively specify style labels and keyframe constraints as explicit conditioning signals. The framework is built upon a controllable camera synthesis network backbone， ControlCamNet， comprising two tightly integrated modules： a keyframe prediction module （KPM） and a controllable trajectory synthesis module （CTSM）. Given an input music sequence and a dance motion sequence with a user-specified style label， the KPM predicts a binary keyframe sequence identifying shot boundaries. KPM employs a Transformer encoder augmented with a learnable style encoder that maps discrete style labels into continuous embeddings. These style-conditioned embeddings， fused with music and dance features， enable prediction of keyframe distributions characteristic of the target style—for instance， sparse keyframes for jazz-style long takes versus dense keyframes for locking-style rapid cuts. The module is trained with a weighted binary cross-entropy loss to address class imbalance. The CTSM synthesizes the complete camera motion sequence using predicted or user-specified keyframes as anchor points. For each shot defined by consecutive keyframes， CTSM predicts camera parameters at boundary frames and learns a style-conditioned， monotonically increasing interpolation function rho（t；s） governing the intra-shot motion profile. Non-keyframe camera parameters are obtained through rho（t；s）-weighted interpolation between boundary predictions. This design ensures smooth intra-shot transitions while allowing style-dependent intra-shot motion profiles. When users provide hard constraints by specifying camera parameters at particular keyframe indices， CTSM substitutes these values for model predictions， enabling precise trajectory-level editing. To enhance style-conditioning responsiveness and synthesis quality， we design a style-aware adversarial training （SAAT） strategy with three complementary loss components： a controllable style contrastive loss that compels the generator to produce outputs judged as realistic under the correct style label while penalized under incorrect labels， a style classification loss with label smoothing for style-consistent generation， and a base Wasserstein adversarial loss with gradient penalty. These losses synergistically prevent inter-style confusion， ensure intra-style consistency， and promote overall realism. To support style-controllable training， we augment the dance-camera-music （DCM） dataset with fine-grained style annotations. Through multiple rounds of discussion with professional animators， and referencing the FineDance style taxonomy for semantic alignment with upstream music-to-dance generation， each sample is annotated with one of eight styles： K-pop， Choreography， Traditional Chinese Dance， Hip-hop， Popping， Locking， Urban， and Jazz. Statistical analysis across these styles reveals quantifiable differences in keyframe density （ranging from 2.640% for Jazz to 31.484% for Locking）， shot length， field of view （FOV） variation， and camera velocity， confirming the validity of the labeling scheme and motivating the need for style-adaptive keyframe prediction.ResultComprehensive experiments on the DCM dataset—the only publicly available dance-camera paired benchmark—demonstrate the effectiveness of CtrlDanceCam. In terms of generation quality， the full model achieves Fréchet inception distances of 3.022 （FIDk） and 0.251 （FIDs）. The average Euclidean distances are 3.101 （Distk） and 1.355 （Dists）. For dancer tracking fidelity， the dancer missing rate （DMR） is 0.0019 and the limbs capture difference （LCD） is 0.140. These results outperform DanceCamera3D， DanceCamAnimator， and TemMEGA on five of the six metrics， while remaining competitive on Distk. Ablation studies confirm the contribution of each architectural component. Removing SAAT degrades FIDk to 3.512 and FIDs to 0.345. Removing the controllable style contrastive loss raises FIDs to 0.317， confirming its contribution to style discrimination. Replacing the KPM-predicted keyframes with uniform sampling drives FIDk to 3.742 and FIDs to 0.366， while stripping style conditioning from KPM disproportionately harms FIDs to 0.331， confirming that style-conditioned keyframe prediction forms the upstream basis of the framework's style controllability. Substituting the learnable interpolation function rho（t；s） with fixed linear interpolation causes FIDs to rise sharply from 0.251 to 0.346—a substantially larger degradation than the corresponding FIDk increase from 3.022 to 3.351—revealing that rho（t；s） contributes an independent， intra-shot layer of style expression beyond KPM's inter-shot style differentiation. Keyframe density sensitivity analysis further shows that the KPM adaptive strategy achieves the optimal balance across all metrics， outperforming every fixed-density configuration from 3% to 30%. Qualitative visualizations confirm that distinct style labels yield clearly differentiated camera trajectories， and that sparse user-specified keyframe constraints effectively adjust the generated camera motion toward the desired result， jointly validating the dual-dimensional controllable synthesis.ConclusionCtrlDanceCam introduces a novel paradigm for dance camera synthesis by enabling simultaneous trajectory-level and style-level user control. Its modular architecture decomposes controllable synthesis into interpretable sub-problems of style-conditioned keyframe prediction and style-conditioned intra-shot interpolation. The design facilitates future extensions to richer control modalities including path constraints， velocity constraints， and fine-grained shot composition parameters. The style labeling scheme established for DCM provides a foundation for future style-aware camera synthesis research. CtrlDanceCam demonstrates the feasibility of integrating interactive user control into cross-modal generative frameworks， offering a practical technical pathway for deploying automatic camera synthesis in professional creative workflows such as virtual production， digital performance， and immersive media creation. Code is available at ScienceDB：https：//www.scidb.cn/s/vUr2Av关键词：dance camera movement synthesis;controllable synthesis;style-aware;adversarial training;keyframe prediction;digital human animation;cross-modal generation86|107|0更新时间：2026-10-08 -
Semantic dynamic boundary enhancement for referring remote sensing image segmentation： SDM-SAM2 AI导读
摘要：ObjectiveReferring remote sensing image segmentation （RRSIS） identifies and segments the ground object described by a natural-language expression. The target category and spatial extent are jointly determined by the image and text， requiring fine-grained alignment among language semantics， object appearance， spatial relations， and pixel boundaries. Large scale variations， dense objects of the same category， complex backgrounds， and weak boundaries make this alignment difficult in remote sensing scenes. Although the Segment Anything Model 2 （SAM2） provides strong generic visual representations and mask decoding， its direct adaptation to RRSIS has three limitations： language guidance may weaken across hierarchical visual stages， the final decoder query may underuse complementary intermediate states， and region-overlap losses do not explicitly constrain boundaries. This study develops a semantic dynamic boundary enhancement framework， termed SDM-SAM2， to address these limitations in a unified architecture.MethodThe framework is built on a SAM2-based RRSIS baseline and contains three complementary components. The image and expression are jointly encoded by Bidirectional Encoder representation fromImage Transformers 3 （BEiT-3）， which produces a global classification-token representation， token-level text features， visual tokens， and a coarse mask-related representation. The Hiera backbone extracts hierarchical image features， while language information is introduced repeatedly into the visual hierarchy. The visual tokens and mask-related representation generate a coarse mask prompt， which the prompt encoder converts into dense and sparse embeddings. The global multimodal representation supplies sentence-level semantic guidance， whereas the token-level text sequence preserves word and relational information. The mask decoder receives the final image embedding， high-resolution image features， prompt embeddings， and text features， and progressively updates its output-token queries through text-image interaction. Thus， the global classification token is a semantic prompt， the token-level text features are language context， and the decoder queries are learnable states updated to predict the mask； these tensors cooperate but are not interchangeable. The first component is stage-aware text memory. At each Hiera stage， it retains the full token-level text state and its mask-aware pooled summary. The pooled current text， historical text summaries， and visual context determine the relevance of each historical state， and softmax produces normalized memory weights. These weights are applied to the complete historical text sequences， preserving word-level structure. A learned gate， estimated from the pooled current state， retrieved memory， and visual context， then combines the retrieved and current text sequences. Its scalar value is broadcast over token and channel dimensions， enabling stage-dependent retention of historical language information. The second component is task-adaptive multi-layer query fusion. The initial output-token state and intermediate query states from successive decoder layers form a query set. A lightweight network maps each state to a scalar score， and normalized scores weight the initial and intermediate states to produce the final query. The weights are learned globally with the segmentation objective and represent task-level contributions of decoder depths， rather than sample-specific attention values. Thus， early prompt and localization information can complement the semantic and contour cues in deeper queries. The third component is a text-weighted Sobel boundary loss. Horizontal and vertical Sobel operators extract differentiable gradient-magnitude maps from the predicted probability and ground-truth mask. The sentence-level representation is projected to a sample-level boundary weight and spatially repeated to the mask resolution. This weight modulates the mean squared error between the two gradient maps， relating boundary supervision to the referring expression while constraining boundary discontinuity and contour displacement. The total objective combines cross-entropy， Dice， and boundary losses. Accordingly， text memory maintains encoder guidance， query fusion integrates decoder states， and Sobel supervision constrains the final boundary. Experiments use RefSegRS and RRSIS-D， two public datasets with different sample sizes， resolutions， category compositions， and expression distributions. RefSegRS contains 4，420 image-expression-mask triplets covering 14 categories at 512 × 512 pixels， whereas RRSIS-D contains 17，402 triplets covering 20 categories at 800 × 800 pixels. Metrics include mean intersection over union （mIoU）， overall intersection over union （oIoU）， and the proportion of samples whose intersection over union exceeds threshold K （Pr@K）. Ablations evaluate all individual and combined components， while an operator comparison examines Sobel， Laplacian， and Prewitt boundaries under the same setting.ResultOn the RefSegRS test set， SDM-SAM2 obtains an mIoU of 78.05% and an oIoU of 84.16%. Compared with the SAM2-based RS2-SAM2 baseline， it improves mIoU by 4.15 percentage points and oIoU by 3.29 percentage points. The improvements at the stricter Pr@0.7， Pr@0.8， and Pr@0.9 thresholds are 6.05， 9.08， and 15.85 percentage points， respectively， indicating that the proposed method increases not only average overlap but also the number of high-quality predictions. On the RRSIS-D test set， the proposed model reaches an mIoU of 68.49% and an oIoU of 78.59%. Relative to RS2-SAM2， mIoU increases by 1.77 percentage points， while Pr@0.5， Pr@0.6， and Pr@0.7 increase by 2.39， 2.26， and 2.10 percentage points， respectively. The oIoU and Pr@0.9 values are slightly lower than those of the baseline， showing that the benefit is not uniform across every metric but is concentrated in mean region quality and the low-to-high intersection-over-union threshold range. The ablations confirm different component roles. Among single-module variants， stage-aware text memory gives the highest mIoU on both datasets： 67.89% on RRSIS-D and 78.47% on RefSegRS. The complete configuration is most balanced on RRSIS-D， whereas RefSegRS combinations are not strictly additive， indicating dataset-dependent marginal and joint effects. With the same boundary-loss setting， Sobel reaches 67.72% mIoU on RRSIS-D， exceeding Laplacian at 66.83% and Prewitt at 67.49%. Qualitative results show fewer incomplete regions， disconnected boundaries， and spatial deviations. The full model has 961.05 million parameters， 26.24 million or approximately 2.8% more than the 934.81-million-parameter baseline.ConclusionSDM-SAM2 maintains stage-dependent language guidance， exploits complementary decoder queries， and introduces expression-conditioned boundary supervision. Results on both datasets show consistent mIoU gains and improvements in most threshold-based measures with a limited parameter increase. Cross-dataset ablations further reveal that module combinations can have different marginal effects under different data distributions. Overall， the framework improves semantic localization and boundary delineation while retaining the general SAM2 mask-decoding structure. The codes are available at：https：//github.com/3038794185-cloud/SDM-SAM2.关键词：referring remote sensing image segmentation;Segment Anything Model 2 （SAM2）;vision-language fusion;stage-aware text memory;query fusion;boundary loss71|68|0更新时间：2026-10-08 -
摘要：ObjectiveAccurate segmentation of individual teeth and adjacent gingiva is a prerequisite for reliable digital orthodontic diagnosis， virtual treatment planning， clear aligner fabrication， prosthodontic design， and implant-related assessment. In daily clinical practice， a visually interpretable segmentation result can also shorten the communication loop between dentists and patients by converting complex intraoral geometry into intuitive tooth-level models. Nevertheless， dental scans collected from patients who require orthodontic intervention or implant restoration frequently contain missing teeth， severe crowding， overlapped crowns， extraction gaps， irregular gingival margins， orthodontic attachments， and partially visible tooth surfaces. These factors make instance separation substantially more difficult than ordinary semantic segmentation. Existing geometric approaches are usually sensitive to curvature thresholds， manually selected seed points， and boundary line assumptions， while many deep learning approaches either lose three-dimensional spatial cues during 2D projection or suffer from instance confusion and high computational cost when the whole jaw point cloud is processed directly. Therefore， this study aims to develop a robust， clinically oriented segmentation framework that can maintain the efficiency of 2D detection while preserving the geometric accuracy of 3D point cloud learning， with particular attention to missing-tooth and overlapping-tooth conditions.MethodTo address the above challenges， this paper proposes HaTransTooth， a multi-dimensional hybrid deep learning framework based on a coarse-to-fine workflow. First， the original 3D intraoral scan is transformed into a 1024 × 1024 depth image through parallel projection along the estimated normal direction of the dental arch occlusal plane. During this projection process， the camera pose， coordinate correspondence， and depth information are retained so that each detected 2D region can be accurately mapped back to the original 3D space. An improved YOLOv8 detector is then retrained for dental depth images to perform rapid tooth localization and preliminary numbering. Considering the dense， sparse， and partially missing distribution patterns of teeth， the anchor settings and training samples are adjusted， and data augmentation strategies such as random rotation and Mosaic augmentation are adopted to enhance robustness in complex oral scenes. Second， the predicted 2D bounding boxes are expanded by a small spatial margin and back-projected to generate 3D regions of interest（3D ROI）. Each 3D ROI contains one target tooth， a limited amount of surrounding gingiva， and small parts of adjacent teeth， which reduces the ambiguity caused by full-jaw instance separation while still preserving boundary context. Before fine segmentation， point cloud sampling， centroid normalization， unit-sphere scaling， and statistical outlier removal are applied to standardize the input distribution and suppress scanning noise. Third， each ROI is segmented by an improved Point Cloud Transformer network（PCT）. A hierarchical adaptive attention mechanism is introduced into the PCT backbone to combine local geometric details with global semantic dependencies. Local attention focuses on neighboring points near the tooth-gingiva junction， whereas global attention captures the overall shape pattern of the tooth instance； a learnable gate dynamically balances the two feature streams. To further handle class imbalance between tooth and gingival points and to improve the learning of hard boundary samples， the model is optimized with a combined Focal Loss and Dice Loss， together with a gradient accumulation strategy under limited GPU memory. The framework is trained and evaluated on 2，000 real clinical cases collected by intraoral scanners， each containing both upper and lower jaws. The dataset includes approximately normal dentitions， pre-orthodontic crowding and overlap， post-extraction and missing-tooth cases， aligned cases with spaces， orthodontic attachments， implants， wisdom teeth， and mixed dentition. All tooth and gingiva labels are manually annotated with CloudCompare and verified by professional dentists. The data are divided into 1，400 training cases， 300 validation cases， and 300 testing cases， and the experiments are conducted on a single NVIDIA GeForce RTX 3090 GPU.ResultExperiments on the self-collected clinical dataset demonstrate that HaTransTooth achieves competitive and stable performance when compared with seven representative segmentation methods.Although its mean accuracy（mACC） reaches 96.74%， ranking a close second to TSegNet， the proposed method shows a clearer advantage in metrics that better reflect region overlap and boundary consistency. Specifically， it achieves a mean Intersection over Union（mIoU） of 95.26% and a Dice Similarity Coefficient（DSC） of 97.50%， exceeding the second-best method by 4.86% and 2.41%， respectively. Additional evaluation also confirms the clinical usefulness of the framework， with an Average Symmetric Surface Distance（ASSD） of 0.0074， a recall of 98.10%， and an F1 score of 97.41%. On the difficult missing-tooth subset， the method obtains 97.19% mAcc， 95.41% mIoU， and 97.49% DSC； on the crowded and overlapping-tooth subset， it still reaches 96.51% mAcc， 94.39% mIoU， and 97.18% DSC. These results indicate that the method can maintain high segmentation quality in cases where adjacent crowns are in close contact， tooth numbers are incomplete， or gingival boundaries are unclear. Cross-validation on the public Teeth3DS dataset further verifies its generalization ability. Without retraining on Teeth3DS， the proposed framework achieves 97.21% mAcc， 96.13% mIoU， and 97.91% DSC， outperforming the compared approaches in the main segmentation metrics. Ablation experiments show that the 2D projection detection and 3D ROI extraction module provides the largest performance gain by reducing instance confusion， while the hierarchical adaptive attention module and the combined loss with gradient accumulation further improve boundary discrimination and training stability. Visual comparisons also reveal that HaTransTooth produces more complete tooth surfaces， fewer gingival misclassifications， and clearer inter-tooth boundaries than conventional full-jaw point cloud segmentation methods.ConclusionThe proposed HaTransTooth framework provides an effective solution for automatic dental instance segmentation under complex intraoral conditions. By transforming the initial instance separation problem into efficient 2D detection and then performing attention-enhanced 3D fine segmentation within localized ROIs， the framework integrates the strengths of image-based detection and point-cloud-based geometric modeling. It improves segmentation accuracy at ambiguous tooth-gingiva boundaries and remains robust in missing-tooth， crowded， overlapping， and attachment-bearing cases. The results on both a large real clinical dataset and a public benchmark suggest that the framework has strong potential for supporting digital orthodontic diagnosis， aligner design， prosthodontic planning， and other computer-aided dental applications.关键词：oral point cloud;segmentation of instances;missing teeth;dimension transformation;object detection;hierarchical adaptive attention mechanism68|90|0更新时间：2026-10-08
- Postal code：100190
- Tel：010-58887035/58887030/58887418 Email：jig@aircas.ac.cn
- Technical support is provided by Beijing Founder electronics co., LTD 京ICP备20021838号-20
京公网安备11010802024621 - It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
- Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
0