论文检索

Probabilistic Methods · Spectral Methods

Jiashun Jin, Jingming Wang

The degree-corrected block model (DCBM), latent space model (LSM), and $\beta$-model are all popular network models. We combine their modeling ideas and propose the logit-DCBM as a new model. Similar as the $\beta$-model and LSM, the logit-DCBM contains nonlinear factors, where fitting the parameters is a challenging open problem. We resolve this problem by introducing a cancellation trick. We also propose R-SCORE as a recursive community detection algorithm, where in each iteration, we first use the idea above to update our parameter estimation, and then use the results to remove the nonlinear factors in the logit-DCBM so the renormalized model approximately satisfies a low-rank model, just like the DCBM. Our numerical study suggests that R-SCORE significantly improves over existing spectral approaches in many cases. Also, theoretically, we show that the Hamming error rate of R-SCORE is faster than that of SCORE in a specific sparse region, and is at least as fast outside this region.

Applications · Physics

Tiexin Qin, Mengxu ZHU, Chunyang Li, Terry Lyons, Hong Yan, Haoliang Li

Understanding protein dynamics are essential for deciphering protein functional mechanisms and developing molecular therapies. However, the complex high-dimensional dynamics and interatomic interactions of biological processes pose significant challenge for existing computational techniques. In this paper, we approach this problem for the first time by introducing Deep Signature, a novel computationally tractable framework that characterizes complex dynamics and interatomic interactions based on their evolving trajectories. Specifically, our approach incorporates soft spectral clustering that locally aggregates cooperative dynamics to reduce the size of the system, as well as signature transform that collects iterated integrals to provide a global characterization of the non-smooth interactive dynamics. Theoretical analysis demonstrates that Deep Signature exhibits several desirable properties, including invariance to translation, near invariance to rotation, equivariance to permutation of atomic coordinates, and invariance under time reparameterization. Furthermore, experimental results on three benchmarks of biological processes verify that our approach can achieve superior performance compared to baseline methods.

Applications · Computer Vision

Haijin Zeng, Benteng Sun, Yongyong Chen, Jingyong Su, Yong Xu

Spectral Compressive Imaging (SCI) reconstruction is inherently ill-posed, offering multiple plausible solutions from a single observation. Traditional deterministic methods typically struggle to effectively recover high-frequency details. Although diffusion models offer promising solutions to this challenge, their application is constrained by the limited training data and high computational demands associated with multispectral images (MSIs), complicating direct training. To address these issues, we propose a novel Predict-and-unmixing-driven-Subspace-Refine framework (PSR-SCI). This framework begins with a cost-effective predictor that produces an initial, rough estimate of the MSI. Subsequently, we introduce a unmixing-driven reversible spectral embedding module that decomposes the MSI into subspace images and spectral coefficients. This decomposition facilitates the adaptation of pre-trained RGB diffusion models and focuses refinement processes on high-frequency details, thereby enabling efficient diffusion generation with minimal MSI data. Additionally, we design a high-dimensional guidance mechanism with imaging consistency to enhance the model's efficacy. The refined subspace image is then reconstructed back into an MSI using the reversible embedding, yielding the final MSI with full spectral resolution. Experimental results on the standard KAIST and zero-shot datasets NTIRE, ICVL, and Harvard show that PSR-SCI enhances visual quality and delivers PSNR and SSIM metrics comparable to existing diffusion, transformer, and deep unfolding techniques. This framework provides a robust alternative to traditional deterministic SCI reconstruction methods. Code and models are available at [https://github.com/SMARK2022/PSR-SCI](https://github.com/SMARK2022/PSR-SCI).

Social Aspects · Trustworthy Machine Learning

Gaojie Jin, Sihao Wu, Jiaxu Liu, Tianjin Huang, Ronghui Mu

Recent research has highlighted a critical issue known as ``robust fairness", where robust accuracy varies significantly across different classes, undermining the reliability of deep neural networks (DNNs). A common approach to address this has been to dynamically reweight classes during training, giving more weight to those with lower empirical robust performance. However, we find there is a divergence of class-wise robust performance between training set and testing set, which limits the effectiveness of these explicit reweighting methods, indicating the need for a principled alternative.In this work, we derive a robust generalization bound for the worst-class robust error within the PAC-Bayesian framework, accounting for unknown data distributions. Our analysis shows that the worst-class robust error is influenced by two main factors: the spectral norm of the empirical robust confusion matrix and the information embedded in the model and training set. While the latter has been extensively studied, we propose a novel regularization technique targeting the spectral norm of the robust confusion matrix to improve worst-class robust accuracy and enhance robust fairness.We validate our approach through comprehensive experiments on various datasets and models, demonstrating its effectiveness in enhancing robust fairness.

David R. Palmer 0001, Albert Chern, Justin Solomon 0001

Directional fields, including unit vector, line, and cross fields, are essential tools in the geometry processing toolkit. The topology of directional fields is characterized by their singularities. While singularities play an important role in downstream applications such as meshing, existing methods for computing directional fields either require them to be specified in advance, ignore them altogether, or treat them as zeros of a relaxed field. While fields are ill-defined at their singularities, the graphs of directional fields with singularities are well-defined surfaces in a circle bundle. By lifting optimization of fields to optimization over their graphs, we can exploit a natural convex relaxation to a minimal section problem over the space of currents in the bundle. This relaxation treats singularities as first-class citizens, expressing the relationship between fields and singularities as an explicit boundary condition. As curvature frustrates finite element discretization of the bundle, we devise a hybrid spectral method for representing and optimizing minimal sections. Our method supports field optimization on both flat and curved domains and enables more precise control over singularity placement.

Yilin Liu, Yunkui Pang, Jiang Li, Yong Chen, Pew-Thian Yap

Untrained networks inspired by deep image prior have shown promising capabilities in recovering a high-quality image from noisy or partial measurements, without requiring training data. Their success has been widely attributed to the spectral bias acting as an implicit regularization induced by suitable network architectures. However, applications of such network-based priors often entail superfluous architectural decisions, overfitting risks, and slow optimization, all of which hinder their practicality. In this work, we propose efficient, architecture-agnostic methods for a more direct frequency control over the network priors: 1) constraining the bandwidth of the white-noise input, 2) controlling the bandwidth of the interpolation-based upsamplers, and 3) regularizing the Lipschitz constants of the layers. We show that even with just one extra line of code, the overfitting issues in underperforming architectures can be alleviated such that their performance gaps with the high-performing counterparts can be largely closed despite their distinct configurations, mitigating the need for architecture tuning. This then makes it possible to employ a more compact model to achieve similar or superior performance to larger models with greater efficiency. Our regularized network priors compare favorably with current supervised and self-supervised methods on MRI reconstruction and image inpainting tasks, serving as a stronger zero-shot baseline reconstructor. Our code will be made publicly available.

Jian-Li Wang, Xi-Le Zhao

Recently, the transform-based low-rank tensor factorization (t-LRTF) has emerged as a promising tool for multi-dimensional data recovery. However, the discrete transforms along the third (i.e., temporal/spectral) dimension are dominating in existing t-LRTF methods, which hinders their performance in addressing temporal/spectral degeneration scenarios, e.g., video frame interpolation and multispectral image (MSI) spectral super-resolution. To break this barrier, we propose a novel Functional Transform-based Low-Rank Tensor Factorization (FLRTF), where the learnable functional transform is expressed by the implicit neural representation with positional encodings. The continuity brought by this function allows FLRTF to capture the smoothness of data in the third dimension, which will benefit the recovery of temporal/spectral degeneration problems. To examine the effectiveness of FLRTF, we establish a general FLRTF-based multi-dimensional data recovery model. Experimental results, including video frame interpolation/extrapolation, MSI band interpolation, and MSI spectral super-resolution tasks, substantiate that FLRTF has superior performance as compared with representative data recovery methods.

Lintao Peng, Siyu Xie, Liheng Bian

Recently, learning-based Hyperspectral image (HSI) reconstruction methods have demonstrated promising performance and dominated the mainstream research direction. However, existing learning-based methods still have two issues. 1) Unable to consider both the spatial sparsity and the inter-spectral similarity prior of HSI. 2) Treat all regions equally, ignoring that texture-rich regions and edge regions are more difficult to reconstruct than smooth regions. To address these issues, we propose an uncertainty-driven HSI reconstruction method termed Specformer. Specifically, we first introduce a frequency-wise self-attention (FWSA) and combine it with spatial-wise local-window self-attention (LWSA) with a parallel design to form a Spatial-Frequency (SF) block. LWSA can guide the network to focus on the regions with dense spectral information, and FWSA can capture the inter-spectral similarity. Parallel design helps the network to model cross-window connections, expand its receptive fields while maintaining linear complexity. We use SF-block as the main building block in a multi-scale U-shape network to form our Specformer. In addition, we introduce an uncertainty-driven self-adaptive loss function, which can reinforce the network's attention to challenging regions with rich textures and edges. Comprehensive experiments show that our Specformer significantly outperforms state-of-the-art methods on simulation and real HSI datasets while requiring cheaper computational and memory costs. The code will be publicly available.

JiaHua Xiao, Yang Liu, Xing Wei

Properly spectral modeling within hyperspectral image (HSI) is critical yet highly challenging for HSI denoising. In contrast to existing methods that model long-range spectral dependencies with a huge cost and directly explore spatial-spectral information without region discrimination, we introduce RAS2S—a simple yet effective sequence-to-sequence (Seq2Seq) learning framework for better HSI denoising. RAS2S treats HSI denoising as a Seq2Seq translation problem, which converts the noisy spectral sequence to its clean ones in an autoregressive fashion. In addition, spatial-spectral information exploration without region discrimination contradicts the intrinsic spatial-spectral diversity of HSIs, leading to negative interference from spatial-spectral unrelated regions. Thus we propose a novel spatial-spectral region-aware module to distinctively perceive the semantic regions with different spatial-spectral representations, maximizing the spectral modeling potential of Seq2Seq learning. With such an improved Seq2Seq learning paradigm, RAS2S not only shows huge potential in capturing long-range spectral dependencies, but also maintains the flexibility to handle HSIs with arbitrary spectral numbers. Extensive experiments demonstrate that RAS2S outperforms existing state-of-the-art methods quantitatively and qualitatively with a minimal model size, merely 0.08M.

Ekaterina Grishina, Mikhail Gorbunov, Maxim Rakhuba

Controlling the spectral norm of convolution layers has been shown to enhance generalization and robustness in CNNs, as well as training stability and the quality of generated samples in GANs. Existing methods for computing singular values either result in loose bounds or lack scalability with input and kernel sizes. In this paper, we obtain a new upper bound that is independent of input size, differentiable and can be efficiently computed during training. Through experiments, we demonstrate how this new bound can be used to improve the performance of convolutional architectures.

Satoshi Ikehata, Yuta Asano

In this paper, we present a groundbreaking spectrally multiplexed photometric stereo approach for recovering surface normals of dynamic surfaces without the need for calibrated lighting or sensors, a notable advancement in the field traditionally hindered by stringent prerequisites and spectral ambiguity. By embracing spectral ambiguity as an advantage, our technique enables the generation of training data without specialized multispectral rendering frameworks. We introduce a unique, physics-free network architecture, SpectraM-PS, that effectively processes multiplexed images to determine surface normals across a wide range of conditions and material types, without relying on specific physically-based knowledge. Additionally, we establish the first benchmark dataset, SpectraM14, for spectrally multiplexed photometric stereo, facilitating comprehensive evaluations against existing calibrated methods. Our contributions significantly enhance the capabilities for dynamic surface recovery, particularly in uncalibrated setups, marking a pivotal step forward in the application of photometric stereo across various domains.

Amir Mehrpanah, Erik Englesson, Hossein Azizpour

Understanding the behavior of deep networks is crucial to increase our confidence in their results. Despite an extensive body of work for explaining their predictions, researchers have faced with reliability issues, which can be attributed to insufficient formalism. In our research, we adopt novel probabilistic and spectral perspectives to formally analyze explanation methods. Our study reveals a pervasive spectral bias stemming from the use of gradient, and sheds light on some common design choices that have been discovered experimentally, in particular, the use of squared gradient and input perturbation. We further characterize how the choice of perturbation hyperparameters in explanation methods, such as SmoothGrad, can lead to inconsistent explanations and introduce two remedies based on our proposed formalism: (i) a mechanism to determine a standard perturbation scale, and (ii) an aggregation method which we call SpectralLens. Finally, we substantiate our theoretical results through quantitative evaluations.

Haijin Zeng, Yuxi Liu, Yongyong Chen, Youfa Liu, Chong Peng, Jingyong Su

Hyperspectral image (HSI) reconstruction is vital for recovering spatial-spectral information from compressed measurements in coded aperture snapshot spectral imaging (CASSI) systems. Despite the effectiveness of end-to-end and deep unfolding methods, their reliance on substantial training data poses challenges, notably the scarcity of labeled HSIs. Existing approaches often train on limited datasets, such as KAIST and CAVE, leading to biased models with poor generalization capabilities. Addressing these challenges, we propose a universal Self-Supervised Adapter for Hyperspectral Snapshot Compressive Imaging (SAH-SCI). Unlike full fine-tuning or linear probing, SAH-SCI enhances model generalization by training a lightweight adapter while preserving the original model's parameters. We propose a novel approach that combines spectral and spatial adaptation to enhance an image model's capacity for spatial-spectral reasoning. Additionally, we introduce a customized adapter self-supervised loss function that captures the consistency, group invariance and image uncertainty of CASSI imaging. This approach effectively reduces the solution space for ill-posed HSI reconstruction. Experimental results demonstrate SAH's superiority over previous methods with fewer parameters, offering simplicity and adaptability to any end-to-end or unfolding methods. Our approach paves the way for leveraging more robust image foundation models in future hyperspectral imaging tasks.

Zongliang Wu, Ruiying Lu, Ying Fu, Xin Yuan

Snapshot compressive spectral imaging reconstruction aims to reconstruct three-dimensional spatial-spectral images from a single-shot two-dimensional compressed measurement. Existing state-of-the-art methods are mostly based on deep unfolding structures but have intrinsic performance bottlenecks: i) the ill-posed problem of dealing with heavily degraded measurement, and ii) the regression loss-based reconstruction models being prone to recover images with few details. In this paper, we introduce a generative model, namely the latent diffusion model (LDM), to generate degradation-free prior to enhance the regression-based deep unfolding method by a two-stage training procedure. Furthermore, we propose a Trident Transformer (TT), which extracts correlations among prior knowledge, spatial, and spectral features, to integrate knowledge priors in deep unfolding denoiser, and guide the reconstruction for compensating high-quality spectral signal details. To our knowledge, this is the first approach to integrate physics-driven deep unfolding with generative LDM in the context of CASSI reconstruction. Numeric and visual comparisons on synthetic and real-world datasets illustrate the superiority of our proposed method in both reconstruction quality and computational efficiency. Code will be released.

Jina Kim, Jihoo Lee, Jewon Kang

Neural representation for video (NeRV), which employs a neural network to parameterize video signals, introduces a novel methodology in video representations. However, existing NeRV-based methods have difficulty in capturing fine spatial details and motion patterns due to spectral bias, in which a neural network learns high-frequency (HF) components at a slower rate than low-frequency (LF) components. In this paper, we propose spectra-preserving NeRV (SNeRV) as a novel approach to enhance implicit video representations by efficiently handling various frequency components. SNeRV uses 2D discrete wavelet transform (DWT) to decompose video into LF and HF features, preserving spatial structures and directly addressing the spectral bias issue. To balance the compactness, we encode only the LF components, while HF components that include fine textures are generated by a decoder. Specialized modules, including a multi-resolution fusion unit (MFU) and a high-frequency restorer (HFR), are integrated into a backbone to facilitate the representation. Furthermore, we extend SNeRV to effectively capture temporal correlations between adjacent video frames, by casting the extension as additional frequency decomposition to a temporal domain. This approach allows us to embed spatio-temporal LF features into the network, using temporally extended up-sampling blocks (TUBs). Experimental results demonstrate that SNeRV outperforms existing NeRV models in capturing fine details and achieves enhanced reconstruction, making it a promising approach in the field of implicit video representations. The codes will be available at https://github.com/qwertja/SNeRV.

Haijun Xiong, Bin Feng, Xinggang Wang, Wenyu Liu

Gait recognition, a biometric technology, aims to distinguish individuals by their walking patterns. However, we reveal that discriminative identity features often become entangled with non-identity clues, posing a challenge for extracting identity features effectively and efficiently in previous methods. To address this challenge, we propose CLTD, a causality-inspired discriminative feature learning module designed to effectively eliminate the influence of confounders in triple domains, \ie, the spatial, temporal, and spectral domains. Specifically, we utilize the Cross Pixel-wise Attention Generator (CPAG) to generate attention distributions for factual and counterfactual features in spatial and temporal domains. Then, we introduce the Fourier Projection Head (FPH) to project spatial features into the spectral space, preserving essential information while reducing computational costs. Furthermore, we employ an optimization method with contrastive learning to enforce semantic consistency constraints across sequences from the same subject. The significant performance improvements on challenging datasets demonstrate the effectiveness of our method. In addition, our method can seamlessly integrate into existing gait recognition methods.

Yang Shaolei, Shen Cheng, Mingbo Hong, Haoqiang Fan, Xing Wei, Shuaicheng Liu

In this paper, we propose Neural Spectrum Decomposition, a generic decomposition framework for dataset distillation. Unlike previous methods, we consider the entire dataset as a high-dimensional observation that is low-rank across all dimensions. We aim to discover the low-rank representation of the entire dataset and perform distillation efficiently. Toward this end, we learn a set of spectrum tensors and transformation matrices, which, through simple matrix multiplication, reconstruct the data distribution. Specifically, a spectrum tensor can be mapped back to the image space by a transformation matrix, and efficient information sharing during the distillation learning process is achieved through pairwise combinations of different spectrum vectors and transformation matrices. Furthermore, we integrate a trajectory matching optimization method guided by a real distribution. Our experimental results demonstrate that our approach achieves state-of-the-art performance on benchmarks, including CIFAR10, CIFAR100 and Tiny Imagenet.

Mingyu Yang, Daizong Liu, Keke Tang, Pan Zhou, Lixing Chen, Junyang Chen

With the maturity of depth sensors, point clouds have received increasing attention in various 3D safety-critical applications, while deep point cloud learning models have been shown to be vulnerable to adversarial attacks. Most existing 3D attackers rely on implicit global distance losses to perturb whole points, failing to restrict the proper 3D geometry as point clouds are highly structured. To this end, in this paper, we propose a novel Wavelet Patches Attack (WPA), which leverages local spectral attributes to identify curvature-aware patches for hiding imperceptible perturbations aligned with their local geometric characteristics. Specifically, WPA first transforms the point cloud into the spectral domain using a wavelet operator, obtaining potential geometric structures in different local regions. Each wavelet corresponds to different curvature contexts of local points. Then, by decomposing the 3D object with different curvature-aware levels through the wavelet coefficients, we can perceive the local geometric characteristics and get various curvature-consistent patches. At last, based on the curvature variations of patches, WPA introduces two-type perturbations along the tangent plane and normal vector direction to hide imperceptible noise in slow- and fast-variation patches for preserving the geometric-sensitive local characteristics of smoothness and sharpness, respectively. Experiments demonstrate the superior imperceptibility of our attack method, achieving favorable results on existing 3D classification models while exhibiting robust resistance to various defense mechanisms.

Sehyung Lee, Mijung Kim, Yeongnam Chae, Bjorn Stenger

This paper introduces a pioneering approach to linearly controllable generative adversarial network (LC-GAN) driven by unsupervised learning. Departing from traditional methods relying on supervision signals or post-processing for latent feature disentanglement, our proposed technique enables unsupervised learning using only image data through contrastive feature categorization and spectral regularization. In our framework, the discriminator constructs geometry- and appearance-related feature spaces using a combination of image augmentation and contrastive representation learning. Leveraging these feature spaces, the generator autonomously categorizes input latent codes into geometry- and appearance-related features. Subsequently, the categorized features undergo projection into a subspace via our proposed spectral regularization, with each component adeptly controlling a distinct aspect of the generated image. Beyond providing fine-grained control over the generative model, our approach achieves state-of-the-art image generation quality on benchmark datasets, including FFHQ, CelebA-HQ, and AFHQ-V2.

Hongyuan Wang, Lizhi Wang, Jiang Xu, Chang Chen, Xue Hu, Fenglong Song, Youliang Yan

Spectral super-resolution that aims to recover hyperspectral image (HSI) from easily obtainable RGB image has drawn increasing interest in the field of computational photography. The crucial aspect of spectral super-resolution lies in exploiting the correlation within HSIs. However, two types of bottlenecks in existing Transformers limit performance improvement and practical applications. First, existing Transformers often separately emphasize either spatial-wise or spectral-wise correlation, disrupting the 3D features of HSI and hindering the exploitation of unified spatial-spectral correlation. Second, existing self-attention mechanism always establishes full-rank correlation matrix by learning the correlation between pairs of tokens, leading to its inability to describe linear dependence widely existing in HSI among multiple tokens. To address these issues, we propose a novel Exhaustive Correlation Transformer (ECT) for spectral super-resolution. First, we propose a Spectral-wise Discontinuous 3D (SD3D) splitting strategy, which models unified spatial-spectral correlation by integrating spatial-wise continuous splitting strategy and spectral-wise discontinuous splitting strategy. Second, we propose a Dynamic Low-Rank Mapping (DLRM) model, which captures linear dependence among multiple tokens through a dynamically calculated low-rank dependence map. By integrating unified spatial-spectral attention and linear dependence, our ECT can model exhaustive correlation within HSI. The experimental results on both simulated and real data indicate that our method achieves state-of-the-art performance. Codes and pretrained models are available at https://anonymous.4open.science/r/ECT-FC89.