论文检索

Deep Learning · Graph Neural Networks

Yilin Liu, Hongchao Zhang, Ahmad Taha, Taylor T Johnson, Meiyi Ma

Graph anomaly detection methods aim to distinguish anomalous nodes. While prior methods characterize anomalies through increased variation in the spectral energy distributions, they overlook those that result in decreased variation, i.e., camouflaged anomalies that appear normal. We show that this type of anomaly persists across multiple datasets and remains undetectable by existing spectral approaches. To address this limitation, we propose a node-level spectral energy formulation that is fully compatible with message passing and enables the detection of camouflaged anomalies. Building on this formulation, we introduce an energy-aware graph learning framework that models spectral shifts through energy-driven message passing in both static and time-series graphs. Besides, our unified architecture extends to temporal settings without introducing specialized sequence modules, enabling efficient learning under long sliding windows. Extensive experiments on large-scale benchmarks demonstrate the effectiveness and scalability of our approach.

Theory · Optimization

Yangnan Li, Xuanyu Cao, Shenghui Song

Resilience against Byzantine attackers and faster convergence on sparse networks are critical for decentralized optimization, yet existing methods fail to achieve both simultaneously. Existing DSGD-based Byzantine-resilient methods suffer from high transient complexity of $\mathcal{O}\left((1-\lambda)^{-6}\right)$, where $1-\lambda$ denotes the spectral gap of the network. While bias-correction methods such as Exact Diffusion can improve topology dependence, directly combining them with robust aggregators can lead to error accumulation. To address this issue, we introduce the scaled dual ascent (SDA) within the augmented Lagrangian framework for decentralized optimization, which mitigates error accumulation by scaling the dual update steps. Based on this, we propose BRED, which integrates Byzantine-robust Exact Diffusion with the SDA framework. We prove that BRED attains linear speedup, and achieves transient complexity of $\mathcal{O}\left((1-\lambda)^{-2}\right)$ when the Byzantine fraction $\delta$ is small. We further propose the momentum variant BRED-M, which reduces the Byzantine-affected transient complexity from $\mathcal{O}\left(\delta^2(1-\lambda)^{-6}\right)$ to $\mathcal{O}\left(\delta^2(1-\lambda)^{-4}\right)$. Empirical results on benchmark datasets demonstrate the efficacy of the proposed methods across diverse network topologies.

Deep Learning · Large Language Models

Md Kowsher, Ali Polat, Ehsan Ardehaly, Mehrdad Salehi, Zia Ghiasi, Prasanth Murali, Chen Chen

This paper presents a theoretical framework that explains why fine-tuning small, randomly selected subnetworks (slices) within pre-trained models is sufficient for downstream adaptation. We prove that pretrained networks exhibit a universal winning slice property, arising from two phenomena: (1) spectral balance— the eigenspectra of different weight matrix slices are remarkably similar—and (2) high task energy—their backbone representations (pretrained weights) retain rich, task-relevant features. This leads to the Universal Winning Slice Hypothesis, which provides a theoretical foundation for parameter-efficient fine-tuning (PEFT) in large-scale models. Inspired by this, we propose SliceFine, a PEFT method that uses this inherent redundancy by updating only selected slices of the origi- nal weights—introducing zero new parameters, unlike adapter-based approaches. Empirically, SliceFine matches the performance of SOTA PEFT methods across various language and vision tasks, while significantly improving training speed, memory efficiency, and model compactness. Our work bridges theory and prac- tice, offering a theoretically grounded alternative to existing PEFT techniques.

Probabilistic Methods · Spectral Methods

Roddy Taing, Keith Levin

As network data has become ubiquitous in the sciences, there has been growing interest in network models whose structure is driven by latent node-level variables in a (typically low-dimensional) latent geometric space. These "latent positions" are often estimated via embeddings, whereby the nodes of a network are mapped to points in Euclidean space so that "similar" nodes are mapped to nearby points. Under certain model assumptions, these embeddings are consistent estimates of the latent positions, but most such results require the embedding dimension to be chosen correctly. Methods for choosing the embedding dimension have been studied extensively, but little is known about the behavior of embeddings when the dimension is misspecified. In this work, we provide a theoretical description of the effects of dimension misspecification under the random dot product graph, a class of latent space network models that includes several widely-used network models, most notably the stochastic blockmodel, as special cases. We show that when the dimension is chosen too large, consistent estimation still holds, albeit at a slower rate than when the embedding dimension is chosen correctly. On the other hand, when the dimension is chosen too small, there is a fundamental estimation error lower bound that need not go to zero in the large-network limit. A range of synthetic data experiments support our theoretical results. Our main technical result, which may be of independent interest, is a generalization of earlier work in random matrix theory showing that all non-signal eigenvectors of a low-rank matrix subject to additive noise are delocalized.

Applications · Computer Vision

Yong Yang, Xuran Zhang, Shuying Huang, Xiaozheng Wang, Weiguo Wan, Hangyuan Lu

Hyperspectral Image Single Image Super-Resolution (HSI-SISR) faces a conflict between computational efficiency and global non-local modeling. Existing Transformers suffer from quadratic complexity, while window-based methods compromise global capture. To address this, we propose the Spatial-Spectral Dual-Clustering-based Network (SSDCN). Our method introduces three innovations. First, we design a Spatial-Spectral Dual-Cluster Block (SSDCB). Replacing expensive point-to-point attention, it uses content-driven clustering to learn low-rank structural bases, achieving global modeling with linear complexity $\mathcal{O}(KN)$. Second, we propose a pyramid progressive hierarchical architecture with a Feature Reuse Reconstruction Block (FRRB). It reuses the core tensor and spectral factors from coarse levels, updating only spatial factors to minimize redundancy. Third, we propose a Pyramid Hierarchical Reconstruction Joint Loss to supervise intermediate levels, ensuring structural accuracy and preventing error accumulation. Experiments demonstrate that SSDCN surpasses SOTA methods in metrics and visual quality with significantly fewer parameters and FLOPs, achieving an optimal efficiency-performance balance.

Social Aspects · Alignment

JianKui Zhou, Jing Yao, Xiaoyuan Yi, Peng Zhang, Ning Gu, Zhan Hu, Xing Xie, Tun Lu

With the widespread deployment of large language models (LLMs), aligning model outputs with pluralistic human values has become an important research problem. Recent approaches that train task-specific experts and merge them through parameter aggregation have shown promise for pluralistic alignment. However, these methods often overlook the intrinsic complexity of real-world value data, where multiple correlated value dimensions coexist, resulting in highly similar and entangled expert representations. Consequently, modifying the contribution of one value expert may unintentionally influence other values, limiting fine-grained controllability. To address this issue, we propose DisAlign, a model-merging framework that explicitly decomposes value representations into consensus and value-specific components using an information-geometric perspective. DisAlign first extracts a consensus anchor and subspace to capture shared structure across values, and then applies spectral decomposition to the residual representations to construct disentangled value subspaces. This design enables more precise and independent modulation of multiple values. Experiments on three datasets covering different value frameworks demonstrate that DisAlign consistently improves value disentanglement and achieves more accurate pluralistic value control compared to existing baselines. Our code is available at \url{https://anonymous.4open.science/r/DisAlign-7F35}

Deep Learning · Large Language Models

Guanghui Min, Tianhao Huang, Ke Wan, Chen Chen

Targeted data selection has emerged as a crucial paradigm for efficient instruction tuning, aiming to identify a small yet influential subset of training examples for a specific target task. In practice, influence is often measured through the effect of an example on parameter updates. To make selection scalable, many approaches leverage optimizer statistics (e.g., Adam states) as an axis-aligned surrogate for update geometry (i.e., diagonal precondition), implicitly treating parameters as coordinate-wise independent. We show that this assumption breaks down in parameter-efficient fine-tuning (PEFT) methods such as LoRA. In this setting, the induced optimization geometry exhibits strong cross-parameter coupling with non-trivial off-diagonal interactions, while the task-relevant update directions are confined to a low-dimensional subspace. Motivated by this mismatch, we propose **GIST** (Gradient Isometric Subspace Transformation), a simple yet principled alternative that replaces axis-aligned scaling with robust subspace alignment. **GIST** recovers a task-specific subspace from validation gradients via spectral filtering (SVD), projects training gradients into this coupled subspace, and scores examples by their alignment with target directions. Extensive experiments have demonstrated that **GIST** matches or outperforms the state-of-the-art baseline with only 0.29% of the storage and 25% of the computational time under the same selection budget. Our code is available at https://anonymous.4open.science/r/GIST-1464.

Applications · Chemistry, Physics, and Earth Sciences

Tianchi Yu, Ivan Oseledets

For low-dimensional problems ($d\leq3$), spectral methods can achieve exceptionally high accuracy. For middle-dimensional problems ($4 \leq d \lesssim 10$), spectral methods remain feasible through specific techniques such as sparse grids or hyperbolic cross. However, for high-dimensional problems ($d\gg 10$), spectral methods suffer from the curse of dimensionality. Physics-informed neural networks (PINNs) have emerged as a promising approach to overcome this challenge, offering scalability to high dimensions, but often suffer from limited accuracy and efficiency. Recently proposed spectral-informed neural networks (SINNs) combine spectral methods with PINNs, operating directly in the spectral domain to avoid spatial derivative computations and to reduce memory consumption. In this work, we introduce Modified SINNs, which integrate coefficient decay scaling and basis embeddings motivated by harmonic analysis to enhance accuracy in high-dimensional problems and enable accurate approximation of unknown spectral coefficients. Numerical experiments on steady and time-dependent partial differential equations demonstrate that Modified SINNs outperform sparse grid spectral methods on middle-dimensional problems with incomplete spectral information and achieve superior accuracy compared to PINNs on high-dimensional problems.

Applications · Chemistry, Physics, and Earth Sciences

Yingheng Wang, Tao Yu, Shufeng Kong, Yingheng Wang, John Gregoire, Carla Gomes

Structured scientific spectra encode rich physical information while satisfying hard constraints such as conservation and spectral geometry. We study a canonical example, the electronic density of states (eDOS), whose accurate prediction is central to materials discovery. Prior methods often (i) decouple band gaps from eDOS, (ii) violate total-state conservation, or (iii) collapse crystals into global embeddings that obscure atom-projected contributions. We introduce \textbf{DeepSciReasoner}, a design paradigm for deep scientific reasoning under physical constraints. Instantiated for eDOS prediction, DeepSciReasoner combines structure-aware spectrum decoding with constraint-preserving physical reasoning, in this case, mass-conserving iterative refinement. It substantially improves eDOS accuracy while maintaining physical validity, enabling reliable high-throughput screening. Beyond eDOS, DeepSciReasoner offers a reusable blueprint for predicting structured scientific spectra under hard physical constraints.

Applications · Chemistry, Physics, and Earth Sciences

Zherui Yang, Haiyang Xin, Tao Du, Ligang Liu

Neural operators have emerged as data-driven surrogates for solving partial differential equations (PDEs), and their success hinges on efficiently modeling the long-range, global coupling among spatial points induced by the underlying physics. In many PDE regimes, the induced global interaction kernels are empirically compressible, exhibiting rapid spectral decay that admits low-rank approximations. We leverage this observation to unify representative global mixing modules in neural operators under a shared low-rank template: compressing high-dimensional pointwise features into a compact latent space, processing global interactions within it, and reconstructing the global context back to spatial points. Guided by this view, we introduce Low-Rank Spatial Attention (LRSA) as a clean and direct instantiation of this template. Crucially, unlike prior approaches that often rely on non-standard aggregation or normalization modules, LRSA is built purely from standard Transformer primitives, i.e., attention, normalization, and feed-forward networks, yielding a concise block that is straightforward to implement and directly compatible with hardware-optimized kernels. In our experiments, such a simple construction is sufficient to achieve high accuracy, yielding an average error reduction of over 17\% relative to second-best methods, while remaining stable and efficient in mixed-precision training.

Deep Learning · Foundation Models

Arnesh Batra, Arush Gumber, Aniket Khandelwal, Jashn Khemani, Anubha Gupta

Foundational Models pretrained on huge amount of data learn representations that evolve across depth, forming a hierarchy of embeddings with distinct semantic content and geometric structure. Contrary to the widespread practice of using only the final layer or shallow mixtures, we show that task-relevant information is distributed non-monotonically across layers and cannot be recovered by naive aggregation. Through a geometric and empirical study across multiple modalities, we show that effective transfer depends on identifying which layers encode task-discriminative structure and how their embeddings are geometrically organized. We introduce Layer-wise Optimal Embedding Selection (LOES), a constructive spectral method that identifies task-discriminative subspaces by minimizing residual error under orthogonality and isotropy constraints. To align fine-tuning with this selection principle, we further propose Geometric Regularization Loss (GeoReg), which enforces a simplicial structure on class manifolds and stabilizes representation geometry during fine-tuning. Across a wide range of architectures, depths, modalities, and data regimes, LOES consistently outperforms standard baselines, with gains that grow as model depth increases. Beyond accuracy, our method reveals how semantic factors are distributed across layers, thereby enabling cross-lingual and cross-modal interpretability analyses. Together, our results provide strong evidence that layerwise embedding geometry is not incidental but central to how deep models represent and transfer knowledge.

Reinforcement Learning · Multi-agent

Jiajun Wu, Xuefeng Du, Yuduo Zheng, Fengqi Li

Decentralized multi-agent reinforcement learning faces a persistent exploration–coordination tension: intrinsic rewards promote exploration under sparse feedback, yet effective cooperation requires agents’ behaviors to remain consistent over a limited communication graph. Existing methods often combine exploration bonuses and coordination regularizers with fixed-weight schedules, making them hard to tune and prone to either fragmented conventions or premature behavioral collapse. We propose the IEC (Isomorphic Exploration-Consensus) framework that couples exploration and coordination through a single constrained objective: maximize task return augmented with two complementary exploration signals, dynamics-based information gain and state-coverage novelty, while constraining graph-induced policy disagreement via a spectral smoothness penalty on neighboring agents, which can be interpreted as a Dirichlet-energy regularizer on the communication graph. IEC optimizes the resulting Lagrangian with a lightweight primal–dual update that adapts the consensus multiplier from observed constraint violations, yielding an automatic shift from diverse exploration to stable cooperative conventions. Across three distinct benchmarks, IEC achieves superior performance.

Applications · Computer Vision

Bingxuan Zhao, Qing Zhou, Yu Wang, Chuang Yang, Qi Wang

While Diffusion Transformers (DiTs) have revolutionized high-fidelity image synthesis, the prohibitive computational costs of training at ultra-high resolutions necessitate robust inference-time extrapolation. Existing extrapolation methods typically operate under a *scale-agnostic* assumption, treating the denoising dynamics identically across resolutions. In this work, we identify a critical oversight in this paradigm: the spectral evolution of the diffusion process, transitioning from low-frequency structural construction to high-frequency texture refinement, is inherently scale-dependent. Consequently, applying a uniform strategy across scales causes a spectral misalignment, manifesting as *structural collapse* or *textural degradation*. To bridge this gap, we introduce **SigMa ($\sigma$)**, a training-free framework that utilizes Sigmoid Modulation for *scale-adaptive* calibration of the extrapolation process. SigMa orchestrates the spectral evolution via a parameterized schedule with two core mechanisms: *Decoupled Geometric Center Alignment*, which synchronizes the transition timing to secure global structure, and *Iso-Variance Rate Adaptation*, which scales the transition velocity to ensure a smooth feature handover. Extensive experiments demonstrate that SigMa effectively rectifies spectral deviations, enabling training-free extrapolation up to 16 megapixels and achieving state-of-the-art performance on standard benchmarks.

Applications · Computer Vision

Noam Issachar, Guy Yariv, Sagie Benaim, Yossi Adi, Dani Lischinski, Raanan Fattal

Diffusion Transformer models can generate images with remarkable fidelity and detail, yet training them at ultra-high resolutions remains extremely costly due to the self-attention mechanism's quadratic scaling with the number of image tokens. In this paper, we introduce Dynamic Position Extrapolation (DyPE), a novel, training-free method that enables pre-trained diffusion transformers to synthesize images at resolutions far beyond their training data, with no additional sampling cost. DyPE takes advantage of the spectral progression inherent to the diffusion process, where low-frequency structures converge early, while high-frequencies take more steps to resolve. Specifically, DyPE dynamically adjusts the model's positional encoding at each diffusion step, matching their frequency spectrum with the current stage of the generative process. This approach allows us to generate images at resolutions that exceed the training resolution dramatically, e.g., 16 million pixels using FLUX. On multiple benchmarks, DyPE consistently improves performance and achieves state-of-the-art fidelity in ultra-high-resolution image generation, with gains becoming even more pronounced at higher resolutions.

General Machine Learning · Unsupervised and Semi-supervised Learning

Zhixin Zhou, Navin Souda, Arash Amini

This paper addresses the problem of linear seriation: recovering the intrinsic order of noisy samples drawn from an unknown one-dimensional manifold embedded in a higher-dimensional space. We propose a multi-stage approach that first robustly estimates local tangent directions using Principal Component Analysis (PCA) on neighborhoods, establishing theoretical consistency for these local estimates. Global orientation consistency of these tangents is then achieved through a spectral relaxation of a pairwise alignment objective. Finally, a globally consistent 1D embedding is computed by solving a carefully formulated linear system (or equivalently, a spectral problem on a derived Laplacian) that aligns the embedding with the oriented local projections. This method effectively leverages local geometric information while ensuring global coherence, producing an ordering robust to noise, curvature, and initial data rotation. We demonstrate its performance on simulated manifold data and discuss the theoretical underpinnings of its core components.

Applications · Neuroscience, Cognitive Science

Yifan Wang, Yijia Ma, Wen Li, Chenyu You

High-fidelity EEG generation is critical for alleviating data scarcity and addressing privacy constraints in large-scale neural modeling. Despite recent progress, most existing approaches formulate EEG generation via discrete denoising objectives, which inadequately reflect the inherently continuous temporal dynamics and spectral structure of neural activity. As a result, these methods often struggle to preserve long-range temporal dependencies and exhibit mismatches in the spectral and temporal structure of the generated signals. In this work, we argue that effective EEG generation requires models that operate directly on the continuous evolution of neural signals. We introduce Just EEG Transformer (JET), a generative framework based on conditional flow matching that models EEG as raw sequences evolving along continuous trajectories. By learning a smooth vector field that transports noise to the EEG data distribution, JET captures temporal continuity and transient dynamics without relying on discretized denoising schemes or domain-specific representations. To ensure that the learned dynamics remain consistent with key properties of EEG signals, we introduce principled constraints that preserve spectral structure, temporal stationarity, and signal-level statistics. Across three large-scale benchmarks, JET consistently achieves state-of-the-art performance, reducing TS-FID by over 40\% compared to strong baselines. Extensive analyses show that JET captures key structural properties of neural dynamics, providing a scalable and principled approach to EEG generation.

Deep Learning · Sequential Models, Time series

Wei Shao, Ziquan Fang, Zheqi Lu, Yongfeng Su, Yuzhu Wang, Yunjun Gao

Time-series analysis is critical in real-world applications, yet the explosion of time-series data imposes severe burdens on storage and computational resources. Recently, dataset condensation has emerged as a promising data-centric solution by synthesizing compact yet informative datasets to replace large-scale raw data. However, existing methods are largely vision-centric, failing to capture unique temporal properties of time series, or task-specific, tightly coupling the condensed data to a particular downstream objective. As a result, these approaches suffer from feature mismatch and fail to generalize across diverse time-series tasks. To bridge this gap, we propose UniTSC, the first unified dataset condensation framework for general time-series analysis. UniTSC employs a multi-view hybrid encoder to capture task-invariant representations across temporal, spectral, and topological perspectives. Building upon this representation, we design a tri-space alignment paradigm that jointly aligns optimization trajectories, power spectral densities, and multivariate dependency structures, enabling comprehensive information preservation under extreme compression. Extensive experiments show that UniTSC retains up to 97.9\% of downstream performance using as little as 0.01\% of the original training data, revealing that a single batch-equivalent budget ($\textless$ 128 samples) is sufficient to capture the essential dynamics of complex time-series data.

Applications · Chemistry, Physics, and Earth Sciences

Yunlong Zhou, Chen Zhao, danyang peng, Fanfan Ji, Xiaotong Yuan

Accurate precipitation nowcasting is vital for disaster mitigation, but deep learning methods suffer a key trade-off: regression models produce over-smoothed, spectrally decaying predictions that blur convective details and violate turbulence power laws; diffusion models generate realistic yet unanchored hallucinations lacking physical grounding. We propose Spectral-Decoupled Iterative Refinement (SDIR), a deterministic framework that reformulates nowcasting as progressive frequency-decoupled refinement. SDIR first extracts a stable low-frequency synoptic skeleton, then iteratively refines high-frequency textures under physical constraints, eliminating both blurring and hallucinations. It features a dual-path design: the Synoptic Frequency-Guided Former (SFG-Former) with Scale-Adaptive Transformers for global structure, and the Fourier Residual Refiner (FR-Refiner) with Scale-Conditioned Fourier Neural Operators for fine residuals. A Physically-Consistent Power Spectral Density (PCPSD) loss with dynamic masking enforces turbulence-consistent spectral distribution. Experiments on three benchmarks show SDIR significantly outperforms SOTA in spatial accuracy while achieving spectral fidelity competitive with diffusion-based methods, enabling reliable high-resolution operational nowcasting.

Deep Learning · Theory

Fares Mehouachi, Saif Jabari

Neural operators learn mappings between function spaces, enabling fast PDE surrogates. Despite architectural diversity, these methods often share a common input representation: raw coordinate-value pairs. This ignores the differential structure that defines the underlying physics. We study whether derivative features can improve neural operator learning. Through Picard iteration on mild solutions, we show that derivatives of the input naturally enter PDE solution operators, and we prove that providing them improves approximation rates substantially. But the optimal derivative order is not what one might expect. We show that the statistically optimal order is strictly less than the PDE order, for any finite sample size. This gap arises from a bias-variance tradeoff: higher-order derivatives carry more information but amplify noise. We characterize this tradeoff in closed form and show that learning the derivative order from data achieves automatic spectral regularization. We introduce $\partial$-NO (derivative-augmented neural operators), a simple augmentation that provides learnable fractional derivative features to any neural operator backbone. Across benchmarks, this consistently improves accuracy. Learned orders reflect dominant PDE structure while adapting to finite-sample constraints, confirming the theory.

General Machine Learning · Clustering

Ben Jourdan, Peter Macgregor, Gregory Schwartzman

We present a fully dynamic data structure that supports edge and node updates and cluster membership queries for the Normalised Cut problem with strong theoretical guarantees. Furthermore, our data structure outperforms the state of the art significantly on real world datasets. At the heart of our data structure is the novel notion of *Just-in-Time Sampling Trees*. The worst-case edge update time of our data structure is $O(\log n)$ where $n$ is the number of nodes in the current graph. Let $d_{\max}$ be the maximum degree of the current graph, let $T_{NC}(n',k')$ be the running time of an $\alpha$-approximation algorithm for the Normalised Cut problem on $n'$ vertices and $k'$ clusters, and let $\text{vol}(Y)$ be the sum of the unweighted degrees of all nodes in a set $Y$. The worst-case query time of our data structure to label all nodes in $Y$ is $O\left(kd_{max}^2 \log(n) + \gamma(n,\epsilon,k,Y)\right)$, with approximation ratio $\alpha \frac{1+\epsilon}{1-\epsilon}$, where $\gamma(n,\epsilon,k,Y)$ is defined to be $\log(n)\log^\star(n)\epsilon^{-4}k^2 + T_{NC}(\epsilon^{-4}k^2,k) + \epsilon^{-8}k^4 +vol(Y)$. Assuming $d_{\max}$ is polylogarithmic, as is the case with many sparse real-world graphs, our method achieves the best known trade-off between query time and update time.