论文检索

Deep Learning · Graph Neural Networks

Ayushman Raghuvanshi, Thummaluru Siddartha Reddy, Sundeep Prabhakar Chepuri, Mahesh Chandran

Continuous-time dynamic graphs (CTDGs) provide a richer framework to capture fine-grained temporal patterns in evolving relational data. Long-range information propagation is a key challenge while learning representations, wherein it is important to retain and update information over long temporal horizons. Existing approaches restrict models to capture one-hop or local temporal neighborhoods and fail to capture multi-hop or global structural patterns. To mitigate this, we derive a parameter-efficient state-space modeling framework for continuous-time dynamic graphs $\texttt{(CTDG-SSM)}$ from first principles. We first introduce continuous-time Topology-Aware higher order polynomial projection operator ($\texttt{CTT-HiPPO}$), a novel memory-based reformulation of $\texttt{HiPPO}$ to jointly encode temporal dynamics and graph structure. The solution from $\texttt{CTT-HiPPO}$ are obtained by projecting the classical HiPPO solution through a polynomial of the Laplacian matrix, yielding topology-aware memory updates that admit an equivalent state-space formulation for CTDGs ($\texttt{CTDG-SSM}$). Then a computationally efficient discrete formulation is obtained using the zero-order hold approach for model implementation. Across benchmarks on dynamic link prediction, dynamic node classification, and sequence classification, $\texttt{CTDG-SSM}$ achieves state-of-the-art performance. Notably, it achieves large performance gains on datasets that require long range temporal (LRT) and spatial reasoning.

Deep Learning · Graph Neural Networks

Jiaming Zhuo, Shuai Zhai, Ziyi Ma, Kun Fu, Chuan Wang, Di Jin, Zhen Wang, Xiaochun Cao, Huazhu Fu, Liang Yang

Brain network classification is pivotal for diagnosing neurological disorders, yet clinical interpretability and the identification of discriminative biomarkers fundamentally rely on precise functional parcellation. However, existing graph learning models for brain network analysis typically suffer from a critical limitation termed spectral unreachability. This stems from the widely used coupled encoder-pooling architecture, where the inherent representation smoothing property of graph encoders (including Graph Neural Networks and Graph Transformers) inevitably corrupts the high-frequency topological signals essential for delineating sharp functional boundaries. To solve this issue, the Hierarchical Spectral Parcellation Network (HiSP-Net) is proposed. Adopting a *project-then-align* philosophy, HiSP-Net structurally decouples partition learning from representation smoothing. Specifically, this model is constructed as a hierarchy of Spectral Parcellation blocks. Within each block, node (or module) representations are mapped directly via a topology-agnostic projection into a partition space to preserve high-frequency details, while a Topology-Aware Alignment mechanism enforces spatial coherence using a joint structural objective. Extensive evaluations on real-world datasets show the capability of HiSP-Net in achieving superior classification performance and extracting interpretable functional biomarkers. The source code is publicly available at https://anonymous.4open.science/r/HiSP-Net-demo-0F62/

Optimization · Zero-order and Black-box Optimization

Alexander Chebykin, Tanja Alderliesten, Peter A.N Bosman

Hyperparameter Optimization (HPO) can lift the burden of tuning hyperparameters (HPs) of neural networks. HPO algorithms from the Population Based Training (PBT) family are efficient thanks to dynamically adjusting HPs every few steps of the weight optimization. Recent results indicate that the number of steps between HP updates is an important meta-HP of all PBT variants that can substantially affect their performance. Yet, no method or intuition is available for efficiently setting its value. We introduce Iterated Population Based Training (IPBT), a novel PBT variant that automatically adjusts this HP via restarts that reuse weight information in a task-agnostic way and leverage time-varying Bayesian optimization to reinitialize HPs. Evaluation on 8 image classification and reinforcement learning tasks shows that, on average, our algorithm matches or outperforms 5 previous PBT variants and other HPO algorithms (random search, ASHA, SMAC3), without requiring a budget increase or any changes to its HPs.

Deep Learning · Generative Models and Autoencoders

David Zagardo

Tabular data synthesis is critical for privacy-preserving data sharing and augmentation, yet existing diffusion models rely on implicit attention mechanisms to capture inter-column relationships. We introduce Geometry-Aware Tabular Diffusion, which augments diffusion models with explicit pairwise geometric features - angles and lengths - computed directly from column value differences. Our method achieves state-of-the-art performance on standard benchmarks while using 3.5 times fewer parameters on average (up to 25 times for classification tasks) than transformer-based approaches. On ten datasets, we win on 8/10 for Shape (marginal fidelity) with 27% error reduction, 7/10 for Trend (correlation preservation) with 20% error reduction, and 9/10 for downstream utility (F1/RMSE). These results demonstrate that explicit relational structure can substitute for model capacity, enabling state-of-the-art tabular synthesis with simple, efficient architectures.

Deep Learning · Foundation Models

Hongyu Ke, Jack Morris, Yongkang Liu, Satoshi Kitai, Kentaro Oguchi, Yi Ding, Haoxin Wang

State Space Models (SSMs) have emerged as a powerful and efficient alternative to Transformers, demonstrating linear-time complexity and exceptional sequence modeling capabilities. However, their application to vision tasks remains challenging. First, existing vision SSMs largely depend on manually designed fixed scanning methods to flatten image patches into sequences, which imposes predefined geometric structures and increases the complexity. Second, the broader adoption of vision SSMs is hindered in domains that require query-based interactions between distinct information streams. This is a result of the inherently causal and self-referential nature of SSMs designed for 1D sequence modeling tasks. This fusion mechanism is indispensable for critical perception tasks such as multi-view 3D fusion. To address these limitations, we propose Deformba, a context adaptive method that dynamically augments the spatial structural information while maintaining the linear complexity of SSMs. Deformba also allows multi-modal fusion, analogous to standard cross attention. To demonstrate the effectiveness and general applicability of Deformba, we test its performance on general 2D vision tasks such as image classification, object detection, instance segmentation, and semantic segmentation, as well as 3D vision tasks like BEV perception. Extensive experiments show that Deformba achieves strong performance across various visual perception benchmarks.

General Machine Learning · Representation Learning

Panagiotis Koromilas, Theodoros Giannakopoulos, Mihalis Nicolaou, Yannis Panagakis

Supervised classifier learning has a theoretical optimum — Neural Collapse (NC) — yet standard training does not reach it in practice. We trace this failure to a geometric limitation: cross-entropy is invariant to joint rescaling of features and weights, leaving radial degrees of freedom unconstrained and the loss landscape degenerate. Projecting optimization onto the unit hypersphere eliminates this degeneracy and exposes a hidden equivalence: normalized softmax classification and supervised contrastive learning are conceptually the same, both optimizing angular similarity to class prototypes. We formalize this unification by proving that supervised contrastive learning already produces an optimal classifier during training, the prototype classifier whose weights are given by class-wise feature means, rendering subsequent classifier learning through linear probing redundant. Building on this framework, we identify two computational bottlenecks that slow convergence to NC: the small effective negative set in classifier learning (limited to K class prototypes), and the coupling of competing optimization terms through a shared normalization. We address these with NTCE, which expands the negative set from K classes to M batch instances, and NONL, which normalizes only over negatives to decouple intra-class alignment from inter-class repulsion. Empirically, our methods surpass cross-entropy accuracy on four benchmarks including ImageNet-1K, achieve $\ge$95\% NC across all metrics, and yield consistent gains in transfer learning (+5.5\% mean relative improvement), long-tailed classification (up to +8.7\%), and robustness (lower mCE), while eliminating hours of post-hoc classifier training.

Deep Learning · Foundation Models

Seunghan Lee

TabPFN has recently gained attention as a foundation model for tabular datasets, achieving strong performance by leveraging in-context learning on synthetic data. However, we find that TabPFN is vulnerable to label shift, often overfitting to the majority class in the training dataset. To address this limitation, we propose DistPFN, the first test-time posterior adjustment method designed for tabular foundation models. DistPFN rescales predicted class probabilities by downweighting the influence of the training prior (i.e., the class distribution of the context) and emphasizing the contribution of the model’s predicted posterior, without architectural modification or additional training. We further introduce DistPFN-T, which incorporates temperature scaling to adaptively control the adjustment strength based on the discrepancy between prior and posterior. We evaluate our methods on over 250 OpenML datasets, demonstrating substantial improvements for various TabPFN-based models in classification tasks under label shift, while maintaining strong performance in standard settings without label shift.

Deep Learning · Foundation Models

Juntong Chen, Sophie Langer, Johannes Schmidt-Hieber

The recent statistical theory of neural networks focuses on nonparametric denoising problems that treat randomness as additive noise. Variability in image classification datasets does, however, not originate from additive noise but from variation of the shape and other characteristics of the same object across different images. To address this problem, we introduce a tractable model for supervised image classification. While from the function estimation point of view, every pixel in an image is a variable, and large images lead to high-dimensional function recovery tasks suffering from the curse of dimensionality, increasing the number of pixels in the proposed image deformation model enhances the image resolution and makes the object classification problem easier. We introduce and theoretically analyze three approaches. Two methods combine image alignment with a one-nearest neighbor classifier. Under a separation condition, it is shown that perfect classification is possible. The third method fits a convolutional neural network (CNN) to the data. We derive a rate for the misclassification error that depends on the sample size and the complexity of the deformation class. An empirical study corroborates the theoretical findings.

Social Aspects · Alignment

Yulong Lin, Pablo Bernabeu-Pérez, Benjamin Arnav, Lennie Wells, Mary Phuong

As language models grow more capable, accurate capability evaluation becomes essential for safety decisions. If models can deliberately underperform on dangerous capability evaluations---a behavior known as \emph{sandbagging}---they may evade safety measures designed for their true capability level. We introduce Cross-Context Consistency (C³), a general framework for unsupervised black-box sandbagging detection that exploits a fundamental asymmetry: when a model truly lacks capability, its confusion manifests consistently across paraphrased questions, but when a capable model feigns incompetence, its strategic choices about \emph{how} to appear weak create detectable inconsistencies. The framework is agnostic to the specific consistency metric and aggregation method; we present a simple instantiation using embedding cosine distance and mean aggregation that requires no training data or model-specific adaptation. We evaluate C³ across prompted and fine-tuned sandbagging scenarios, across instructed and more naturalistic setups, maintaining a classification signal where other black-box methods fail. Our findings show the limitations of existing sandbagging detection methods, and reveal the efficacy of consistency-checking as a detection mechanism for dangerous capabilities.

Social Aspects · Safety

Arkadiy Saakyan, Charvi Rastogi, Lora Aroyo

Safe global deployment of AI models requires alignment with pluralistic human values, yet in existing safety evaluation datasets the rater pools remain largely homogeneous along geo-cultural dimensions. Through a meta-analysis of existing safety datasets, we observe that the vast majority does not include any geo-cultural information, and the ones that do, lack a robust approach to collect and understand cultural differences in safety ratings. Using the Inglehart-Welzel dimensions of cross-cultural variation, we demonstrate via hierarchical linear modeling that geo-cultural values predict safety ratings significantly better than demographic factors alone ($p<0.05$ in $6$ datasets). Further, our analysis shows that several safety datasets contain at least 10\% of culturally-sensitive items, where lack of cultural representation in the rater pool would lead to a false negative in safety classification. Finally, we provide empirical evidence that fine-tuned LLMs can identify culturally sensitive items but are not reliable at emulating judgments of raters from diverse cultural backgrounds, underscoring the critical need for continuous geo-culturally stratified (pluralistic) safety evaluations.

General Machine Learning · Kernel methods

Hachem Kadri, Joachim Tomasi, Yuka Hashimoto, Sandrine Anthoine

Quantum kernels are reproducing kernel functions built using quantum-mechanical principles and have emerged as a centerpiece of quantum machine learning. The initial enthusiasm for quantum kernel machines has been tempered by recent studies suggesting that quantum kernels could not offer significant computational or statistical advantages when learning from classical data. However, most of the research in this area has been devoted to scalar-valued kernels in standard classification or regression settings for which classical kernel methods are efficient and effective, leaving very little room for improvement with quantum kernels. In this position paper, we argue that progress in this field requires moving beyond scalar-valued kernels toward more expressive kernel frameworks. Scalar-valued kernels lack the degrees of freedom necessary to fully exploit intrinsically quantum resources such as entanglement and are not rich enough to deal with complex learning tasks where classical learning methods struggle. Building on recent advances in operator-valued kernel learning and C*-algebraic kernel representations, we propose a roadmap for designing quantum kernels capable of leveraging entanglement and non-commutative structures to tackle complex structured prediction problems. To support this viewpoint, we present an initial proof-of-concept illustrating how quantum entangled operator-valued kernel formulations can reveal structural dependencies that remain difficult to access for scalar-valued kernel methods. This shift in focus could open a pathway toward a new generation of quantum kernel machines and a more faithful exploration of their potential advantages.

Applications · Health / Medicine

Marta Hasny, Laura Daza, Keno Bressem, Maxime Di Folco, Julia Schnabel

Large-scale medical biobanks provide imaging data complemented by extensive tabular information, such as clinical measurements or demographics. However, this abundance of tabular attributes does not reflect real-world datasets, where only a subset of attributes may be available. This discrepancy calls for methods that remain robust to missing values at inference. To address this challenge, we propose RoVTL (Robust Vision-Tabular Learning), a framework designed to handle any level of tabular data availability, from 0% to 100%. RoVTL comprises two key stages: contrastive pretraining, where we introduce tabular attribute missingness as data augmentation to promote robustness, and downstream task tuning, where tabular missingness is complemented by a novel Tabular More vs. Fewer loss that ranks performance based on the amount of available tabular data. Combined with gated-cross attention fusion module, our tuning approach enables consistent performance across all tabular data completeness scenarios. We evaluate RoVTL on cardiac MRI scans from the UK Biobank, demonstrating superior robustness to missing tabular data compared to prior methods. Furthermore, RoVTL successfully generalizes to an external cardiac MRI dataset for multimodal disease classification, and extends to the natural images domain, achieving robust performance on a car advertisements dataset. Model weights and code will be released.

Theory · Learning Theory

Mehryar Mohri, Yutao Zhong

The choice of loss function in classification involves a fundamental trade-off: smooth losses (like Cross-Entropy) enable fast optimization rates but yield slow square-root consistency bounds, while piecewise-linear losses (like Hinge) offer fast linear consistency rates but suffer from non-differentiability. We propose *Linear-Core (LC) Surrogates*, a new family of convex loss functions that resolve this tension by stitching a linear core to a smooth tail. We prove that these surrogates are differentiable everywhere while retaining strict linear $H$-consistency bounds, effectively combining the optimization benefits of smoothness with the statistical efficiency of margin-based losses. In the structured prediction setting, we show that this smoothness unlocks a massive computational and energy advantage: it allows for an unbiased stochastic gradient estimator that bypasses the quadratic complexity $O(|\mathcal{Y}|^2)$ of exact inference (e.g., Viterbi). Empirically, our method achieves a *23$\times$ speedup* over Structured SVMs on large-vocabulary sequence tagging tasks and demonstrates superior robustness to instance-dependent label noise, outperforming Cross-Entropy by *2.6\%* on corrupted CIFAR-10.

Probabilistic Methods · Everything Else

Anton Conrad, Eric Moulines, julien perez

In multi-target regression and multi-class classification, uncertainty is inherently multivariate: prediction regions must capture joint dependencies across correlated outputs. Conformal prediction provides distribution-free guarantees, yet extending it to vector-valued outputs remains challenging—scalar aggregation discards geometric structure, while optimal transport (OT) approaches are computationally demanding and sensitive to outliers. We introduce two conformal methods based on geometric quantiles and spatial ranks: Geometric Conformalized Quantile Regression (GCQR) constructs prediction regions from learned conditional geometric quantiles, while Geometric Rank Conformal Prediction (GRCP) uses the radial rank of vector-valued conformity scores as the nonconformity measure. We propose multiple estimators offering different tradeoffs between computational cost and adaptivity to feature-dependent heterogeneity, with scalable learning via partially input-convex neural networks. On multi-target regression and multi-class classification benchmarks, GCQR and GRCP attain near-nominal coverage with consistently tighter prediction regions than scalarized and multivariate baselines.

Probabilistic Methods · Bayesian Models and Methods

Zier Mensch, Lars Holdijk, Samuel Duffield, Maxwell Aifer, Patrick Coles, Max Welling, Miranda C. N. Cheng

Stochastic-gradient MCMC methods enable scalable Bayesian posterior sampling but often suffer from sensitivity to minibatch size and gradient noise. To address this, we propose Stochastic Gradient Lattice Random Walk (SGLRW), an extension of the Lattice Random Walk discretization. Unlike conventional Stochastic Gradient Langevin Dynamics (SGLD), SGLRW introduces stochastic noise only through the off-diagonal elements of the update covariance; this yields greater robustness to minibatch size while retaining asymptotic correctness. Furthermore, as comparison we analyze a natural analogue of SGLD utilizing gradient clipping. Experimental validation on Bayesian regression and classification demonstrates that SGLRW remains stable in regimes where SGLD fails, including in the presence of heavy-tailed gradient noise, and matches or improves predictive performance.

Applications · Neuroscience, Cognitive Science

Jiayu Lu, Yujin Wang, Xiaofeng Liu, Dandan Li, Bin Wang

Functional brain network analysis plays an important role in understanding and diagnosing psychiatric disorders. However, current methods struggle with subject variations, impairing the model’s generalization ability to the test set. To address this issue, we propose the Subject Invariance-aware Inverse Graph Contrastive Learning (SI-IGCL) model, which adopts a two-stage paradigm with self-supervised subject-invariant pre-training followed by supervised fine-tuning for identification. During the pre-training phase, we construct an inverse contrastive objective that reshapes the embedding space by repelling intra-subject and attracting inter-subject embeddings to learn subject-invariant representations, with an auxiliary correction term to avoid early optimization plateaus. Meanwhile, we incorporate a structure-preserving reconstruction constraint to preserve discriminative information. Moreover, a Hierarchical Topology Enhanced Transformer (HTET) module is designed to enable multi-level modeling of subject-invariant functional patterns. During the fine-tuning phase, a supervised classifier is integrated to perform psychiatric disorder classification. Extensive experiments demonstrate that our method outperforms all state-of-the-art methods. The code is available at https://anonymous.4open.science/r/SI-IGCL.

General Machine Learning · Scalable Algorithms

Nasib Ullah, Jinbin Zhang, Jean Lucien Randrianantenaina, Erik Schultheis, Rohit Babbar

Extreme multi-label classification (XMC) involves learning deep learning models over large output spaces with millions of labels, making the output layer of the network a major bottleneck in memory and compute. While sparsity-based methods reduce arithmetic complexity, they often fail to yield proportional wall-clock gains due to irregular memory access, poor hardware utilization, or reliance on auxiliary architectural components in extreme long-tailed regimes. We introduce group-shared fixed fan-in sparsity, a semi-structured output-layer design in which groups of semantically related labels share a common sparse input pattern while retaining independent weights. This grouping introduces a task-aligned inductive bias---encouraging related labels to attend to similar feature subsets---while simultaneously reducing index memory overhead, increasing feature reuse across labels, and enabling efficient GPU execution via custom CUDA kernels that leverage modern accelerator primitives. As an alternative to auxiliary objectives, we exploit the long-tailed structure of XMC datasets by decomposing the output layer into a small dense head over frequent labels and a group-shared sparse tail over the remainder, providing an informative gradient pathway while preserving the memory benefits of sparsity. Through kernel-level microbenchmarking, we show that group-shared fixed fan-in converts reductions in arithmetic complexity into proportional wall-clock gains, achieving up to $4.4\times$ speedup in the forward pass and up to $25\times$ speedup in backward passes compared to standard fixed fan-in sparsity, while operating within a few percent of a FLOPs-matched dense bottleneck. Across large-scale XMC benchmarks, our approach matches or improves precision@k compared to prior sparse baselines, while substantially narrowing the performance gap to dense.

Applications · Computer Vision

Zhenggang Wang, Wang Wu, Lianghuazhe, Tai-Xiang Jiang

Pansharpening fuses high-resolution panchromatic (PAN) images with low-resolution multispectral (LMS) images. For spatial-spectral fusion, Fast Fourier Transform (FFT)-based methods provide a global receptive field to capture long-range dependencies and naturally separate frequency components. However, most existing approaches directly transplant spatial operators like convolution or self-attention, while disregarding the fundamental structure of the spectrum: a strict spatial correspondence where each coordinate represents a specific frequency component, and a highly non-uniform, radially decaying energy distribution. To address this, we revisit the classical Butterworth filter, a frequencydomain operator defined directly on spectral coordinates that is inherently suited for processing such structured representations. We generalize the standard isotropic Butterworth filter into an anisotropic, learnable frequency-domain gating mechanism, establishing an efficient alternative to self-attention, and propose the Anisotropic Butterworth Fusion Network (ABFNet). Its core is a novel dual-branch gating module that employs learnable anisotropic Butterworth filters to perform adaptive direction-aware feature selection, integrating global context and local details with linear complexity. Extensive experiments show that ABFNet achieves state-of-the-art (SOTA) performance on pansharpening benchmarks with low computational overhead. Furthermore, its superior accuracy on CIFAR-100 classification validates the broader applicability of this frequency-domain learning paradigm.

Theory · Optimization

Ioannis Anagnostides, Rohan Chauhan, Ioannis Panageas, Tuomas Sandholm, Jingming Yan

Performative prediction captures the phenomenon where deploying a predictive model shifts the underlying data distribution. While simple retraining dynamics are known to converge linearly when the performative effects are weak ($\rho < 1$), the complexity in the regime $\rho > 1$ was hitherto open. In this paper, we establish a sharp phase transition: computing an $\epsilon$-performatively stable point is PPAD-complete---and thus polynomial-time equivalent to Nash equilibria in general-sum games---even when $\rho = 1 + O(\epsilon)$. This intractability persists even in the ostensibly simple setting with a quadratic loss function and linear distribution shifts. One of our key technical contributions is to extend this PPAD-hardness result to general convex domains, which is of broader interest in the complexity of variational inequalities. Finally, we address the special case of strategic classification, showing that computing a strategic local optimum is PLS-hard.

Deep Learning · Robustness

Hang Ren, Xin Wang, Tong Yue, Chen Wen, Junqing Le

Dataset Distillation (DD) has emerged as a promising technique for compressing large-scale datasets into compact synthetic sets while preserving model performance. However, the security implications of this paradigm, particularly within the Transformer-based text classification domain, remain underexplored. In this paper, we identify "Distilled Attention Labels" as a pivotal yet overlooked vulnerability. We propose Attention Hijacking (AH), a stealthy backdoor attack that manipulates the bi-level optimization process to explicitly hijack the attention mechanism of target models via synthetic data. Distinct from traditional poisoning that often compromises clean accuracy, AH achieves stealthiness without utility degradation. To explain this, we formulate the "Semantic Anchoring Hypothesis", characterizing the interaction between trigger semantics and attack mechanisms. We demonstrate that AH functions as a semantic-adaptive mechanism: when triggers align with domain-specific semantic anchors (e.g., "film" in sentiment analysis), our method achieves a synergistic effect, boosting both Attack Success Rates (ASR >99%) and Clean Test Accuracy (CTA). Conversely, for functional or noise triggers, AH enforces attention segregation to prevent utility collapse, maintaining exceptional robustness where baseline attacks fail. Extensive experiments across multiple datasets and varying model scales—ranging from BERT-Tiny to BERT-Base—validate the scalability and dominance of AH. Our findings reveal that attention-based distillation is a double-edged sword, underscoring the urgent need for robust defenses in the era of data-efficient learning.