论文检索

Deep Learning · Graph Neural Networks

Baoheng Zhu, Deyu Bo, Delvin Zhang, Xiao Wang

Graph generation is a fundamental task with broad applications, such as drug discovery. Recently, discrete flow matching-based graph generation, \aka, graph flow model (GFM), has emerged due to its superior performance and flexible sampling. However, effectively aligning GFMs with complex human preferences or task-specific objectives remains a significant challenge. In this paper, we propose Graph-GRPO, an online reinforcement learning (RL) framework for training GFMs under verifiable rewards. Our method makes two key contributions: (1) We derive an analytical expression for the transition probability of GFMs, replacing the Monte Carlo sampling and enabling fully differentiable rollouts for RL training; (2) We propose a refinement strategy that randomly perturbs specific nodes and edges in a graph, and regenerates them, allowing for localized exploration and self-improvement of generation quality. Extensive experiments on both synthetic and real datasets demonstrate the effectiveness of Graph-GRPO. With only 50 denoising steps, our method achieves 95.0\% and 97.5\% Valid-Unique-Novelty scores on the planar and tree datasets, respectively. Moreover, Graph-GRPO achieves state-of-the-art performance on the molecular optimization tasks, outperforming graph-based and fragment-based RL methods as well as classic genetic algorithms.

Optimization · Discrete and Combinatorial Optimization

Zubin Zheng, Jiahao Wu, Shengcai Liu

The quantum approximate optimization algorithm (QAOA) holds promise for combinatorial optimization but is constrained by limited qubits. While divide-and-conquer frameworks like QAOA$^2$ address scalability by partitioning graphs into subgraphs, existing methods suffer from two fundamental limitations: i) misalignment between heuristic partitioning metrics and quantum optimization goals, and ii) topology-blind parameter initialization that leads to optimization cold starts. To bridge these gaps, we propose **Neural QAOA$^2$**, an end-to-end differentiable framework that jointly generates graph partitions and initial parameters. By integrating a generative evaluative network (GEN), our method utilizes a differentiable quantum evaluator as a high-fidelity performance surrogate to provide direct gradient guidance, enabling the joint generator to learn the intrinsic mapping from graph topology to high-quality partition and parameter configurations. Extensive experiments on 183 QUBO, Ising, and MaxCut instances (21 to 1000 variables) demonstrate that our gradient-driven approach significantly outperforms heuristic baselines, ranking first on 101 instances. It exhibits zero-shot generalization across out-of-distribution graph topologies and scales.

General Machine Learning · Methodology

Yang Zhao, Jungeun Lee, Jeong hwan Jeon, Sze Zheng Yong

We present a novel framework for embedding hard constraint satisfaction into neural network (NN) architectures, specifically feedforward neural networks and transformers, with input-dependent affine constraints of arbitrary cardinality. Traditional constraint enforcement approaches either rely on penalty-based soft constraints, which offer no guarantee of satisfaction, or on post-processing methods that enforce constraints after the NN is trained, which may lead to suboptimality. We introduce a trainable constraint-affine (CAffine) layer into NNs, yielding CAffNet, which goes beyond enforcing affine constraints via fixed orthogonal or parallel projections and enables joint optimization with network parameters. Moreover, we impose no restrictions on the constraint space dimensions and establish that our construction preserves the universal approximation properties of NNs, while providing provable guarantees on constraint adherence for all inputs. Experimental validation demonstrates robust performance across diverse domains requiring guaranteed constraint satisfaction.

General Machine Learning · Online Learning, Active Learning and Bandits

Humaira Kousar, Hasnain Irshad Bhatti, Jaekyun Moon

Efficient data selection is crucial for enhancing the training efficiency of deep neural networks and minimizing annotation requirements. Traditional methods often face high computational costs, limiting their scalability and practical use. We introduce PruneFuse, a novel strategy that leverages pruned networks for data selection and later fuses them with the original network to optimize training. PruneFuse operates in two stages: First, it applies structured pruning to create a smaller pruned network that, due to its structural coherence with the original network, is well-suited for the data selection task. This small network is then trained and selects the most informative samples from the dataset. Second, the trained pruned network is seamlessly fused with the original network. This integration leverages the insights gained during the training of the pruned network to facilitate the learning process of the fused network while leaving room for the network to discover more robust solutions. Extensive experimentation on various datasets demonstrates that PruneFuse significantly reduces computational costs for data selection, achieves better performance than baselines, and accelerates the overall training process.

Deep Learning · Theory

Mansour ZOUBEIROU A MAYAKI

We develop a theory of generalization and scaling for Mixture-of-Experts (MoE) Transformers that cleanly separates active per-input capacity from routing combinatorics. Conditioning on fixed routing patterns and union-bounding across them, we obtain a sup-norm covering-number bound whose metric entropy scales with the active parameter budget and incurs a MoE-specific overhead. Combining this with a standard ERM argument for squared loss we provided a generalization bound under a $d$-dimensional manifold model ($d$ is the intrinsic dimension of the training data) and $C^\beta$ targets, showing that approximation and estimation trade off in the same way as in dense networks once active parameters are counted appropriately. We further prove a constructive approximation theorem for MoE architectures, demonstrating that accuracy can be improved either by scaling active capacity or by increasing the number of available experts, with the better of the two mechanisms prevailing. From these results we derive neural scaling laws, covering model scaling, data scaling and compute–optimal tradeoffs. The theory highlights that enlarging the expert pool at fixed sparsity influences performance only through a mild logarithmic routing term, whereas increasing active capacity per input drives the main gains in generalization and approximation. These insights provide principled guidance for the design of efficient sparse Transformer systems and clarify the fundamental tradeoffs underlying their empirical scaling behavior.

Yifei Li, Haixu Wu, Zeyi Xu, Tuur Stuyck, Wojciech Matusik

Learning-based methods have made significant progress in physics simulation, typically approximating dynamics with a monolithic end-to-end optimized neural network. Although these models offer an effective way to simulation, they may lose essential features compared to traditional numerical simulators, such as physical interpretability and reliability. Drawing inspiration from classical simulators that operate in a modular fashion, this paper presents Neural Modular Physics (NMP) for elastic simulation, which combines the approximation capacity of neural networks with the physical reliability of traditional simulators. Beyond the previous monolithic learning paradigm, NMP enables direct supervision of intermediate quantities and physical constraints by decomposing elastic dynamics into physically meaningful neural modules connected through intermediate physical quantities. With a specialized architecture and training strategy, our method transforms the numerical computation flow into a modular neural simulator, achieving improved physical consistency and generalizability. Experimentally, NMP demonstrates superior generalization to unseen initial conditions and resolutions, stable long-horizon simulation, better preservation of physical properties compared to other neural simulators, and greater feasibility in scenarios with unknown underlying dynamics than traditional simulators.

Social Aspects · Accountability, Transparency, and Interpretability

Mateo Espinosa Zarlenga

Concept-based models (CMs), deep neural networks that ground their predictions on representations aligned with human-understandable concepts (e.g., "round", "stripes", etc.), have been shown to learn representations that *leak* concept-irrelevant information. As the traditional narrative goes, this leakage is undesirable and should be eradicated as it leads to uninterpretable models. In this paper, we posit that this conventional view of leakage in CMs is not only ill-posed, as the evidence of how leakage makes a model less interpretable is often inconclusive, but also bound to lead to impractical CMs under common real-world constraints. Specifically, we argue that *in real-world settings where concept incompleteness is the norm, some leakage is often necessary for constructing accurate and intervenable CMs*. To this end, we propose that there is such a thing as *benign* leakage and show that, by optimizing a reframing of the typical CM training objective, CMs can encourage and exploit this form of leakage without sacrificing accuracy or intervenability.

General Machine Learning · Representation Learning

Ali Kayyam

Whether deep vision models recognize objects primarily by shape or texture remains a central and unresolved question in computer vision. Early studies report a strong texture bias in convolutional neural networks (CNNs), while other work reports shape-biased representations. We argue that much of this apparent discrepancy reflects methodological confounds and a conflation of local contour sensitivity with genuine global shape understanding. Using minimal, tightly controlled stimuli, we directly compare cue-conflict and cue-suppression paradigms within a unified experimental framework. We show that standard CNNs consistently prioritize texture over global shape when cues compete, even when shape information is explicitly available. Evidence for shape bias typically reflects reliance on local fragments rather than invariant, relational representations of object structure. Our findings support the view that texture bias is fundamentally rooted in architectural inductive biases rather than data or optimization alone. This gap has direct consequences for robustness, safety, and generalization, and motivates the development of architectures that explicitly support global integration and relational reasoning, moving beyond incremental data-driven fixes.

Theory · Deep Learning

Hang Zhou, Ju-Sheng Hong, Xiucai Ding, Jane-Ling Wang

Regression with functional covariates poses fundamental challenges due to the infinite-dimensional nature of functional data, and its theoretical properties have been studied under specialized frameworks in classical nonparametric statistics. While deep neural networks (DNNs) have demonstrated remarkable empirical success in high-dimensional regression, their theoretical behavior in settings involving infinite-dimensional covariates remains largely unexplored. In this work, we study the theoretical performance of DNN-based estimators for regression problems with functional covariates. We extend existing theoretical techniques, which are developed for finite-dimensional covariates supported on compact sets, to the infinite-dimensional and non-compact functional data setting. Under mild conditions, we show that DNN estimators attain minimax-optimal polynomial rates of convergence for both functional linear models and functional generalized linear models. For fully nonparametric regression with functional covariates, we establish a lower bound on the prediction error, and further discuss the fundamental obstacles inherent to this problem and their connections to existing state-of-the-art methods in the literature.

Deep Learning · Theory

Zixiong Yu, Guhan Chen, Jianfa Lai, Bohan Li, Songtao Tian

Scaling factors in residual branches have emerged as a prevalent method for boosting neural network performance, especially in normalization-free architectures. While prior work has primarily examined scaling effects from an optimization perspective, this paper investigates their role in residual architectures through the lens of generalization theory. Specifically, we establish that wide residual networks (ResNets) with constant scaling factors become asymptotically unlearnable as depth increases. In contrast, when the scaling factor exhibits rapid depth-wise decay combined with early stopping, over-parameterized ResNets achieve minimax-optimal generalization rates. To establish this, we demonstrate that the generalization capability of wide ResNets can be approximated by the kernel regression associated with a specific kernel. Our theoretical findings are validated through experiments on synthetic data and real-world classification tasks, including MNIST and CIFAR-100.

Theory · Probabilistic Methods

Laura Lützow, Michael Eichelbeck, Mykel Kochenderfer, Matthias Althoff

Conformal prediction is a popular uncertainty quantification method that augments a base predictor to return sets of predictions with statistically valid coverage guarantees. However, current methods are often computationally expensive and data-intensive, as they require constructing an uncertainty model before calibration. Moreover, existing approaches typically represent the prediction sets with intervals, which limits their ability to capture dependencies in multi-dimensional outputs. We address these limitations by introducing zono-conformal prediction, a novel approach inspired by interval predictor models and reachset-conformant identification that constructs prediction zonotopes with assured coverage. By placing zonotopic uncertainty sets directly into the model of the base predictor, zono-conformal predictors can be identified via a single, data-efficient linear program. While we can apply zono-conformal prediction to arbitrary nonlinear base predictors, we focus on feed-forward neural networks in this work. Aside from regression tasks, we also construct optimal zono-conformal predictors in classification settings where the output of an uncertain predictor is a set of possible classes. We provide probabilistic coverage guarantees and present methods for detecting outliers in the identification data. In extensive numerical experiments, we show that zono-conformal predictors are less conservative than interval predictor models and standard conformal prediction methods, while achieving a similar coverage over the test data.

Applications · Neuroscience, Cognitive Science

Ezekiel Williams, Alexandre Payeur, Guillaume Lajoie

Biological and neuromorphic recurrent neural networks (RNNs) are subject to spatial and temporal locality constraints on the information that can plausibly be used during learning. A common strategy to satisfy these constraints is to modify gradient descent by neglecting non-local terms to varying degrees, as in random feedback local online (RFLO) learning and truncated backpropagation through time (tBPTT). However, the learning dynamics of these algorithms, and how they compare with BPTT, remain poorly understood. Here, we apply dynamical systems theory to data-aligned linear RNNs to compare stationary solutions, stability properties, and convergence rates, finding a close relationship between BPTT and one-step tBPTT—in contrast with qualitatively distinct behavior for RFLO. We further observe that the solutions learned by RFLO are restricted to low-rank perturbations of initial parameters, a result which holds beyond the data-aligned setting. Our work provides fundamental insights into how locality constraints shape RNN learning dynamics, with implications for neuroscientific models of learning and alternative optimization approaches for state-space models.

Deep Learning · Graph Neural Networks

Yumeng Zhao, Huiying Hu, Steve Wen, Junjie Shen, Bei Hua

Graph Neural Networks (GNNs) are increasingly adopting the "pre-training, adaptation" paradigm, which first pre-train GNNs on large-scale unlabeled graph data and then adapt them to specific downstream tasks. As a common pattern, graph prompting adapts to the frozen encoder by modifying the input graph structure, rather than fine-tuning the model parameters. However, most existing graph prompting approaches optimize the continuous and weighted adjacency structure in the adaptation phase, while requiring a hard discretization at inference time. This difference causes a train-test mismatch which is particularly harmful in few-shot regimes. To address the issue, we propose **Di**screte **P**rompting for **G**raphs, a discrete prompting framework that directly learns task-specific topology prompts in the combinatorial space. DiP-G operates on multi-hop local candidate subgraphs to ensure scalability, generates hard (k)-sparse prompts through a perturbed Top-(k) solver, and optimizes the discrete structures using an I-MLE gradient estimator. To improve the efficiency of backward pass, we further introduce an adaptive active-set screening rule that accelerates the target solve and can provably maintain the accuracy of the solution. Extensive experiments conducted on multiple benchmark datasets have validated the effectiveness of our proposed method. Our main code is available in the supplementary materials.

Applications · Neuroscience, Cognitive Science

Abdülkadir Gökce, Yingtian Tang, Martin Schrimpf

Task-optimized neural networks are the leading in-silico models of sensory cortex, yet the field lacks a unified understanding of which modeling choices drive improved brain alignment. Prior NeuroAI work is fragmented across datasets and modalities, making it difficult to determine robust scaling trends. Here, we systematically investigate the scaling laws of model-to-brain alignment across 8 neural datasets (spanning electrophysiology, fMRI, EEG, and MEG) and over 600 models with diverse architectures and pretraining configurations. We report three key scaling trends: (1) *Pretraining saturation*: Alignment improves with pretraining compute and data scale but saturates across all recording modalities. (2) *Complementary fine-tuning*: Hybrid task & neural data optimization yields consistent improvements in alignment that generalize across datasets and modalities. (3) *Mapping scaling*: Increasing the number of neural samples to fit model-to-brain mappings yields log-linear gains with the largest impact on alignment. Finally, we propose a novel subject-shared cross-attention mapping which drastically reduces parameter count and improves alignment. Taken together, these results establish multimodal scaling laws that guide resource allocation for next-generation brain models.

Deep Learning · Everything Else

Yury Gorishniy, Akim Kotelnikov, Ivan Rubachev, Artem Babenko

Deep learning models for supervised learning on tabular data are rapidly improving. Notably, ensembles (mixtures of multiple models) often play an important role in achieving top performance, which motivates designing ensemble-first systems rather than treating ensembling as an ad hoc trick. In this work, we present TabPack --- a new ensembling approach that packs many base model-optimizer pairs with different hyperparameters into a single neural network and a single optimizer. The base model-optimizer hyperparameters are sampled randomly, after which all base models are trained in parallel, and the final ensemble is built on the fly during training. As a result, TabPack produces powerful ensembles in a single run, with substantial efficiency gains over traditional approaches. With its remarkable efficiency, strong performance on public benchmarks, and reduced reliance on traditional hyperparameter tuning, TabPack becomes an appealing solution for practitioners, and suggests a new avenue for designing better tabular deep learning systems.

Optimization · Convex

Aikaterini Karoni, Rajit Rajpal, Benedict Leimkuhler, Gabriel Stoltz

Momentum Stochastic Gradient Descent (mSGD) relies on a fixed momentum coefficient shared across all parameters, failing to account for the heterogeneous structure of modern loss landscapes. In this work, we adopt a continuous-time formulation to introduce individual, adaptive momentum coefficients regulated by the kinetic energy of each model parameter. This mechanism automatically adjusts to local curvature to maintain stability without sacrificing convergence speed. We demonstrate that this adaptive friction is inextricably linked to cubic damping, a suppression mechanism from structural dynamics. We additionally introduce two optimization schemes by augmenting the continuous dynamics of mSGD and Adam with a cubic damping term. Empirically, our methods demonstrate robustness and match or outperform Adam on training ViT, BERT, and GPT2; tasks where mSGD typically struggles. We further provide theoretical results establishing the exponential convergence of the proposed schemes.

Deep Learning · Large Language Models

Jianhao Huang, Baharan Mirzasoleiman

Masked Diffusion Language Models have recently emerged as a powerful generative paradigm, yet their generalization properties remain understudied compared to their auto-regressive counterparts. In this work, we investigate these properties within the setting of the $k$-parity problem (computing the XOR sum of $k$ relevant bits), where neural networks typically exhibit grokking—a prolonged plateau of chance-level performance followed by sudden generalization. We theoretically decompose the Masked Diffusion (MD) objective into a Signal regime which drives feature learning, and a Noise regime which serves as an implicit regularizer. By training nanoGPT using MD objective on the $k$-parity problem, we demonstrate that MD objective fundamentally alters the learning landscape, enabling rapid and simultaneous generalization without experiencing grokking. Furthermore, we leverage our theoretical insights to optimize the distribution of the mask probability in the MD objective. Our method significantly improves perplexity for 50M-parameter models and achieves superior results across both pre-training from scratch and supervised fine-tuning. Specifically, we observe performance gains peaking at $8.8$% and $5.8$%, respectively, on 8B-parameter models, confirming the scalability and effectiveness of our framework in large-scale masked diffusion language model regimes.

Deep Learning · Large Language Models

Zhi Hong, Qian Zhang, Jiahang Sun, Zhiwei Shang, Mingze Kong, Xiangyi Wang, Yao Shu, Zhongxiang Dai

Large Language Models (LLMs) have achieved significant success across a wide range of tasks, serving as the cognitive backbone for Multi-Agent Systems (MAS) designed to orchestrate complex practical workflows. Given that MAS performance is highly sensitive to input prompts and many deployment scenarios preclude MAS architecture modifications, prompt optimization emerges as a critical strategy for performance enhancement. However, real-world deployment is impeded by three key challenges: (1) the need for high sample efficiency due to prohibitive evaluation costs, (2) topology-induced coupling among prompts, and (3) the combinatorial explosion of the search space. To address these challenges, we introduce **MASPOB** (**M**ulti-**A**gent **S**ystem **P**rompt **O**ptimization via **B**andits), a novel sample-efficient framework based on bandits. By leveraging Upper Confidence Bound (UCB) to quantify uncertainty, the bandit framework balances exploration and exploitation, maximizing gains within a strictly limited budget. To handle topology-induced coupling, MASPOB integrates Graph Neural Networks (GNNs) to capture structural priors, learning topology-aware representations of prompt semantics. Furthermore, it employs coordinate ascent to decompose the optimization into univariate sub-problems, reducing search complexity from exponential to linear. Extensive experiments across diverse benchmarks demonstrate that MASPOB achieves state-of-the-art performance, consistently outperforming existing baselines.

Optimization · Discrete and Combinatorial Optimization

Yang Huang, Yifan Zhang, Jian Cheng

We propose an OT-guided sampling framework for solving graph combinatorial optimization through exact multilinear relaxation. Graph combinatorial optimization problems can be written as quadratic unconstrained binary optimization(QUBO). Leveraging a classical result in combinatorial optimization, we obtain a continuous multi-linear relaxation of QUBO that is exact, in the sense that it preserves the optimal binary solutions. The challenge is that the resulting energy landscape is highly nonconvex. We address this by treating the objective as an energy function and optimizing via sampling from the induced Boltzmann distribution to escape poor local optima. Viewing sampling as transporting a simple reference distribution to the target distribution, we use optimal transport to characterize more efficient probability flow and derive a stochastic optimal control problem whose solution yields an optimal sampling dynamics. We parameterize the control policy with graph neural networks to approximate the optimal control. Experiments show improved solution quality and efficiency over strong combinatorial and learning-based baselines.

Optimization · Convex

Junchi Yan, Liangliang Shi, Jiaxi Liu, Fangyuan Zhou, Wenzheng Pan, Zhongteng Gui, Yihui Tu

One essential limitation of neural networks is how to enforce (hard) constraints on prediction. We propose a plug-in, differentiable layer, which involves a fast implicit (convex) optimization procedure to enforce the general linear constraint. It aims to minimize a divergence between unconstrained and constrained outputs. Connecting to and beyond existing handcrafted layers, we show that our layer degrades to classic layers like Softmax, Sinkhorn and tanh etc. when the corresponding constraint is enforced by KL-divergence minimization. We further show that by replacing the KL-div with a Euclidean distance, a closed-form solution can be derived for highly-efficient constraint enforcing. We evaluate the above two variants of layers, termed as BLCLayer and GLCLayer, with their corresponding neural solver BLCNet and GLCNet with simple MLP/GNN-like backbone. Experiments on linear programming, as well as two real-world problems: partial graph matching and portfolio allocation which involve other discrete constraints.