论文检索

General Machine Learning · Causality

Yuxi Du, Zhiheng Zhang, Haoxuan Li, Cong Fang, Jixing Xu, Zhen Peng, Jiecheng Guo

Causal inference in modern large-scale systems faces growing challenges, including high-dimensional covariates, multi-valued treatments, massive observational (OBS) data, and limited randomized controlled trial (RCT) samples due to cost constraints. We formalize treatment-induced structural non-overlap and show that, under this regime, commonly used weighted fusion methods provably fail to satisfy randomized identifying restrictions.To address this issue,we propose a constrained joint estimation framework that minimizes observational risk while enforcing causal validity through orthogonal experimental moment conditions. We further show that structural non-overlap creates a feasibility obstruction for moment enforcement in the original covariate space.We also derive a penalized primal–dual algorithm that jointly learns representations and predictors, and establish oracle inequalities decomposing error into overlap recovery, moment violation, and statistical terms.Extensive synthetic experiments demonstrate robust performance under varying degrees of non-overlap. A large-scale ride-hailing application shows that our method achieves substantial gains over existing baselines, matching the performance of models trained with significantly more RCT data.

General Machine Learning · Causality

YiXin Ren, Hongquan Liu, Juncai Zhang, Yewei Xia, Zichuan Lin, Deheng Ye, Hao Zhang, Jihong Guan, Shuigeng Zhou

In this paper, we present a novel federated independence testing method that addresses both theoretical and practical challenges arising from client heterogeneity. We begin by revisiting existing federated independence testing methods and showing why they fail to provide valid guarantees or maintain statistical power under data distributional shift across clients. Building on this analysis, we develop a copula-based marginal alignment technique together with a stacking-based aggregation strategy that amplifies intra-client dependence while mitigating inter-client variation, resulting in a theoretically sound and powerful global test. For practicality, we further accelerate the aggregation step and incorporate a privacy-preserving mechanism. On the theoretical side, we prove both the correctness of our method and the validity of the test. Empirically, we conduct extensive experiments on both synthetic and real-world datasets, which demonstrate the superiority of our solution over existing methods.

Xinyi He, Qian Liu, Mingzhe Du, Lin Yan, ZhiJie Fan, Yiming Huang, Yin Zheng, Zejian Yuan, Zejun MA

Code performance optimization is paramount in real-world software engineering and critical for production-level systems. While Large Language Models (LLMs) have demonstrated impressive capabilities in code generation and bug fixing, their proficiency in enhancing code performance at the repository level remains largely unexplored. To address this gap, we introduce SWE-Perf, the first benchmark specifically designed to systematically evaluate LLMs on code performance optimization tasks within authentic repository contexts. SWE-Perf comprises 140 carefully curated instances, each derived from performance-improving pull requests from popular GitHub repositories. Each benchmark instance includes the relevant codebase, target functions, performance-related tests, expert-authored patches, and executable environments. Through a comprehensive evaluation of representative methods that span file-level and repo-level approaches (e.g., Agentless and OpenHands), we reveal a substantial capability gap between existing LLMs and expert-level optimization performance, highlighting critical research opportunities in this emerging field.

Deep Learning · Generative Models and Autoencoders

Hyoseok Lee, Sohwi Lim, Eunju Cha, Tae-Hyun Oh

While latent diffusion models (LDMs) have emerged as powerful priors for inverse problems, existing LDM-based solvers frequently suffer from instability. In this work, we first identify the instability as a discrepancy between the solver dynamics and stable reverse diffusion dynamics learned by the diffusion model, and show that reducing this gap stabilizes the solver. Building on this, we introduce *Measurement-Consistent Langevin Corrector (MCLC)*, a theoretically grounded plug-and-play stabilization module that remedies the LDM-based inverse problem solvers through measurement-consistent Langevin updates. Compared to prior approaches that rely on linear manifold assumptions, which often fail to hold in latent space, MCLC provides a principled stabilization mechanism, leading to more stable and reliable behavior in latent space.

General Machine Learning · Causality

Yi Wan, Xin Wang, Huanhuan Chen

In domains such as healthcare and marketing, learning optimal individualized dosing policies to maximize utility is crucial, yet high experimental costs impose strict budget constraints, necessitating efficient active policy learning. Existing active learning methods in causal inference primarily focus on binary treatments and effect estimation, leaving continuous dosing and policy optimization underexplored. To address this gap, we propose an active learning framework tailored for optimal policy learning. Exploiting the inherent structure of dose-response curves, we theoretically show that the policy optimization regret is bounded by the expected posterior gradient variance at the estimated optimal doses. Motivated by this result, we introduce Gradient Variance Active Learning for Individualized Dosing (GVALID), a batch acquisition strategy that greedily selects samples to minimize target gradient variance for efficient policy learning. Experiments demonstrate that GVALID achieves superior performance under strict budget constraints.

General Machine Learning · Causality

Robert Ganian, Marlene Gründel, Simon Wietheger

Pearl’s Causal Hierarchy (PCH) is a central framework for reasoning about probabilistic, interventional, and counterfactual statements, yet the satisfiability problem for PCH formulas is computationally intractable in almost all classical settings. We revisit this challenge through the lens of parameterized complexity and identify the first gateways to tractability. Our results include fixed-parameter and XP-algorithms for satisfiability in key probabilistic and counterfactual fragments, using parameters such as primal treewidth and the number of variables, together with matching hardness results that map the limits of tractability. Technically, we depart from the dynamic programming paradigm typically employed for treewidth-based algorithms and instead exploit structural characterizations of well-formed causal models, providing a new algorithmic toolkit for causal reasoning.

Applications · Social Sciences

Kunal Pradeep Pimparkhede, Chirayu Chaurasia, Jatin Roy, Mahesh Mohan Mohanachandran Radhamany

Responsible investing aims to generate positive impact across Environment (E), Society (S), and Governance (G), and rating companies along these dimensions is now widespread, making ESG scores highly popular. Allocating retail capital with sustainability in mind could be transformational, yet it remains unclear how individual investors can do so in practice. Current ESG solutions cannot model high-dimensional, multi-modal time series capturing the joint evolution of ESG risks, financial returns, news, and sentiment, even though this domain requires jointly reasoning over distinct numerical signals where both numerical proximity and semantic type must be preserved. To bridge this gap, we introduce a novel domain-aware $\textbf{representation learning framework}$ that learns geometry-preserving representations for heterogeneous time series using value-aware tokens with block-wise $\textbf{orthogonal embeddings}$. To capture trajectory-level structure, we introduce $\textbf{FACET}$ tokens and train the model using a geometry-preserving loss. The resulting model jointly learns to forecast future values and to organize entities in a representation space that reflects their temporal evolution. Trained on ESG, returns, news, and sentiment, the domain-aware LLM learns a representation space that enables accurate ESG forecasting, trajectory-based grouping, and latent-space search for superior asset selection and downstream application like portfolio rebalancing

Deep Learning · Large Language Models

Rachel Lawrence, Jacqueline Maasch

Autonomous reasoning is among the most scientifically and economically motivating topics in AI today. Historically the purview of symbolic AI, recent advances have mainly emerged from deep probabilistic generative models. Despite immense interest and rapid progress, the generative AI community has not clearly converged on operational definitions for reasoning and often implicitly rejects the historical treatment of this topic in logic, verifiable automated reasoning, and symbolic methods in general. **This position contends that definitional ambiguity leaves the construct validity of reasoning evaluation unverifiable, and undermines quantifiable progress toward the collective goal of trustworthy autonomous reasoning.** We also contend that this ambiguity is addressable. To that end, we provide (1) general and extensible definitions for *valid* and *sound reasoning* based on a synthesis of the literature, which can serve as an accessible reference and a starting point for community discussion; and (2) a checklist for best practices in the communication of AI reasoning research.

Optimization · Large Scale, Parallel and Distributed

Chengjie Ma, Seungeun Oh, Jihong Park, Seong-Lyun Kim

Federated learning (FL) enables distributed model training, yet in heterogeneous deployments, Bandwidth-Constrained Clients (BCCs) often contribute inefficiently due to limited uplink bandwidth. In model-heterogeneous FL with fixed small sub-models, BCCs with sub-models may improve quickly in early rounds but become under-parameterized later, resulting in slow convergence and poor generalization. To address this challenge, we propose FedGMR, a federated learning framework centered around Gradual Model Restoration (GMR), where GMR progressively increases each client’s sub-model density during training, allowing BCCs to remain effective contributors throughout optimization. To make GMR practical under real-world heterogeneity, FedGMR is realized as an end-to-end workflow with asynchronous coordination and stable, mask-aware aggregation. We further establish convergence guarantees, showing that the aggregation error scales with the average sub-model density across clients and rounds, and that GMR provably narrows the gap toward full-model FL. Extensive experiments on FEMNIST, CIFAR-10, ImageNet-100, and StackOverflow demonstrate that FedGMR improves both convergence speed and final accuracy, especially under severe heterogeneity and non-IID data distributions.

Deep Learning · Large Language Models

Haiyu Wang, Yuanyuan Lin

Recent studies have revealed two intriguing phenomena in large language models: massive activations, characterized by a small number of activations exhibiting abnormally large magnitudes, and attention sink, where a disproportionate amount of attention is consistently allocated to specific tokens regardless of their semantic relevance. However, the co-emergence and co-existence of these two phenomena remain poorly understood. In this work, we revisit the prevailing view that massive activations are the primary mechanism responsible for concentrating attention on sink tokens, and provide a more nuanced interpretation of their relationship. Through both theoretical analysis and empirical evidence, we demonstrate that massive activations and attention sink jointly act to prevent excessive token mixing in self-attention. Specifically, attention sink suppresses mixing among non-sink tokens, whereas massive activations suppress mixing between sink tokens and non-sink tokens. Furthermore, our theory provides a principled explanation of how the location of massive activations depends on the placement of layer normalization, and why KV-biases and gating mechanisms can remove massive activations while largely preserving attention sink. We further conduct intervention analyses and find that removing the value vector of the sink token can recover attention sink even when massive activations are entirely suppressed. Overall, this work provides a mechanistic perspective on how massive activations and attention sink interact under normalization and self-attention, offering new insights into their functional roles in Transformer models.

Applications · Computer Vision

Weirong Chen, Keisuke Tateno, Hidenobu Matsuki, Michael Niemeyer, Daniel Cremers, Federico Tombari

We address the problem of 4D reconstruction from partial point cloud sequences, where observations from depth sensors are incomplete, unordered, and lack explicit point correspondence over time. Recovering coherent 4D geometry in this geometry-only setting is challenging due to missing observations and ambiguous dynamics. While recent progress has largely been driven by image-based methods, existing point-based approaches typically focus on single-object scenarios, assume relatively complete inputs, and rely on explicit correspondence. To mitigate these limitations, we propose DynaTok, a point-based framework for correspondence-free 4D reconstruction from partial point cloud sequences that operates without images. DynaTok encodes each frame into compact latent tokens, aggregates incomplete observations over time with a Transformer-based spatiotemporal encoder, and decouples geometry and motion via a residual token design within a single unified model. Conditioned on the aggregated tokens, a point flow-matching decoder reconstructs complete and temporally consistent 4D point cloud sequences using only point cloud supervision. Experiments on object-level and scene-level benchmarks demonstrate improved reconstruction quality and temporal coherence under partial point cloud observations.

Applications · Chemistry, Physics, and Earth Sciences

Gnankan Landry Regis N'guessan, Bum Jun Kim

Physical systems near singularities, interfaces, and critical points exhibit power-law scaling, yet standard neural networks leave the governing exponents implicit. We introduce physics-informed M\"untz-Sz\'asz Networks (MSN-PINN), a power-law basis network that treats scaling exponents as trainable parameters. The model outputs both the solution and its scaling structure. We prove identifiability, or unique recovery, and show that, under these conditions, the squared error between learned and true exponents scales as $O(|\mu - \alpha|^2)$. Across experiments, MSN-PINN achieves single-exponent recovery with 1--5\% error under noise and sparse sampling. It recovers corner singularity exponents for the two-dimensional Laplace equation with 0.009\% error, matches the classical result of Kondrat'ev (1967), and recovers forcing-induced exponents in singular Poisson problems with 0.03\% and 0.05\% errors. On a 40-configuration wedge benchmark, it reaches a 100\% success rate with 0.022\% mean error. Constraint-aware training encodes physical requirements such as boundary condition compatibility and improves accuracy by three orders of magnitude over naive training. By combining the expressiveness of neural networks with the interpretability of asymptotic analysis, MSN-PINN produces learned parameters with direct physical meaning.

Reinforcement Learning · Policy Search

Jiaqi Wang, Haoge Deng, Ting Pan, Yang Liu, Chengyuan Wang, Fan Zhang, Yonggang Qi, Xinlong Wang

Uniform Discrete Diffusion (UDM) has recently emerged as a promising paradigm for discrete generative modeling; however, its integration with reinforcement learning remains largely unexplored. We observe that naively adapting GRPO to UDM leads to unstable training and marginal performance. To address this, we propose \Ours, the first framework that integrates UDM with RL. Our method is guided by two key insights: (i) treating the final clean sample, rather than intermediate predicted sample, as the action provides more accurate and stable optimization signals; and (ii) adopting the forward process to reconstruct the training trajectories helps the model learn probability paths that are more consistent with pretraining. For efficiency, we introduce Reduction-Step and CFG-Free training strategies. \Ours significantly improves the performance of the base model across multiple T2I tasks. Notably, GenEval accuracy improves from $69\%$ to $96\%$ and PickScore increases from $20.46$ to $23.81$, achieving state-of-the-art performance in both continuous and discrete settings. On the OCR benchmark, accuracy improves from $4\%$ to $57\%$, further validating the effectiveness and generalization capability of our method.

General Machine Learning · Sequential, Network, and Time Series Modeling

Nitin Bisht, Xiuwen Gong, Huan Huo, Guandong Xu

Modern recommender systems are typically trained offline and deployed with parameters held fixed between periodic refreshes, yet user behavior can evolve substantially during deployment. This can cause ranking utility to degrade over time and makes it difficult to provide formal guarantees about recommendation quality. We propose **CARE**, an adaptive calibration framework that wraps an arbitrary backbone recommender and outputs variable-size recommendation sets with finite-sample performance guarantees over interaction streams. CARE combines (i) a loss-based monitoring module that localizes behavioral changes and triggers threshold recalibration, and (ii) an online aggregation rule that promotes compact recommendation sets by dynamically reweighting candidate set predictors. We provide theoretical results establishing finite-sample guarantees for utility-based risk control and bounds on the expected set size relative to the best constituent predictor. Experiments across multiple datasets and backbone models demonstrate that CARE improves robustness and maintains compact recommendation sets while preserving the desired statistical guarantees. The code and implementation are available in https://anonymous.4open.science/r/CARE-FCBD.

General Machine Learning · Online Learning, Active Learning and Bandits

Nitin Bisht, Linjiang Guo, Xiuwen Gong, Huan Huo, Guandong Xu

Recommender systems (RS) may suffer from feedback-driven exposure concentration, where repeated engagement optimization collapses exposure onto a narrow set of categories, reducing catalog coverage and degrading long-horizon learning. Existing methods are often post hoc and typically lack principled uncertainty-aware risk estimates for regulating exposure under endogenous feedback. We therefore propose **CORAL**, a model-agnostic, uncertainty-aware framework that formulates exposure regulation as a constrained sequential decision problem. Specifically, we model self-reinforcing interactions to construct an exposure-saturation state, then derive an upper confidence bound on category-conditioned violation risk from observed history and incorporate it through a state-dependent penalty for adaptive intervention near saturation. Moreover, we provide theoretical guarantees for risk bounds, finite-time recovery, and efficient long-term performance. Extensive experiments on real-world datasets and controlled simulations validate the effectiveness of the proposed framework, which aligns with our theoretical analysis. Our code is available at: https://anonymous.4open.science/r/Coral_Rec-8400.

Social Aspects · Accountability, Transparency, and Interpretability

Guanzheng Qin, Chenghao Sun, Zhining Xie, Xinmei Tian

Large Language Models (LLMs) demonstrate remarkable potential in role-playing tasks but frequently suffer from personality decay—termed "Out-of-Character" (OOC) behavior—during prolonged interactions. While heuristic strategies exist to align model behaviors, the internal computational dynamics driving personality expression remain opaque. A fundamental barrier to decoding these mechanisms is a *metric gap*: while standard causal attribution paradigms target atomic, single-token outcomes, personality manifests as a holistic, multi-token behavioral tendency. We bridge this gap via the *Latent Persona Vector*, a differentiable proxy enabling the first fine-grained causal tracing of personality circuits. This metric reveals a structured "Preparation-Establishment-Expression" dynamic and identifies the mechanistic root of OOC behavior not as knowledge erasure, but as *generic prior dominance*. Specifically, we find that intrinsic assistant priors suppress emergent persona intents during the critical "Establishment" phase. Guided by this diagnosis, we propose surgically recalibrating the signal magnitude in fewer than 5% of attention heads. This targeted intervention effectively counteracts prior suppression, significantly restoring character consistency while preserving general reasoning capabilities.

Theory · Optimization

Yi Feng, Weiming Ou, Xiao Wang

The remarkable success of the Adam in training neural networks has naturally led to the widespread use of its descent-ascent counterpart, Adam-DA, for solving zero-sum games. Despite its popularity in practice, a rigorous theoretical understanding of Adam-DA still lags behind. In this paper, we derive ordinary differential equations (ODEs) that serve as continuous-time limits of the Adam-DA. These ODEs closely approximate the discrete-time dynamics of Adam-DA, providing a tractable analytical framework for understanding its behavior in zero-sum games. Using this ODE approach, we investigate two fundamental aspects of Adam-DA: local convergence and implicit gradient regularization. Our analysis reveals that the roles of the first- and second-order momentum parameters in zero-sum games are exactly the opposite of their well-documented effects in minimization problems. We validate these predictions through GAN experiments across multiple architectures and datasets, demonstrating the practical implications of this reversed momentum effect.

Deep Learning · Large Language Models

Gal Yona, Mor Geva, Yossi Matias

Despite significant improvements in factuality, confident errors continue to reappear as benchmarks probe more niche knowledge that models lack. We argue that most gains have come from expanding the model's knowledge boundary (encoding more facts) rather than improving awareness of that boundary (distinguishing known from unknown). We conjecture that this stems from the fact that the latter is inherently difficult: in the absence of strong ability to separate correct from incorrect answers (discrimination), fully eliminating hallucinations requires aggressive abstention, imposing a significant utility tax. Given this limitation, we propose complementing knowledge expansion with faithful uncertainty -- honestly conveying whatever uncertainty remains. This metacognitive capability becomes even more critical for tool-augmented models, where it serves as the control layer that determines when to search and how to weigh conflicting information. We conclude by highlighting the key challenges and open problems that must be tackled to make progress toward this objective.

Theory · Learning Theory

Mehryar Mohri, Yutao Zhong

The choice of loss function in classification involves a fundamental trade-off: smooth losses (like Cross-Entropy) enable fast optimization rates but yield slow square-root consistency bounds, while piecewise-linear losses (like Hinge) offer fast linear consistency rates but suffer from non-differentiability. We propose *Linear-Core (LC) Surrogates*, a new family of convex loss functions that resolve this tension by stitching a linear core to a smooth tail. We prove that these surrogates are differentiable everywhere while retaining strict linear $H$-consistency bounds, effectively combining the optimization benefits of smoothness with the statistical efficiency of margin-based losses. In the structured prediction setting, we show that this smoothness unlocks a massive computational and energy advantage: it allows for an unbiased stochastic gradient estimator that bypasses the quadratic complexity $O(|\mathcal{Y}|^2)$ of exact inference (e.g., Viterbi). Empirically, our method achieves a *23$\times$ speedup* over Structured SVMs on large-vocabulary sequence tagging tasks and demonstrates superior robustness to instance-dependent label noise, outperforming Cross-Entropy by *2.6\%* on corrupted CIFAR-10.

Theory · Learning Theory

Mingyue Xu, Gal Vardi, Itay Safran

We study *grokking* - the onset of generalization long after overfitting - in a classical ridge regression setting. We prove end-to-end grokking results for learning over-parameterized linear regression models using gradient descent with weight decay. Specifically, we prove that the following stages occur: (i) the model overfits the training data early during training; (ii) poor generalization persists long after overfitting has manifested; and (iii) the generalization error eventually becomes arbitrarily small. Moreover, we show, both theoretically and empirically, that grokking can be amplified or eliminated in a principled manner through proper hyperparameter tuning. To the best of our knowledge, these are the first rigorous quantitative bounds on the generalization delay (which we refer to as the "grokking time") in terms of training hyperparameters. Lastly, going beyond the linear setting, we empirically demonstrate that our quantitative bounds also capture the behavior of grokking on non-linear neural networks. Our results suggest that grokking is not an inherent failure mode of deep learning, but rather a consequence of specific training conditions, and thus does not require fundamental changes to the model architecture or learning algorithm to avoid.