论文检索

Deep Learning · Other Representation Learning

Jiawei Gu, Ziyue Qiao, Xiao Luo

Each layer of a Transformer refines the hidden state toward a prediction, an iterative process resembling fixed-point iteration. Yet when should this iteration terminate? Existing early exit methods rely on output confidence as a proxy for internal convergence. We take a more direct approach by examining the geometry of the hidden state trajectory. We find that layer-wise updates exhibit a two-phase structure: large, volatile updates in early layers, followed by small, aligned updates as the model propagates an already-formed representation. The transition is remarkably sharp. This yields a simple criterion: exit when step size vanishes and direction stabilizes. We track the normalized update norm and cosine similarity between consecutive updates, exiting when both indicate convergence. The overhead is $O(d)$ per layer, independent of vocabulary size, requiring no learned components or architectural modifications. On LLaMA-2-7B and LLaMA-2-13B across question answering and commonsense reasoning tasks, this geometric criterion reduces FLOPs by 30--35\% while retaining over 98\% of full-depth accuracy.

Deep Learning · Algorithms

Trong Khiem Tran, Duc Chu Anh, Quang Hung Pham, Phi Le Nguyen, Nghia Hoang

Cross-modal knowledge distillation (CMKD) aims to transfer knowledge from a teacher model in one modality to a student model in another modality. Existing CMKD methods have demonstrated strong empirical performance when paired multimodal data with aligned semantics are available, but such paired data are often costly or infeasible to obtain. This paper studies CMKD in the more challenging and practically relevant setting of unpaired data. We establish a distributional relationship between teacher and student models under cross-modal distillation and characterize two fundamental quantities that underlie effective knowledge transfer: feature and label alignments. These quantities capture semantic discrepancy between modalities at the level of representation distributions and prediction distributions, respectively. Guided by this theoretical insight, we propose a principled framework, with theoretical guarantees, that enables effective cross-modal knowledge distillation by aligning distributions rather than individual samples, thereby eliminating the need for data-level pairing. Extensive experiments across a wide range of multimodal benchmarks show that our framework is highly effective in both unpaired and paired data settings, improving significantly over prior work.

Deep Learning · Generative Models and Autoencoders

Yeongmin Kim, Donghyeok Shin, Byeonghu Na, Minsang Park, Richard Lee Kim, IL CHUL MOON

Diffusion models have demonstrated strong generative performance; however, generated samples often fail to fully align with human intent. This paper studies a test-time scaling method that enables sampling from regions with higher human-aligned reward values. Existing gradient guidance methods approximate the expected future reward (EFR) at an intermediate particle $\mathbf{x}_t$ using a Taylor approximation, but this approximation at each time step incurs high computational cost due to sequential neural backpropagation. We show that the EFR at any $\mathbf{x}_t$ can be computed using only marginal samples from a pre-trained diffusion model. The proposed EFR formulation detaches the neural dependency between $\mathbf{x}_t$ and the EFR, enabling closed-form guidance computation without neural backpropagation. To further improve efficiency, we introduce lookahead sampling to collect marginal samples. For final sample generation, we use an accurate solver that guides particles toward high-reward lookahead samples. We refer to this sampling scheme as LiDAR sampling. LiDAR achieves substantial performance improvements using only three samples with a 3-step lookahead solver, exhibiting steep performance gains as lookahead accuracy and sample count increase; notably, it reaches the same GenEval performance as the latest gradient guidance method for SDXL with a 9.5× speedup.

Deep Learning · Large Language Models

Yize Wu, KE GAO, Ling Li, Yanjun Wu

Load Balancing has emerged as a critical problem in expert-parallel distributed inference of Mixture-of-Experts (MoE) models. As routing distributions are typically skewed across experts, devices hosting lighter-loaded experts must idle to wait for the heaviest during expert computing, leading to inefficiency. Existing load-balancing approaches primarily rely on expert replication or migration within each layer, which introduce additional overhead and limit their flexibility and scalability. To address this problem, we propose EasyBalance, a \textbf{cross-layer} load balancing strategy for expert-parallel MoE inference. EasyBalance requires no modifications to the expert-device mapping, enabling instant adaptability and incurring essentially no additional overhead. Our key insights are that (1) experts from other layers can be viewed as naturally redundant, and (2) expert workloads of multiple layers, if from different micro-batches, can be jointly executed. Based on these observations, EasyBalance greedily schedules subsets of cross-layer workloads at each expert-computation stage, while deferring the remaining workloads for future balancing opportunities. Extensive experiments across different models, tasks, and parallelism configurations demonstrate that EasyBalance consistently accelerates expert-parallel inference, reducing GPU idling by uniformly over 40\%.

Deep Learning · Large Language Models

Zexuan Wang, Chenghao Yang, Yingqi Que, Zhoufutu Wen, Zaiyuan Wang, Jiashuo Liu, Zhixin Yao, Zhenzhu Yang, Huaqing Yuan, Yiwen Wang 等

Real-world autonomous planning requires coordinating tightly coupled constraints where a single decision dictates the feasibility of all subsequent actions. However, existing benchmarks predominantly feature loosely coupled constraints solvable through local greedy decisions and rely on idealized data, failing to capture the complexity of extracting parameters from dynamic web environments. We introduce $\textbf{WorldTravel}$, a benchmark comprising 150 real-world travel scenarios across 5 cities that demand navigating an average of 15+ interdependent temporal and logical constraints. To evaluate agents in realistic deployments, we develop $\textbf{WorldTravel-Webscape}$, a multi-modal environment featuring over 2,000 rendered webpages where agents must perceive constraint parameters directly from visual layouts to inform their planning. Our evaluation of 10 frontier models reveals a significant performance collapse: even the state-of-the-art GPT-5.2 achieves only 28.0\% feasibility in text-only settings, which plummets to 3.4\% in multi-modal environments. We identify a critical Perception-Action Gap and a Planning Horizon threshold at approximately 10 constraints where model reasoning consistently fails, suggesting that perception and reasoning remain independent bottlenecks. These findings underscore the need for next-generation agents that unify high-fidelity visual perception with long-horizon reasoning to handle brittle real-world logistics.

Deep Learning · Everything Else

Jiawei Gu, Fengyuan Nie, Hao Tang, Yanpeng Sun

Low-precision arithmetic is pervasive in neural network training and deployment, yet its effect on prediction \textit{confidence}, not just accuracy, remains unexamined. We show that the softmax function amplifies logit-space quantization errors in an input-dependent manner: confidence distortion scales with the product of precision-dependent error bound $\epsilon$ and logit norm, peaking when the model is confident but not saturated. This explains why identical models report different confidence values across precisions, a phenomenon we term \textit{Precision Split}. During training, the same mechanism causes gradient underflow: when logit margins exceed a precision-dependent threshold, gradients vanish and samples silently stop contributing to learning. Since logit norm serves as a computable proxy for precision-induced risk, we propose Precision-Aware Confidence Scaling (PACS), which applies sample-adaptive temperature inversely related to this risk, with sub-one-percent overhead and no full-precision computation required. On ImageNet with mixed-precision ResNet-50, PACS reduces Expected Calibration Error from 5.82\% to 1.92\% while maintaining accuracy, with consistent improvements across architectures, precision formats, and modalities.

General Machine Learning · Causality

Medha Agarwal, Alex Luedtke

We introduce the Sinkhorn treatment effect, an optimal transport measure of divergence between counterfactual distributions. Unlike classical quantities such as the average treatment effect, this measure captures differences across entire distributions. We analyze this divergence as a statistical functional and show it can be written as a smooth transformation of counterfactual mean embeddings with an appropriate kernel. This characterization allows us to establish first-order pathwise differentiability in general, and second-order pathwise differentiability under the null hypothesis of equal counterfactual distributions. Leveraging this smoothness, we construct debiased estimators and use them to obtain asymptotically valid tests for distributional treatment effects. Experiments on simulated and image data demonstrate the practical advantages of our estimator and testing procedure.

General Machine Learning · Representation Learning

Linghao Kong, Inimai Subramanian, Yonadav Shavit, Micah Adler, Dan Alistarh, Nir Shavit

This work demonstrates how increasing the number of neurons in a network without increasing its number of non-zero parameters improves performance. We show that this gain corresponds with a decrease in interference between multiple features that would otherwise share the same neurons. On symbolic tasks, specifically Boolean code problems, splitting each neuron into sparser sub-neurons with knowledge of the clauses systematically reduces polysemanticity metrics and yields higher task accuracy. Notably, even random splits of neuron weights approximate these gains, indicating that reduced collisions, not precise assignment, are a primary driver. Consistent with the superposition hypothesis, the benefits of this framework grow with increasing interference: when polysemantic load is high, accuracy improvements are the largest. Transferring these insights to real models—classifiers over CLIP embeddings, CNNs, and deeper multilayer networks—we find that widening networks while maintaining a constant non-zero parameter count consistently increases accuracy. These results identify an interpretability-grounded mechanism to leverage width against superposition, improving performance without increasing the number of non-zero parameters. Such a direction is well matched to modern accelerators, where memory movement of non-zero parameters, rather than raw compute, is often the dominant bottleneck.

Applications · Time Series

Dongjian Song, Yunhao Meng, Songjun Huang, Jiayi Han

Accurately predicting the future trajectories of traffic participants is critical for safe, efficient, and human-friendly autonomous driving. Existing learning-based trajectory prediction methods are predominantly time-domain and insufficiently exploit latent frequency information, which limits their capability to capture low-frequency long-term dependencies and high-frequency short-term dynamics. To address this, we propose TF-FACE, a Time-Frequency learning framework with Frequency-domain Adaptive and Controllable Enhancement. TF-FACE introduces a fusion encoder with learnable gated frequency-domain attention that adaptively manipulates band-specific features for trajectory prediction. Building on the fused representation, we design a dual-stage decoder and a band-specific time–frequency dual-consistency loss to enable controllable decoupling and coupling across long- and short-term temporal scales, global and local scales, and then generate final multimodal predictions. Experiments on Argoverse 1 demonstrate that TF-FACE achieves state-of-the-art accuracy, while maintaining real-time inference for autonomous driving. Additional experiments are conducted on Argoverse 2, further validating the TF-FACE's performance and generalizability.

Social Aspects · Privacy

Guangnian Wan, Gongfan Fang, Xinyin Ma, Xinchao Wang

Gradient Inversion Attack (GIA) poses a significant threat to federated learning, enabling adversaries to reconstruct private training data from the information shared during training. Prior research has predominantly focused on the vanilla SGD, where the server or an eavesdropper can directly observe true gradients. In practical deployments, however, models may be trained with adaptive optimizers (e.g., Adam, RMSProp, and AdaGrad), for which the observable signal is not raw gradients but momentum-based parameter updates. This setting remains underexplored and undermines traditional gradient-matching strategies, which struggle to recover labels and images from non-gradient updates. To address this gap, this paper explores attacks tailored to modern adaptive optimizers. We present an analytical rule for recovering labels from optimizer updates and propose an update-matching objective that optimizes dummy inputs to reproduce the observed updates. The proposed approach is general and can be directly applied to various optimizers such as Adam, AdaGrad, and RMSProp. Furthermore, we find that, despite being introduced for adaptive optimizers, the proposed objective function also yields stronger attacks in the standard SGD setting. Experiments on datasets such as ImageNet and PACS highlight the effectiveness of our method over existing gradient matching techniques.

Social Aspects · Privacy

Xun Ran, Qingqing Ye, Xin Huang, Jianliang Xu, Haibo Hu

Cross-silo recommendation from implicit feedback is a key task in modern recommender systems, where user-item interaction data are distributed across multiple parties and cannot be centrally collected. Unlike explicit feedback, which provides fully observed real-valued ratings, implicit feedback is one-class and extremely sparse, recording only users' actions or inactions (e.g., clicks, visits, or bookmarks), yet it is far more prevalent in real-world applications. Such behavioral data are often highly sensitive, raising significant privacy concerns when used for collaborative model training. Although differential privacy (DP) has been widely applied to explicit feedback-based models, the resulting utility degradation makes it difficult to apply DP effectively to implicit feedback learning. In this work, we propose DPIMF, a differentially private implicit matrix factorization framework for cross-silo recommendation based on objective perturbation. To improve utility, we redesign the loss function and adopt an importance sampling scheme to reduce the noise scale required for privacy preservation. We further provide formal utility guarantees for the proposed techniques and characterize the conditions under which utility improvements are maximized. Experiments on three benchmark datasets validate our theoretical analysis and demonstrate that DPIMF achieves a better privacy-utility trade-off than state-of-the-art methods.

Social Aspects · Privacy

Puwei Lian, Yujun Cai, Songze Li, Bingkun BAO

Diffusion models have achieved tremendous success in image generation, but they also raise significant concerns regarding privacy and copyright issues. Membership Inference Attacks (MIAs) are designed to ascertain whether specific data were utilized during a model's training phase. As current MIAs for diffusion models typically exploit the model's image prediction ability, we formalize them into a unified general paradigm which computes the membership score for membership identification. Under this paradigm, we empirically find that existing attacks overlook the inherent deficiency in how diffusion models process high-frequency information. Consequently, this deficiency leads to member data with more high-frequency content being misclassified as hold-out data, and hold-out data with less high-frequency content tend to be misclassified as member data. Moreover, we theoretically demonstrate that this deficiency reduces the membership advantage of attacks, thereby interfering with the effective discrimination of member data and hold-out data. Based on this insight, we propose a plug-and-play high-frequency filter module to mitigate the adverse effects of the deficiency, which can be seamlessly integrated into any attacks within the general paradigm without additional time costs. Extensive experiments corroborate that this module significantly improves the performance of baseline attacks across different datasets and models.

Deep Learning · Large Language Models

Ferhat Erata, Orr Paradise, Thanos Typaldos, Timos Antonopoulos, ThanhVu Nguyen, Shafi Goldwasser, Ruzica Piskac

Randomized self-reductions (RSRs) express $f(x)$ using $f$ evaluated at random correlated points, enabling self-correcting programs, instance-hiding protocols, and applications in complexity theory and cryptography. Yet discovering RSRs has required manual expert derivation for over 40 years, limiting their practical use. We present Bitween for automated RSR learning. First, we formalize RSR learning with sample complexity analysis under correlated sampling. Second, we develop Vanilla Bitween, which integrates multiple backends (linear regression, genetic programming, symbolic regression, and mixed-integer programming). The linear regression backend outperforms the others, discovering RSRs for 43 of 80 functions (54\%) in RSR-Bench, our benchmark suite, including the first known reduction for sigmoid. Third, we introduce Agentic Bitween, a neuro-symbolic approach where LLM agents propose novel query functions beyond the fixed set ($x+r$, $x-r$, $x \cdot r$, $x$, $r$) in prior work. Agentic Bitween discovers RSRs for 64 of 80 functions (80\%), outperforming pure neural baselines in both RSR discovery and verification accuracy.

Deep Learning · Large Language Models

Vésteinn Snæbjarnarson, Anej Svete, Josef Valvoda, Reda Boumasmoud, Brian DuSell, Ryan Cotterell

Large language models (LLMs) trained on natural language data are capable of translating between languages, predict chess moves, and write poetry. Performance on a given task depends on directly relevant training data, yet confounders abound: data in related languages has been shown to help low-resource languages, and training on code has been shown to improve reasoning capabilities in natural language generation. Formal languages have become a common tool for understanding the learnability of language model architectures and their limitations---we argue that they should also be treated as multi-task learners when studying the learnability of a given \emph{task}. This means that to understand the learnability of a given property of a formal language, confounders from other tasks need to be considered. We propose a causal graphical model and an efficient sampling mechanism for probabilistic finite-state automata that gives full control over the occurrences of a given task while maintaining other language properties. To enable targeted evaluation, we derive task-specific decomposed KL-divergences. These tools allow us to know the \emph{causal} relationship between how often a task appears and its true learnability. Our experiments confirm that the correlation between task occurrences and learnability does not recover the accurate relationship---for this, the causal analysis and machinery is necessary.

Social Aspects · Privacy

Jianzhou Wang, Yirui Wu, Lixin Yuan, WENXIAO ZHANG, Jun Liu

Multimodal unlearning aims to eliminate specific data from pretrained multimodal models, which offers significant advantages in data privacy and model efficiency. Current methods struggle to achieve the desired properties of effectiveness, reliability and locality, due to the complex interdependency of unimodal and multimodal knowledge. By introducing a causal perspective, we propose multimodal unlearning with decoupled knowledge components. To promote fine-grained understanding of multimodal context, we introduce Multimodal Variational Inference (MVI) to infer modal-specific and -consistent factors with incomplete sample observation. With foundation of decoupled knowledge, we propose contrastive semantic editing to regulate multimodal unlearning towards refined forgetting. Experiments on privacy- and copyright-sensitive scenarios validate effectiveness of our method across multiple scenarios, ensuring the unlearned model maintains high reliability and locality.

Social Aspects · Accountability, Transparency, and Interpretability

Joeun Kim, HoEun Kim, Dongsup Jin, Young-Sik Kim

Recent multi-bit watermarking methods for large language models (LLMs) prioritize capacity over reliability, often conflating decoding with detection. Our analysis reveals that existing ECC-based extractors suffer from catastrophic false positive rates (FPR), and applying rejection thresholds merely collapses detection sensitivity (TPR) to random guessing. To resolve this structural limitation, we propose **BREW** (Block-wise Reliable Embedding for Watermarking), a framework shifting the paradigm to *designated verification*. BREW employs a two-stage mechanism: (i) **blind message estimation** via independent block voting, followed by (ii) **window-shifting verification** that rigorously validates the payload against local edits. Experiments demonstrate that BREW achieves a TPR of 0.965 with an FPR of 0.02 under 10\% synonym substitution, demonstrating that the high-FPR issue is not an inherent trade-off of multi-bit watermarking, but a solvable structural flaw of prior decoding-centric designs. Our framework is model-agnostic and theoretically grounded, providing a scalable solution for reliable forensic deployment.

Deep Learning · Large Language Models

Azmine Toushik Wasi, Shahriyar Zaman Ridoy, Koushik Tonmoy, Kinga Tshering, S M Muhtasimul Hasan, Wahid Faisal, Tasnim Mohiuddin, Md Rizwan Parvez

Geo-temporal understanding, the ability to infer location, time, and contextual properties from visual input alone, is a core aspect of human intelligence and underpins applications such as disaster management, traffic planning, embodied navigation, world modeling, and geography education. Although recent vision–language models (VLMs) have made progress in image geo-localization using salient cues like landmarks or road signs, their ability to reason about temporal signals and physically grounded spatial cues remains underexplored. To address this gap, we introduce ***TimeSpot***, a benchmark for evaluating real-world geo-temporal reasoning in VLMs. ***TimeSpot*** consists of 1,455 ground-level images from 80 countries and requires structured prediction of temporal attributes (season, month, time of day, daylight phase) and geographic attributes (continent, country, climate zone, environment type, latitude-longitude) directly from visual evidence. The benchmark further includes spatial–temporal reasoning tasks that probe physical plausibility and cue integration under real-world uncertainty. Evaluations of state-of-the-art open- and closed-source VLMs show consistently low performance, particularly for temporal inference, and while supervised fine-tuning yields measurable gains, it remains insufficient, highlighting the need for new approaches to achieve robust, physically grounded geo-temporal understanding. By jointly evaluating spatial and temporal inference with diagnostic rigor, ***TimeSpot*** provides a principled framework for assessing physically grounded, real-world geo-temporal reasoning. We will release ***TimeSpot*** upon acceptance.

Social Aspects · Privacy

Quentin Sinh, Jan Ramon

We study the problem of computing a U-statistic with a kernel function $f$ of degree $k \geq 2$, i.e., the average of some function $f$ over all $k$-tuples of instances, in a federated learning setting. U-statistics of degree $2$ include several useful statistics such as Kendall's $\tau$ coefficient, the Area under the Receiver-Operator Curve and the Gini mean difference. Existing methods provide solutions only under the lower-utility local differential privacy model and/or scale poorly in the size of the domain discretization. In this work, we propose a protocol that securely computes U-statistics of degree $k \geq 2$ under central differential privacy by leveraging Multi Party Computation (MPC). Our method substantially improves accuracy when compared to prior solutions. We provide a detailed theoretical analysis of its accuracy, communication and computational properties. We evaluate its performance empirically, obtaining favorable results, e.g., for Kendall's $\tau$ coefficient, our approach reduces the Mean Squared Error by up to four orders of magnitude over existing baselines.

Social Aspects · Privacy

Runhua Xu, Guoan Wan, James Joshi

Federated Learning (FL) has become the de facto standard for privacy-preserving intelligence, largely due to Secure Aggregation protocols that guarantee the mathematical invisibility of individual user contributions. However, we contend that this pursuit of perfect privacy has engineered a systemic vulnerability: the Privacy-Auditability Paradox. By rendering user updates computationally indistinguishable, current protocols create a "Sanitization Gap" where malicious poisoning is undetectable and a "Regulatory Dead Zone" where compliance with the EU AI Act's robustness and explainability mandates is mathematically impossible. In this position paper, we argue that the community must transition from "Blind Aggregation" to Controllable Secure Aggregation (CSA). We propose a cryptographic paradigm shift utilizing Decentralized Multi-Client Functional Encryption and Zero-Knowledge Proofs (ZKPs) to replace binary secrecy with fine-grained policy-based governance. This framework introduces "Verified Blindness", where the server remains blind to raw data by default but possesses a cryptographically regulated "Break-Glass" mechanism to audit specific inputs under consensus-based governance. We conclude that adopting CSA is not merely a technical upgrade but an existential necessity to transform Federated Learning from an unregulated academic concept into robust, compliant, and trustworthy critical infrastructure.

Social Aspects · Accountability, Transparency, and Interpretability

Ziyang Guo, Berk Ustun, Jessica Hullman

Explanations of model behavior are commonly evaluated via proxy properties weakly tied to the purposes explanations serve in practice. We contribute a decision theoretic framework that treats explanations as information signals valued by the expected improvement they enable on a specified decision task. This approach yields three distinct estimands: (i) a theoretical benchmark that upper-bounds achievable performance by any agent with the explanation, (ii) a human-complementary value that quantifies the theoretically attainable value that is not already captured by a baseline human decision policy, and (iii) a behavioral value representing the causal effect of providing the explanation to human decision-makers. We instantiate these definitions in a practical validation workflow, and apply them to assess explanation potential and interpret behavioral effects in human–AI decision support and mechanistic interpretability.