论文检索

General Machine Learning · Causality

Adiba Ejaz, Elias Bareinboim

An artificial intelligence must have a model of its environment that is *causal*, supporting reasoning about interventions and counterfactuals, and also *combinatorial*, supporting generalization to unseen combinations of objects. In this work, we formally study when and how such a model can be learned. We develop *relational structural causal models*, extending structural causal models (Pearl 2009) to settings where objects and their relations vary. First, we show how answers to not only causal but also observational queries about unseen combinations of objects can not be identified without further assumptions. To enable such identification---including in the presence of unobserved confounding---we define *relational causal graphs* and derive symbolic identification criteria. Finally, we propose *relational neural causal models*, a provably correct approach that outperforms non-relational baselines on simulated traffic scenes with varying cars, signals, and pedestrians.

Deep Learning · Attention Mechanisms

Yifei Xia, Fangcheng Fu, Hao Yuan, Suhan Ling, Xupeng Miao, Huixia Li, Yuxi Ren, Xin Xia, Xuefeng Xiao, Bin Cui

Diffusion Transformers (DiTs) are increasingly adopted for long-video generation, yet inference is dominated by the quadratic cost of 3D full attention. Sparse attention mitigates this bottleneck by exploiting *token-pair redundancy* and pruning query-key interactions. Nevertheless, its effectiveness on long video generation is often constrained by non-sparse attention heads, making it hard to strike a good balance between inference speed and generation quality. To address this, we identify another pervasive but overlooked redundancy specific to video DiTs: *frame-block similarity*, where frame-blocks in attention weights exhibit highly similar distributions and can be well approximated by lightweight linear calibration. Motivated by this observation, we propose **EchoAttention**, which jointly leverages *token-pair redundancy* (*Sparse* operator) and *frame-block similarity* (*Echo* operator), together with a fine-grained routing policy learned via three-stage distillation. This design enables efficient handling of both sparse and non-sparse heads, overcoming the inherent ceiling of purely sparse attention and yielding a better speed-quality trade-off. Across public video DiTs, EchoAttention consistently improves the speed-quality frontier over SOTA sparse-attention baselines, reducing end-to-end latency up to 2.42$\times$ with minimal quality loss.

Theory · Everything Else

Yash Sarrof, Yupei Du, Katharina Stein, Alexander Koller, Sylvie Thiébaux, Michael Hahn

Transformers have shown inconsistent success in AI planning tasks, and theoretical understanding of when generalization should be expected has been limited. We take important steps towards addressing this gap by analyzing the ability of decoder-only models to verify whether a given plan correctly solves a given planning instance. To analyse the general setting where the number of objects -- and thus the effective input alphabet -- grows at test time, we introduce C*-RASP, an extension of C-RASP designed to establish length generalization guarantees for transformers under the simultaneous growth in sequence length and vocabulary size. Our results identify a large class of classical planning domains for which transformers can provably learn to verify long plans, and structural properties that significantly affects the learnability of length generalizable solutions. Empirical experiments corroborate our theory.

Social Aspects · Accountability, Transparency, and Interpretability

Jinyang Liu, Munir Hiabu

Interpretable machine learning requires models that are accurate and structurally faithful to the data. Existing explainability methods rely heavily on additive representations (e.g., GAMs, SHAP, functional ANOVA), which can suffer from signal cancellation and extrapolation in presence of strong interactions. We propose Tensor Separation Learning (TSL), a regression model that learns a sum of separable (rank-1) tensor products via an orthogonal greedy algorithm. By enforcing separability, TSL avoids the information loss inherent in additive projections caused by marginalizing higher-order interactions. The learned TSL model can be fully reconstructed from first-order partial dependence functions of its fitted factors. We establish approximation-rate guarantees for functions with bounded mixed $ p $-th order partial derivatives and demonstrate that TSL competes with black-box models on regression benchmarks. Crucially, TSL improves interpretability by factorizing interactions, allowing users to explicitly disentangle the magnitude of an effect from its direction directly via the fitted factors.

Fu Feng, Ruixiao Shi, Yucheng Xie, Jianlu Shen, Jing Wang, Xin Geng

Universal morphology control aims to learn a universal policy that generalizes across heterogeneous robot morphologies, with Transformer-based controllers emerging as a dominant choice. However, such architectures incur substantial computational costs, resulting in high deployment overhead, and existing methods exhibit limited cross-task generalization, necessitating training from scratch for each new task. To this end, we propose DivMorph, a modular training paradigm that leverages knowledge diversion to learn \textit{decomposable controllers}. DivMorph factorizes randomly initialized Transformer weights into \textit{basic knowledge units} via SVD and employs dynamic soft gating, conditioned on task and morphology embeddings, to adaptively modulate these units into universal \textit{learngenes} and morphology- and task-specific \textit{tailors} during training, thereby achieving knowledge disentanglement. By selectively activating relevant components, DivMorph adaptively recomposes the controller, enabling efficient policy deployment and effective policy transfer to novel tasks. Extensive experiments demonstrate that DivMorph achieves state-of-the-art performance, improving sample efficiency for cross-task transfer by 3.3$\times$ and reducing model size for single-agent deployment by 16.7$\times$.

General Machine Learning · Supervised Learning

Wei Chen, Rui Ding, Huang Bojun, Yuxuan Liang, Yang Zhang, Qiang Fu, Shi Han, Dongmei Zhang

Supervised Causal Learning has shown promise in causal discovery, yet it often struggles with generalization across diverse interventional settings, particularly when intervention targets are unknown. To address this, we propose TICL (Test-time Interventional Causal Learning), a novel method that synergizes Test-Time Training with Joint Causal Inference (JCI). Specifically, we design a self-augmentation strategy to generate instance-specific training data at test time, effectively avoiding distribution shifts. Furthermore, by integrating JCI, we developed a PC-inspired two-phase supervised learning scheme, which effectively leverages self-augmented data while ensuring theoretical identifiability. Extensive experiments on bnlearn benchmarks demonstrate TICL's superiority in multiple aspects of causal discovery and intervention target detection.

Applications · Chemistry, Physics, and Earth Sciences

Weixin Liao, Mingquan Feng, Zhizhou Zhang, Youjia Wu, Yifan Fu, Junchi Yan

Neural operators have shown promise in accelerating PDE solvers, yet they remain unreliable for the sparse linear systems induced by discretization due to limited generalization across physical parameters and insufficient accuracy, and hybrid neural iterative schemes face stagnation as the residual distribution evolves over iterations. To address these limitations, we propose G-RANS (Generalizable Residual-Aware Neural Solver), a neuralized iterative paradigm that performs residual-aware subspace corrections by mapping the residual to the matrix graph, generating multi-scale correction subspaces via a residual-aware basis generator, and applying projected updates through a differentiable subspace projection. G-RANS is trained fully self-supervised via progressive bootstrap with multi-stage residual distributions. On sparse FEM systems from a representative suite of second-order elliptic PDEs (Poisson, advection--diffusion, reaction--diffusion, and Helmholtz), G-RANS is robust to severe coefficient shifts (up to $\pm70\\%$) and shows strong cross-equation generalization. G-RANS reaches relative residuals on the order of $10^{-5}$, substantially outperforming end-to-end neural operators, and attains the same target residual with $2$--$4\times$ less wall-clock time than classical Krylov solvers.

Deep Learning · Attention Mechanisms

Zhang Wan, Yu Li, Tianze Huang, Haochen Li, Juan Cao, Sheng Tang

Video Diffusion Transformers (VDiTs) represent the state-of-the-art in video generation but are fundamentally constrained by the quadratic computational complexity of self-attention. To accelerate this critical computation, we analyze the pre-softmax matrix ($QK^T$) and reveal two key insights: (1) dense attention patterns inherently reside on a global low-rank manifold characterized by rapid singular value decay; and (2) real motion manifests as hybrid spatio-temporal patterns rather than rigid "spatial vs. temporal" classifications. Guided by these insights, we propose Video-SVD. As a plug-and-play acceleration method that requires no alteration to the original network parameters, it extracts universal bases via offline SVD and employs a dynamic subspace projection strategy at inference, thereby bypassing the expensive full $QK^T$ matrix computation entirely. To ensure high fidelity, we deploy layer-shared dual-stream MLPs to synthesize fine-grained textural details and recover high-frequency RoPE information. Video-SVD achieves significant end-to-end speedup while maintaining high visual quality, reaching 1.92$\times$ on HunyuanVideo, 1.75$\times$ on Wan2.1-1.3B, and 1.79$\times$ on Wan2.1-14B.

Deep Learning · Attention Mechanisms

Xiuying Wei, Caglar Gulcehre

Structured dilated attention has an appealing inference-time efficiency knob: it reduces the FLOPs of the attention and the KV cache size by a factor of the dilation size $\mathtt{D}$, while preserving long-range connectivity. However, we find a persistent failure mode of them -sparsifying a pretrained attention model to a dilated pattern leads to severe accuracy degradation. We introduce **RAT+**, a dense-pretraining architecture that augments attention with *full-sequence recurrence* and *active recurrence learning*. A single RAT+ model is pretrained densely once, then flexibly switched at inference time to dilated attention (optionally with local windows) or hybrid layer/head compositions, requiring only a short 1B-token resolution adaptation rather than retraining separate sparse models. At 1.5B parameters trained on 100B tokens, RAT+ closely matches dense accuracy at $\mathtt{D=16}$ and drops by about 2-3 points at $\mathtt{D=64}$ on commonsense reasoning and LongBench tasks, respectively. Moreover, RAT+ outperforms attention when sparsifying to the top-k block attention. We further scale to 2.6B parameters and 200B tokens and observe the same trend.

General Machine Learning · Sequential, Network, and Time Series Modeling

Shiqi Zhang, Pan Mu, HantingYan, Yuchao Zhu, jinglin zhang, Cong Bai

Multi-modal spatio-temporal forecasting underpins many real-world applications but remains challenging due to the complex and evolving interactions across modalities and time steps. Moreover, the lack of interpretability in existing models limits their reliability in safety-critical scenarios. In this paper, we present CausalX, a unified and causally interpretable plug-and-play model for multi-modal spatio-temporal forecasting. CausalX achieves interpretability by learning a dynamic causal graph across modalities and time, whose edge weights quantify causal attribution strength, and are further refined by a diffusion-based generative process guided by structural priors. To overcome the absence of ground-truth causal structures, CausalX aggregates multi-source constraints from causal analysis techniques and a variational autoencoder, spanning predictive, temporal, interventional, and generative aspects to jointly learn a more comprehensive causal graph. Extensive experiments on real-world forecasting tasks, including pedestrian trajectory prediction and tropical cyclone forecasting, demonstrate that CausalX achieves superior accuracy while producing interpretable causal graphs. CausalX is modular, architecture-agnostic, and generalizable, offering a new perspective on bridging causal inference and spatio-temporal forecasting.

Deep Learning · Attention Mechanisms

Runzhong Li, Renjie Liu, Qing Li, Bo Tang

Causal softmax attention is the algorithmic foundation of modern large language models. Inspired by linear attention, recent work has sought to enhance it by incorporating forget gates. However, these efforts, such as FoX, have been limited to coarse, scalar gates. While fine-grained, element-wise gates are shown to be more effective than scalar ones in linear attention, their direct integration into softmax attention is non-trivial due to algebraic constraints. In this work, we introduce *Fine-grained Forgetting Transformer* (*FiX*), a novel architecture that successfully enables element-wise forget gates in softmax attention. Our core insight is that the softmax denominator becomes mathematically redundant under a subsequent RMSNorm layer, allowing us to reformulate the forgetting mechanism as a direct element-wise multiplication on the value vectors. This formulation makes FiX the first positional encoding applied to value-output (VO) pairs, designed to be complementary to existing query-key (QK) encodings like RoPE. We systematically address implementation challenges including numerical precision, computational efficiency, and inference memory consumption. Extensive experiments show *FiX* achieves lower training loss and superior performance on both short-text common sense benchmarks and long-context tasks, opening a new path for building more powerful transformers.

Deep Learning · Generative Models and Autoencoders

Zhiyang Liu, Wanze Li, Yuwei Wu, Chengran Yuan, JIAWEI SUN, Rui Zheng, Marcelo Ang Jr

Diffusion models have advanced 3D shape generation, yet most methods still denoise in high-cardinality spaces (e.g., voxel/SDF grids, meshes, or point clouds), which is computationally and memory intensive and makes it difficult to scale in terms of both higher resolution and stronger controllability. We rethink the diffusion representation and propose to move diffusion from dense geometry to compact geometric primitives, representing each shape as a small set of **superquadrics**. Instead of operating on thousands to millions of geometric representation values, we leverage 7KB superquadric parameters (pose, size, and shape), drastically reducing diffusion-state dimensionality and per-step compute/memory. Our diffusion-over-superquadrics improves scalability by supporting broader capabilities (e.g., resolution-free point-cloud decoding, part-level editing, and constraint-based design) and achieving competitive surface-fidelity and distributional performance on standard benchmarks after point-cloud decoding, while enabling efficient generation within 0.6s per shape for most conditions.

General Machine Learning · Causality

Jiacan Gao, Xinyan Su, Mingyuan Ma, Yiyan HUANG, Xiao Xu, Xinrui Wan, Tianqi Gu, Enyun Yu, Jiecheng Guo, Zhiheng Zhang

Estimating heterogeneous treatment effects is central to data-driven decision-making, yet industrial applications often face a fundamental tension between limited randomized controlled trial (RCT) budgets and abundant but biased observational data collected under historical targeting policies. Although observational logs offer the advantage of scale, they inherently suffer from severe policy-induced imbalance and overlap violations, rendering standalone estimation unreliable. We propose a budgeted active experimentation framework that iteratively enhances model training for causal effect estimation via active sampling. By leveraging observational priors, we develop an acquisition function targeting uplift estimation uncertainty, overlap deficits, and domain discrepancy to select the most informative units for randomized experiments. We establish finite-sample deviation bounds, asymptotic normality via martingale Central Limit Theorems (CLTs), and minimax lower bounds to prove information-theoretic optimality. Extensive experiments on industrial datasets demonstrate that our approach significantly outperforms standard randomized baselines in cost-constrained settings.

General Machine Learning · Evaluation

Alexandre Bayle, Lucas Janson, Lester Mackey

Cross-validation (CV) is known to provide asymptotically exact tests and confidence intervals for model improvement but only when the model comparison is *relatively stable*. Surprisingly, we prove that even simple, individually stable models can generate relatively unstable comparisons, calling into question the validity of CV inference. Specifically, we show that the Lasso and its close cousin, soft-thresholding, generate relatively unstable comparisons and invalid CV inferences, even in the most favorable of learning settings and when both models are individually stable. These findings highlight the importance of verifying relative stability before deploying CV for model comparison.

Deep Learning · Large Language Models

Yue Feng, Wei Ji, Qijia Lu, Jingrou Zhang, Fei Shen, Jingjing Li, Xiao Li, Yizhen Jia, Qiang Chen, Limin Wang 等

The rapid advancement of AI-driven video generation has transformed content creation, while simultaneously increasing the risk of misinformation through localized manipulations in long-form videos. Existing video forensic methods predominantly operate on short, independent clips, and thus fail to capture realistic scenarios where AI-generated content is sparsely embedded within otherwise authentic footage. To bridge this gap, we formulate the task of Temporal AI-Generated Segment Localization and Explanation, which targets authenticity detection, temporal localization, and interpretable analysis of manipulated segments in untrimmed long videos. We further introduce TASLE, a large-scale benchmark comprising 12,472 untrimmed videos with diverse manipulation patterns and rich annotation signals, including temporal boundaries, authenticity labels, and segment-level rationales. In addition, we propose MSLoc, a coarse-to-fine forensic baseline that combines a boundary-sensitive proposal generation module for efficient long-video scanning with an MLLM-based refinement module for precise boundary localization and interpretable reasoning. Experiments validate the effectiveness of the proposed baseline, highlighting the importance of segment-level explainable forensics for long-form AI-generated video analysis. Dataset and code will be made publicly available.

Fu-Yun Wang, Han Zhang, Michaël Gharbi, Hongsheng Li, Taesung Park

Flow matching models (FMs) have revolutionized text-to-image (T2I) generation, with reinforcement learning (RL) serving as a critical post-training strategy for alignment with reward objectives. In this research, we show that current RL pipelines for FMs suffer from two underappreciated yet important limitations: sample inefficiency due to insufficient generation diversity, and pronounced prompt overfitting, where models memorize specific training formulations and exhibit dramatic performance collapse when evaluated on semantically equivalent but stylistically varied prompts. We present \textbf{PromptRL} (\textbf{P}rompt \textbf{M}atters in \textbf{RL} for Flow-Based Image Generation), a framework that incorporates language models (LMs) as trainable prompt refinement agents directly within the flow-based RL optimization loop. This design yields two complementary benefits: rapid development of sophisticated prompt rewriting capabilities and, critically, a synergistic training regime that reshapes the optimization dynamics. PromptRL achieves state-of-the-art performance across multiple benchmarks, obtaining scores of 0.97 on GenEval, 0.98 on OCR accuracy, and 24.05 on PickScore. Furthermore, we validate the effectiveness of our RL approach on large-scale image editing models, improving the EditReward of FLUX.1-Kontext from 1.19 to 1.43 with only 0.06 million rollouts, surpassing Gemini 2.5 Flash Image (also known as Nano Banana), which scores 1.37, and achieving comparable performance with ReasonNet (1.44), which relied on fine-grained data annotations along with a complex multi-stage training. Our extensive experiments empirically demonstrate that PromptRL consistently achieves higher performance ceilings while requiring over 2$\times$ fewer rollouts compared to naive flow-only RL.

Social Aspects · Safety

Humzah Merchant, Bradford Levy

Large Language Models (LLMs) frequently memorize sensitive training data, creating significant privacy and copyright risks. We present a novel unlearning framework rooted in the principle that learning is easier than forgetting. We first introduce \textbf{Divergence Decoding (DD)}, a mechanism that uses small, efficiently trained auxiliary models to steer the logits of the LLM away from specific data during inference. We then demonstrate this steered distribution can be trivially distilled back into the base model. Our method decisively outperforms \textbf{state-of-the-art (SOTA)} baselines on TOFU and MUSE benchmarks, and we find evidence of generalization in the domain of images. \href{https://anonymous.4open.science/r/targeted_unlearning_icml2026/}{Code is available at this anonymous link. }

Deep Learning · Robustness

Bum Jun Kim, Makoto Kawano, Yusuke Iwasawa, Yutaka Matsuo

While the robustness of vision models is often measured, their dependence on specific architectural design choices is rarely dissected. We investigate why certain vision architectures are inherently more robust to additive Gaussian noise and convert these empirical insights into simple, actionable design rules. Specifically, we performed extensive evaluations on 1,174 pretrained vision models, empirically identifying four consistent design patterns for improved robustness against Gaussian noise: larger stem kernels, smaller input resolutions, average pooling, and supervised vision transformers (ViTs) rather than CLIP ViTs, which yield up to 506 rank improvements and 21.6%p accuracy gains. We then develop a theoretical analysis that explains these findings, converting observed correlations into causal mechanisms. First, we prove that low-pass stem kernels attenuate noise with a gain that decreases quadratically with kernel size and that anti-aliased downsampling reduces noise energy roughly in proportion to the square of the downsampling factor. Second, we demonstrate that average pooling is unbiased and suppresses noise in proportion to the pooling window area, whereas max pooling incurs a positive bias that grows slowly with window size and yields a relatively higher mean-squared error and greater worst-case sensitivity. Third, we reveal and explain the vulnerability of CLIP ViTs via a pixel-space Lipschitz bound: The smaller normalization standard deviations used in CLIP preprocessing amplify worst-case sensitivity by up to 1.91 times relative to the Inception-style preprocessing common in supervised ViTs. Our results collectively disentangle robustness into interpretable modules, provide a theory that explains the observed trends, and build practical, plug-and-play guidelines for designing vision models more robust against Gaussian noise.

Deep Learning · Attention Mechanisms

Zizhuo Fu, Wenxuan Zeng, Runsheng Wang, Meng Li

Large Language Models (LLMs) often assign disproportionate attention to the first token, a phenomenon known as the attention sink. Several recent approaches aim to address this issue, including Sink Attention in GPT-OSS and Gated Attention in Qwen3-Next. However, a comprehensive analysis of the relationship among these attention mechanisms is lacking. In this work, we provide both theoretical and empirical evidence demonstrating that the sink in Vanilla Attention and Sink Attention naturally construct a Mixture-of-Experts (MoE) mechanism within attention layers. This insight explains the head collapse phenomenon observed in prior work, where only a fixed subset of attention heads contributes to generation. To mitigate head collapse, we propose a sink-aware training algorithm with an auxiliary load balancing loss designed for attention layers. Extensive experiments show that our method achieves effective head load balancing and improves model performance across Vanilla Attention, Sink Attention, and Gated Attention. We hope this study offers a new perspective on attention mechanisms and encourages further exploration of the inherent MoE structure within attention layers.

Deep Learning · Attention Mechanisms

Xiaowen Sun, Hui Liu, Gongguan Chen, Ning Mao

Medical diagnosis tasks often rely on heterogeneous information from multiple sources, such as medical images and clinical text. Multimodal fusion is therefore essential for improving classification performance and robustness. However, most existing methods assume a fixed and known modality set, making them less effective when the number or composition of modalities changes. To address this limitation, we propose a modality-agnostic medical multimodal fusion framework that can naturally accommodate an arbitrary number of input modalities. At the coarse-grained modality level, we represent each modality’s estimation of latent semantics as an uncertainty-aware probability distribution, and impose symmetric consistency constraints to achieve global cross-modal semantic alignment. At the fine-grained token level, we further design a token-level consistency constraint based on linear reconstruction. This constraint enables structured mutual verification of local semantics across modalities. Finally, for multimodal fusion, we adopt a multi-view consistency strategy to obtain a unified representation for diagnosis prediction. In particular, each modality is sequentially treated as a conditional view to suppress noise in the remaining modalities and extract shared semantics. Extensive experiments on five public and self-constructed multimodal medical datasets demonstrate the effectiveness and scalability of the proposed approach. Code is available at https://github.com/gjhgjbkg/SMMF.