论文检索

Applications · Computer Vision

Quan Zhang, Jun Li, Weilong Zhu, MINGYANG LI, Qinmu Shen, Yuanxi Peng

Precise alignment is a prerequisite for hyperspectral and multispectral image fusion, yet existing methods struggle with complex non-rigid deformations. Existing techniques either suffer from inter-task error accumulation by treating registration and fusion as disjoint processes or neglect the geometric nature of distortions by relying on isotropic Euclidean metrics. We propose Structure-Aware Riemannian Flow Matching (SA-RFM), a geometry-informed framework for joint registration and fusion of hyperspectral and multispectral images. SA-RFM reformulates registration as dynamic optimal transport on a structure-induced Riemannian manifold, where anisotropic costs are derived from MSI structural cues. To circumvent the complexity of explicit OT solvers, we incorporate this geometry into a conditional flow matching framework via a local cost approximation. This formulation is realized through two synergistic mechanisms: a Riemannian Flow Matching objective that enforces structure-aware error measurement, and an optimal transport direction regularization that aligns the velocity field with the induced metric, thereby resolving the fundamental mismatch between anisotropic costs and conventional Euclidean supervision. Extensive experiments on multiple datasets demonstrate the superiority of our method.

Applications · Health / Medicine

Rohit Dilip, Ayush Varshney, David Van Valen

Tokenization is a promising path to multi-modal models capable of jointly understanding protein sequences, structure, and function. Existing protein structure tokenizers create tokens by pooling information from local neighborhoods, an approach that limits their performance on generative and representation tasks. In this work, we present a method for global tokenization of protein structures in which successive tokens contribute increasing levels of detail to a global representation. This change resolves several issues with generative models based on local protein tokenization: it mitigates error accumulation, provides embeddings without sequence-reduction operations, and allows task-specific adaptation of a tokenized sequence's information content. We validate our method on reconstruction, generative, and representation tasks and demonstrate that it matches or outperforms existing models based on local protein structure tokenizers. We show that our adaptive approach enables inference criteria based on the information content of the generated proteins. We validate representations generated from our tokenizer on CATH classification tasks and demonstrate that non-linear probing on our tokenized sequences outperforms equivalent probing on representations from other tokenizers. Finally, we demonstrate how our method supports zero-shot protein shrinking and affinity maturation.

Applications · Chemistry, Physics, and Earth Sciences

Daria Ledneva, Denis Kuznetsov

Genomic foundation models increasingly adopt large language model architectures, yet almost universally rely on fixed tokenization schemes such as $k$-mers or BPE, which impose arbitrary sequence boundaries and may obscure biologically relevant structure. Although recent work has demonstrated the feasibility of adaptive hierarchical tokenization in autoregressive settings, its extension to masked language modeling and systematic downstream evaluation remain underexplored. We present \textbf{LDARNet}, a 120M-parameter hierarchical genomic foundation model that integrates learnable sequence compression into the masked language modeling paradigm by combining BiMamba-2 state-space layers with selective attention and ratio-based regularization to induce stable token boundaries without supervision. We evaluate LDARNet by fine-tuning on 27 tasks from the Genomics Benchmarks and Nucleotide Transformer suites, covering regulatory, epigenetic, and sequence-level prediction problems, and compare it against state-of-the-art models ranging from 8M to 2.5B parameters. LDARNet achieves 11 out of 18 wins among models under 300M parameters and attains state-of-the-art performance on 5 histone modification tasks, outperforming even substantially larger models on several long-range epigenetic benchmarks. These results indicate that adaptive hierarchical tokenization under masked language modeling can capture long-range genomic dependencies relevant to regulatory biology and highlight learnable compression as a promising direction for efficient and scalable genomic foundation models.

Reinforcement Learning · Batch/Offline

Jiayu Chen, Le Xu, Aravind Venugopal, Jeff Schneider

Offline reinforcement learning (RL) offers a powerful paradigm for data-driven control. Compared to model-free approaches, offline model-based RL (MBRL) explicitly learns a world model from a static dataset and uses it as a surrogate simulator, improving data efficiency and enabling potential generalization beyond the dataset support. However, most existing offline MBRL methods follow a two-stage training procedure: first learning a world model by maximizing the likelihood of the observed transitions, then optimizing a policy to maximize its expected return under the learned model. This objective mismatch results in a world model that is not necessarily optimized for effective policy learning. Moreover, we observe that policies learned via offline MBRL often lack robustness during deployment, and small adversarial noise in the environment can lead to significant performance degradation. To address these, we propose a framework that dynamically adapts the world model alongside the policy under a unified learning objective aimed at improving robustness. At the core of our method is a maximin optimization problem, which we solve by innovatively utilizing Stackelberg learning dynamics. We provide theoretical analysis to support our design and introduce computationally efficient implementations. We benchmark our algorithm on twelve noisy D4RL MuJoCo tasks and three stochastic Tokamak Control tasks, demonstrating its state-of-the-art performance.

Deep Learning · Large Language Models

Mikołaj Zasada, Łukasz Struski, Jacek Tabor, Marcin Kurdziel

Sparse Mixture-of-Experts (MoE) architectures enable scaling LLM parameters under a fixed inference budget by activating only a small subset of experts via top-k routing. While this preserves causality and suits autoregressive language models, the discrete top-k operator is not differentiable, forcing a fixed number of active experts per input and resulting in inefficient use of computation. We propose SoftMoE, which replaces discrete routing with a truncated soft top-k LapSum relaxation, allowing gradient-based optimization of expert routing. We further parameterize the mean number of active experts per layer and impose a global budget constraint, enabling the model to learn how to allocate expert capacity across layers. SoftMoE remains fully compatible with autoregressive modeling and achieves performance comparable to or better than sparse MoE on language modeling and downstream tasks, while activating significantly fewer experts. Notably, the learned allocation is highly non-uniform, with later layers activating more experts.

Applications · Health / Medicine

Shaoqing Duan, Haofei Song, Xintian Mao, Qingli Li, Yan Wang

Defocus deblurring in pathological microscopy remains challenging due to the spatially varying and locally discontinuous nature of optical blur induced by a position-dependent integral imaging process. Existing deep learning methods, constrained by shift-invariance assumptions and limited interpretability, are not well suited to such heterogeneous blur patterns. Neural operators provide a principled alternative by modeling defocus formation directly as an integral operator, offering a new perspective on defocus deblurring. However, most existing neural operator architectures for low-level vision rely on globally parameterized kernels that assume smoothness and stationarity, limiting their ability to model heterogeneous and locally discontinuous blur patterns. To address this limitation, we propose the Discontinuous Galerkin Neural Operator (DGNO), which parameterizes the integral kernel using a discontinuous Galerkin formulation with element-local volume operators and interface numerical fluxes. DGNO provides a principled combination of locality, heterogeneity modeling, and global coherence while preserving the underlying physics of optical image formation. Extensive experiments demonstrate that DGNO surpasses state-of-the-art methods, delivering sharper reconstructions, robust handling of spatially varying blur, and scalable high-resolution performance.

General Machine Learning · Evaluation

Maya Varma, Jean-Benoit Delbrouck, Sophie Ostmeier, Akshay Chaudhari, Curtis Langlotz

Multimodal large language models (MLLMs) often introduce errors when generating image captions, resulting in misaligned image-text pairs. Our work focuses on a class of captioning errors that we refer to as systematic misalignments, where a recurring error in MLLM-generated captions is closely associated with the presence of a specific visual feature in the paired image. Given a vision-language dataset with MLLM-generated captions, our aim in this work is to detect such errors, a task we refer to as systematic misalignment detection. As our first key contribution, we present Symbal, which utilizes a structured, dual-stage setup with off-the-shelf foundation models to identify systematic misalignments and summarize results in natural language. As our second key contribution, we introduce SymbalBench, a benchmark designed to evaluate automated methods on our proposed task. SymbalBench consists of 420 vision-language datasets from two domains (natural images and medical images) with annotated systematic misalignments. Symbal exhibits strong performance on this benchmark, correctly identifying systematic misalignments in 63.8% of datasets, a nearly 4x improvement over the closest baseline. We supplement our evaluations on SymbalBench with real-world evaluations, showing that (1) Symbal can accurately surface systematic misalignments in captions generated by four MLLMs and (2) Symbal is a powerful tool for auditing off-the-shelf image-caption datasets. Ultimately, our novel task, method, and benchmark can aid users with auditing MLLM-generated captions and identifying critical errors, without requiring access to the underlying MLLM.

Deep Learning · Generative Models and Autoencoders

Tian Xia, Fabio De Sousa Ribeiro, Rajat Rasal, Avinash Kori, Raghav Mehta, Ben Glocker

Counterfactual generation aims to simulate realistic hypothetical outcomes under causal interventions. Diffusion models have emerged as a powerful tool for this task, combining DDIM inversion with conditional generation and classifier-free guidance (CFG). In this work, we identify a key limitation of CFG for counterfactual generation: it prescribes a global guidance scale for all attributes, leading to significant spurious changes in inferred counterfactuals. To mitigate this, we propose *Factored Classifier-Free Guidance* (FCFG), a flexible and model-agnostic guidance technique that enables attribute-wise control following a causal graph. FCFG complements recent advances in classifier-free guidance and can be seamlessly extended to advanced guidance schemes such as CFG++ and APG. Our experiments demonstrate that FCFG significantly improves the axiomatic soundness of inferred counterfactuals across both natural and medical image datasets, mitigating spurious amplification effects, and enhancing counterfactual reversibility.

General Machine Learning · Everything Else

Boyang Zhang, Daning Cheng, Yunquan Zhang, Fangming Liu

Parameter sharing is a key model compression technique, yet existing methods overlook the geometric properties of the loss landscape, often causing severe accuracy degradation under high compression ratios. Inspired by second-order optimization, we propose Curvature-aware Graph Coloring (CGC), a cross-layer parameter sharing framework that treats each network layer as a graph node, with each node assigned to a shared low-rank basis. CGC leverages Hessian eigenspace information to group layers with similar curvature profiles, aligning the perturbations introduced by parameter sharing with the low-curvature (flat) directions of the loss ellipsoid. This effectively mitigates performance loss while enabling flexible, global cross-layer sharing. Experiments on LLaMA-7B and Swin Transformer show that CGC achieves 28\%–50\% parameter compression with Top-1 accuracy loss no more than 0.01\% on Swin—or even accuracy gains on LLaMA—while delivering over 60\% higher inference throughput, significantly outperforming SVD-based and heuristic-based methods. This work demonstrates that curvature-guided, geometry-aware sharing is essential for efficient, stable, and high-ratio model compression.

Applications · Computer Vision

Rui Chen, Dongdong Li, Yan Fan, Yan Liu, Yangliu Kuai, Pengfei Zhu

Recently, the rapid development of low-altitude aerial applications has driven the need for drone-based unified detectors. In contrast to task-specific detectors that suffer from poor scalability across diverse scenarios, existing unified detectors leverage the Mixture-of-Experts (MoE) architecture to learn task-aware features from diverse datasets. However, the imbalanced multi-task data distribution leads to over-activation of experts for dominant tasks and under-activation for others. To enable balanced feature learning, this paper combines three detection paradigms (RGB, IR, and RGB-IR) into a unified framework termed DroneDINO. DroneDINO extends DINO by introducing heterogeneous routed MoEs that organize experts into three functional groups: shared, task-specific, and dynamic. Unlike conventional dynamic experts where the top-$k$ experts are activated for each input, the shared expert is activated for all inputs, while each task-specific expert is activated exclusively for the matching task. To ensure inputs are routed to appropriate experts and yield task-discriminative features, we propose a task-recognition auxiliary training strategy to penalize features with low task-discriminability. Experiments demonstrate the effectiveness and generalizability of DroneDINO, which consistently outperforms state-of-the-art unified and task-specific detectors across multiple drone-based detection benchmarks.

Applications · Neuroscience, Cognitive Science

Yujin Potter, Seun Eisape, Shiyang Lai, Alexander Huth, James Evans, Been Kim, Jacob Eisenstein, Dawn Song, Alane Suhr

Researchers have shown that neural similarity among humans predicts social closeness and cooperative success, whereas innovation often emerges from interactions among dissimilar individuals. We investigate whether these principles extend to artificial intelligence by examining interactions between large language models. In our experiments, 276 model pairs interact across eight games spanning both cooperation and novelty. We find that pairs with more similar representation spaces achieve significantly higher cooperation but exhibit reduced novelty and creativity. The effects of representational similarity on cooperation and novelty remain robust even after isolating other factors such as performance disparity and model size. We also find that similarity in the early layers consistently exhibits the strongest effect across games, compared to the middle and later layers. This suggests that a central factor underlying the observed trend is the extent to which the two models share lexical and semantic grounding. These findings suggest that representational similarity can be an important consideration in multi-agent system design.

Theory · Optimization

Woojoo Na, Jennifer Dy

Covariance matrices compactly summarize feature distributions in many machine-learning pipelines, including domain adaptation and Gaussian embeddings. Under a Gaussian approximation, the unregularized Wasserstein-2 *optimal transport* (OT) discrepancy reduces to the *Bures--Wasserstein* (BW) distance between *symmetric positive definite* (SPD) covariance matrices. We introduce **ITSPACE** (Iterative Transport for Stable Proximal Alignment of Covariance Embeddings), a lightweight *few-step* method for covariance alignment under tight compute and memory budgets: it maintains a low-rank representation and produces a valid covariance estimate at every iteration through simple closed-form updates. **ITSPACE** is designed for the rank-budgeted, anytime regime relevant to covariance-based domain adaptation and test-time moment matching: under exact computations, each step provably decreases the *exact* BW distance, and under approximate linear-algebra steps we provide a computable certificate bound that quantifies any deviation from monotone descent. Empirically, in the strict few-step regime **ITSPACE** reaches the same BW distance thresholds faster than BW-targeting gradient descent under a common rank budget, and is more stable than Euclidean, alternative-geometry, and entropically regularized baselines.

Applications · Language, Speech and Dialog

Yifan Yang, Hui Wang, Bing Han, Shujie Liu, Jinyu Li, Yong Qin, Xie Chen

Recent advances in text-to-speech (TTS) technology have enabled systems to generate speech that is often indistinguishable from human speech, bringing benefits to accessibility, content creation, and human-computer interaction. However, current evaluation practices are increasingly inadequate for capturing the full range of capabilities, limitations, and societal impacts of modern TTS systems. This position paper introduces the concept of Responsible Evaluation and argues that it is essential and urgent for the next phase of TTS development, structured through three progressive levels: (1) ensuring the faithful and accurate reflection of a model's true capabilities and limitations, with more robust, discriminative, and comprehensive objective and subjective scoring methodologies; (2) enabling comparability, standardization, and transferability through standardized benchmarks, transparent reporting, and transferable evaluation metrics; and (3) assessing and mitigating ethical risks associated with forgery, misuse, privacy violations, and security vulnerabilities. Through this concept, we critically examine current evaluation practices, identify systemic shortcomings, and propose actionable recommendations. We hope this concept will not only foster more reliable and trustworthy TTS technology but also guide its development toward ethically sound and societally beneficial applications.

General Machine Learning · Representation Learning

Haitao YU, Min Nan, Zheng Fang, Hongyu Zhan, Yusen Tan, Yuhan Wang, Jun Xia

Inferring molecular structures from multimodal spectroscopic measurements requires integrating complementary yet highly heterogeneous signals. However, the common paradigm of directly concatenating multispectral sequences can exhibit anomalous performance degradation, primarily due to pronounced heterogeneity and the resulting multimodal imbalance across modalities. As a remedy, we propose MM-Spectrum, a sparse Mixture-of-Experts framework tailored for multimodal multispectral spectra-to-structure elucidation. To better match the information characteristics under multispectral imbalance, MM-Spectrum introduces an explicit modality-aware routing mechanism that exposes spectral identity to the router in addition to token content representations. Moreover, it incorporates shared and interaction experts, together with heterogeneous expert capacities, to extract multispectral modality-unique and cross-modal synergistic information while suppressing noise-induced interference. Across full-modality, bimodal, and missing-modality settings on molecular structural elucidation, MM-Spectrum achieves consistent and substantial improvements, supported by ablation studies and interpretability analyses. Code is available at https://anonymous.4open.science/r/MM-Spectrum-1727.

Deep Learning · Attention Mechanisms

Hyunchan Moon, Cheonjun Park, Steven Waslander

Vision Transformers (ViTs) have achieved remarkable success across various vision tasks, yet their deployment is often hindered by prohibitive computational costs. While structured weight pruning and token compression have emerged as promising solutions, they suffer from prolonged retraining times and global propagation that creates optimization challenges, respectively. We propose ToaSt, a decoupled framework applying specialized strategies to distinct ViT components. We apply coupled head-wise structured pruning to Multi-Head Self-Attention modules, leveraging attention operation characteristics to enhance robustness. For Feed-Forward Networks (over 60\% of FLOPs), we introduce Token Channel Selection (TCS) that enhances compression ratios while avoiding global propagation issues. Our analysis reveals TCS effectively filters redundant noise during selection. Extensive evaluations across nine diverse models, including DeiT, ViT-MAE, and Swin Transformer, demonstrate that ToaSt achieves superior trade-offs between accuracy and efficiency, consistently outperforming existing baselines. On ViT-MAE-Huge, ToaSt achieves 88.52\% accuracy (+1.64 \%) with 39.4\% FLOPs reduction. ToaSt transfers effectively to downstream tasks, achieving 52.2 versus 51.9 mAP on COCO object detection. Code and models will be released upon acceptance.

Applications · Robotics

Xianzhe Fan, Shengliang Deng, Xiaoyang Wu, Yuxiang Lu, Zhuoling Li, Mi Yan, Yujia Zhang, Zhizheng Zhang, He Wang, Hengshuang Zhao

Existing Vision-Language-Action (VLA) models typically take 2D images as visual input, which limits their spatial understanding in complex scenes. How can we incorporate 3D information to enhance VLA capabilities? We conduct a pilot study across different observation spaces and visual representations. The results show that explicitly lifting visual input into point clouds yields representations that better complement their corresponding 2D representations. To address the challenges of (1) scarce 3D data and (2) the domain gap induced by cross-environment differences and depth-scale biases, we propose Any3D-VLA. It unifies the simulator, sensor, and model-estimated point clouds within a training pipeline, constructs diverse inputs, and learns domain-agnostic 3D representations that are fused with the corresponding 2D representations. Simulation and real-world experiments demonstrate Any3D-VLA's advantages in improving performance and mitigating the domain gap. Our code is available at https://anonymous.4open.science/r/Any3D-VLA-anonymous.

Theory · Game Theory

Renato Leme, Clifford Stein, Yifeng Teng, Pratik Worah

We design efficient approximation algorithms for maximizing the expectation of the supremum of families of Gaussian random variables. In particular, let $OPT:=\max_{\sigma_1,\cdots,\sigma_n}\mathbb{E}\sum_{j=1}^{m}\max_{i\in S_j} X_i$, where $X_i$ are Gaussian, $S_j\subset[n]$ and $\sum_i\sigma_i^2=1$, then our theoretical results include: - We characterize the optimal variance allocation -- it concentrates on a small subset of variables as $|S_j|$ increases, - A polynomial time approximation scheme (PTAS) for computing OPT when $m=1$, and - An $O(\log n)$ approximation algorithm for computing OPT for general $m>1$.

Deep Learning · Generative Models and Autoencoders

Haobo Li, Yanhong Zeng, Yunhong Lu, Jiapeng Zhu, Hao Ouyang, Qiuyu Wang, Ka Leong Cheng, Yujun Shen, Zhipeng Zhang

We present \textbf{AAD-1}, an \textbf{A}symmetric \textbf{A}dversarial \textbf{D}istillation framework for \textbf{O}ne-step autoregressive image-to-video generation. State-of-the-art methods adopt adversarial distillation but suffer from motion collapse and training instability, resulting in static videos. AAD-1 addresses these challenges through two key designs in architecture and training strategy. Our key architectural insight is to break the symmetry between generator and discriminator. While the generator remains causal to preserve autoregressive sampling capability, the discriminator attends bidirectionally over the full spatiotemporal context and produces a single holistic realism score for the entire video sequence. This asymmetric design enables the discriminator to effectively detect global temporal failures and long-range drift that cause motion collapse in autoregressive generation. To stabilize training, we introduce a phased strategy that first uses distribution matching to bootstrap a stable one-step generator, providing a warm-up phase that brings the student distribution closer to the teacher before adversarial distillation begins. Extensive experiments on VBench demonstrate that AAD-1 achieves state-of-the-art performance in one-step autoregressive video generation.

Deep Learning · Large Language Models

Ibraheem Muhammad Moosa, Suhas Lohit, Ye Wang, Moitreya Chatterjee, Wenpeng Yin

Token-level adaptive computation seeks to reduce inference cost by allocating more computation to harder tokens and less to easier ones. However, prior work is primarily evaluated on natural-language benchmarks using task-level metrics, where token-level difficulty is unobservable and confounded with architectural factors, making it unclear whether compute allocation truly aligns with underlying complexity. We address this gap through three contributions. First, we introduce a complexity-controlled evaluation paradigm using algorithmic and synthetic language tasks with parameterized difficulty, enabling direct testing of token-level compute allocation. Second, we propose ANIRA, a unified recurrent Transformer framework that supports per-token variable-depth computation while isolating compute allocation decisions from other model factors. Third, we use this framework to conduct a systematic analysis of token-level adaptive computation across alignment with complexity, generalization, and decision timing. Our results show that compute allocation aligned with task complexity can emerge without explicit difficulty supervision, but such alignment does not imply algorithmic generalization: models fail to extrapolate to unseen input sizes despite allocating additional computation. We further find that early compute decisions rely on static structural cues, whereas online halting more closely tracks algorithmic execution state.

Optimization · Everything Else

Gal Lifshitz, Shahar Zuler, Ori Fouks, Dan Raviv

End-to-end deep learning has achieved impressive results but often relies on large labeled datasets, exhibits limited generalization to unseen scenarios, and incurs substantial computational cost. Classical optimization methods, in contrast, are data-efficient and lightweight but frequently suffer from slow or unstable convergence. Learned optimizers aim to bridge this gap, yet existing approaches predominantly focus on first-order methods, leaving learned second-order optimization largely unexplored. We introduce L-SR1, a novel learned second-order optimizer that augments the classical Symmetric Rank-One (SR1) method with a lightweight, trainable preconditioning unit. This unit generates data-driven rank-one updates that are explicitly aligned with the secant condition via a learned projection mechanism, ensuring stable and consistent curvature estimation. Through controlled analytic benchmarks, we systematically analyze the stability, generalization across problem dimensions, and the quality of the resulting search directions. We further evaluate L-SR1 on the real-world task of Monocular Human Mesh Recovery (HMR), where it outperforms existing optimization-based and learned-optimization baselines. With a compact model and no reliance on task-specific fine-tuning or annotated data, L-SR1 demonstrates strong generalization and provides an effective drop-in optimizer for a wide range of iterative optimization problems.