论文检索

Deep Learning · Large Language Models

Weihao Zeng, Yuzhen Huang, Junxian He

Frontier large language models (LLMs) are increasingly capable of carrying out long-running, real-world tasks. However, as the amount of context grows, their reliability often deteriorate, a phenomenon known as "context rot". Existing long-context benchmarks primarily focus on single-step settings that evaluate a model’s ability to retrieve information from a long snippet. In realistic scenarios, however, LLMs often need to act as agents that explore environments, follow instructions and plans, extract useful information, and predict correct actions under a dynamically growing context. To assess language agents in such settings, we introduce LOCA-bench (a benchmark for **LO**ng-**C**ontext **A**gents). Given a task prompt, LOCA-bench leverages automated and scalable control of environment states to regulate the agent’s context length. This design enables LOCA-bench to extend the context length potentially to infinity in a controlled way while keeping the underlying task semantics fixed. LOCA-bench evaluates language agents as a combination of models and scaffolds, including various context management strategies. While agent performance generally degrades as the environment states grow more complex, advanced context management techniques can substantially improve the overall success rate. We will open-source LOCA-bench to provide a platform for evaluating models and scaffolds in long-context, agentic scenarios.

Deep Learning · Large Language Models

Siqi Lu, Wei Suo, Yongbin Zheng, Jianhang Yao, Wanying XU, Peng Wang

While Large Vision-Language Models (LVLMs) achieves remarkable success, hallucinations remain a significant barrier to their reliable deployment. Recent studies primarily attribute these defects to cross-modal attention imbalances, with most solutions focusing on re-weighting visual tokens or suppressing language priors. Such approaches often overlook the spectral characteristics of the visual information flow and frequently rely on Contrastive Decoding (CD), which doubles the inference time. Instead of following conventional approaches, we identify two distinct hallucination patterns—Perceptual-Semantic Dissociation and Localized Fixation—and accordingly develop FLASH (Frequency-Localized Attention SHaping), a training-free and CD-free framework. FLASH utilizes a Spectral Vortex Score to detect visual heads within multi-head attention layers, applying adaptive spectral modulation to rectify the visual information flow during the decoding phase. Empirical results demonstrate that FLASH offers a superior balance between performance and efficiency compared to SOTA methods.

Applications · Robotics

Zhicheng Fan, Zitong Wu, Zhaoxing Fan, Xiao Zhang, Biao Hou, Bo Ren

Conventional dynamic SLAM approaches typically treat dynamic objects as outliers based on pre-defined categories, creating perceptual blind spots that limit the comprehensive environmental perception required for embodied agents. Although integrating Gaussian Splatting into SLAM enables holistic scene representation, it introduces an optimization paradox: without categorical priors, flexible dynamic primitives rapidly overfit static residuals. This phenomenon undermines the self-supervised error signals necessary for distinguishing motion. In response, we present De4D-SLAM, a novel framework designed for decoupled 4D reconstruction from monocular video. Our approach features a Gradient-Isolated Decoupling strategy, which leverages static reconstruction residuals to supervise a Spatially-Aware Kolmogorov-Arnold Network (SA-KAN), ensuring robust, category-agnostic motion segmentation. Additionally, we propose a Flow-Induced Initialization prior to stabilize the non-convex optimization of 4D Gaussian primitives using dense optical flow. Extensive evaluations on the TUM and Bonn benchmarks demonstrate that De4D-SLAM achieves state-of-the-art performance in both tracking and dynamic reconstruction, successfully reconciling the tension between robust localization and high-fidelity 4D mapping.

Yuhan Zhu, Xiangyu Zeng, Chenting Wang, Xinhao Li, Chunxu Liu, Yicheng Xu, Ziang Yan, Yi Wang, Limin Wang

Multimodal large language models (MLLMs) are emerging as versatile foundations for mixed-modality retrieval. Yet, they often require heavy post-hoc training to convert them into contrastive encoders for retrieval. This work asks: \textit{Can off-the-shelf MLLMs serve as powerful retrievers without additional training?} We present \textbf{FreeRet}, a plug‑and‑play framework that turns any MLLM into a two‑stage retriever. FreeRet first derives semantically grounded embeddings directly from the model for fast candidate search, and then exploits its reasoning ability for precise reranking. The framework contributes three advances: bypassing lexical alignment layers to obtain semantically faithful embeddings, conditioning representation generation with explicit priors, and mitigating framing effect in reranking via neutral choice framing. On the MMEB and MMEB-V2 benchmarks spanning 46 datasets, FreeRet substantially outperforms models trained on millions of pairs. Beyond benchmarks, FreeRet is model-agnostic and scales seamlessly across MLLM families and sizes, preserves their generative abilities, supports arbitrary modality combinations, and unifies retrieval, reranking, and generation into end-to-end RAG within a single model. Our findings demonstrate that pretrained MLLMs, when carefully harnessed, can serve as strong retrieval engines without training, closing a critical gap in their role as generalists.

Xinyu Pi, Qisen Yang, Chuong Nguyen, Hua Shen

Large language models (LLMs) are increasingly used in qualitative data analysis, yet the field lacks a shared way to state what kinds of process LLM-based pipelines intend to produce. This position paper proposes an explicit specification perspective: separating meaning-making from modeling, and making both visible as part of the analytic. We introduce a 4×4 landscape that crosses levels of meaning-making with levels of modeling, and use it to situate and compare qualitative outputs across both human-led studies and LLM-assisted workflows. A structured analysis of prior work suggests that many current LLM pipelines emphasize surface organization and static representations, with fewer systems making explicit commitments to richer causal or dynamical models. We demonstrate that the landscape can be applied consistently through strong agreement in independent labeling, including an LLM-based annotation pass. We conclude with a research agenda for LLM-assisted qualitative analysis focused on explicit level selection, evidence-linked outputs, and governance mechanisms aligned with the strength of semantic and representational claims.

Shaochen (Henry) Zhong

With soaring submission counts, stricter reciprocal review policies, widespread adoption of platforms like OpenReview, and without the offsetting pressure of publication fees, the machine learning (ML) community has one of the largest scholarly presences among all scientific fields. And yet, **almost *everyone* has *many* unpleasant things to share about their review experience.** Worse, there is little public space to seriously discuss — let alone debate — what makes a review system effective or how it might be improved. In this position paper, we expand our discussion on two core problems: *How can we reasonably limit the number of submissions?* and *How can we incentivize good and discourage bad review practices?* We first assess the strengths and shortcomings of existing attempts to address such problems. Specifically, we present four takes on some popular conference mechanisms and propose two alternative designs for improvement. Our general position is that meaningful improvement in ML peer review won't come from polite best-practice suggestions tucked into Calls for Papers or Reviewer Guidelines — it requires **enforceable yet fine-grained procedural safeguards** paired with **a currency-like credit system (what we call *OpenReview Points*)**. ML practitioners can “earn” such points by contributing good review practices, and “spend” across one or multiple major conferences to redeem different kinds of “perks” — such as complimentary registration or the right to request additional review resources.

Deep Learning · Foundation Models

Xinnan Dai, Kai Yang, cheng Luo, Shenglai Zeng, Kai Guo, Jiliang Tang

Reasoning hallucinations in large language models (LLMs) often appear as fluent yet unsupported conclusions that violate either the given context or underlying factual knowledge. Although such failures are widely observed, the mechanisms by which decoder-only Transformers produce them remain poorly understood. We model next-token prediction as a graph search process over an underlying graph, where entities correspond to nodes and learned transitions form edges. From this perspective, contextual reasoning is a constrained search over a sampled subgraph (intrinsic reasoning), while context-free queries rely on memorized structures in the underlying graph (extrinsic reasoning). We show that reasoning hallucinations arise from two fundamental mechanisms: path reuse, where memorized knowledge overrides contextual constraints during early training, and path compression, where frequently traversed multi-step paths collapse into shortcut edges in later training. Together, these mechanisms provide a unified explanation for reasoning hallucinations in LLMs and connected to well-known behaviors observed in downstream applications.

Deep Learning · Large Language Models

Jie Hao, Rui Yu, Wei Zhang, Huixia Judy Wang, Jie Xu, Mingrui Liu

Effective data selection is essential for pretraining large language models (LLMs), enhancing efficiency and improving generalization to downstream tasks. However, existing approaches often require leveraging external pretrained models, making it difficult to disentangle the effects of data selection from those of the external pretrained models. In addition, they often overlook the long-term impact of selected data if the model is trained for a long period of time, primarily due to the prohibitive cost of full-scale LLM pretraining. In this paper, we introduce BLISS (**B**ileve**L** **I**nfluence **S**coring method for data **S**election): a lightweight data selection method that operates entirely \emph{from scratch}, without relying on any external pretrained oracle models, while explicitly accounting for the long-term impact of selected data. BLISS leverages a small proxy model as a surrogate for the LLM and employs a score model to estimate the long-term influence of training samples if the proxy model is trained to convergence. We formulate data selection as a bilevel optimization problem, where the upper-level objective optimizes the score model to assign importance weights to training samples, ensuring that minimizing the lower-level objective (i.e., training the proxy model over the weighted training loss until convergence) leads to best validation performance. Once optimized, the trained score model predicts influence scores for the dataset, enabling efficient selection of high-quality samples for LLM pretraining. We validate BLISS by pretraining 410M/1B/2.8B Pythia and LLaMA-0.5B models on selected subsets of the C4 dataset. Notably, under the 1B model setting, BLISS achieves $1.7\times$ speedup in reaching the same performance as the state-of-the-art method, demonstrating superior performance across multiple downstream tasks.

Haoyu Wang, Haiyan Zhao, Xingyu Yu, Zhangyang Yao, Xu Han, Zhiyuan Liu, Maosong Sun

Post-training quantization at the 2-bit level enables low-cost deployment and inference acceleration for large language models (LLMs). Scalar quantization (SQ) and vector quantization (VQ) are two primary quantization methods, however, the former suffers from significant performance degradation, and the latter incurs computational and storage overhead. We propose UniSVQ, a unified 2-bit quantization framework that bridges scalar and vector quantization by parameterizing codewords as an affine transform of integer lattices. This structure preserves compatibility with optimized integer kernels while retaining much of VQ's flexibility. We further introduce a data-driven block-wise fine-tuning strategy to directly minimize quantization reconstruction error. Extensive experiments across multiple LLM families and zero-shot benchmarks demonstrate that UniSVQ consistently outperforms state-of-the-art SQ methods and achieves performance comparable to advanced VQ methods, while providing higher inference throughput.

Darshan Deshpande, Anand Kannappan, Rebecca Qian

Recent advances in reinforcement learning for code generation have made robust environments essential to prevent reward hacking. As LLMs increasingly serve as evaluators in code-based RL, their ability to detect reward hacking remains understudied. In this paper, we propose a novel taxonomy of reward exploits spanning across 54 categories and introduce TRACE (Testing Reward Anomalies in Code Environments), a synthetically curated and human-verified benchmark containing 517 testing trajectories. Unlike prior work that evaluates reward hack detection in isolated classification scenarios, we contrast these evaluations with a more realistic, contrastive anomaly detection setup on TRACE. Our experiments reveal that models capture reward hacks more effectively in contrastive settings than in isolated classification settings, with GPT-5.2 with highest reasoning mode achieving the best detection rate at 63%, up from 45% in isolated settings on TRACE. Building on this insight, we demonstrate that state-of-the-art models struggle significantly more with semantically contextualized reward hacks compared to syntactically contextualized ones. We further conduct qualitative analyses of model behaviors, as well as ablation studies showing that the ratio of benign to hacked trajectories and analysis cluster sizes substantially impact detection performance. We release the benchmark and evaluation harness to enable the community to expand TRACE and evaluate their models.

Reinforcement Learning · Everything Else

Yang Shengtian, Ziteng Cui, Shuo He, Yewen Li, Qingpeng Cai, Peng Jiang, Lei Feng

Reinforcement learning (RL) has equipped LLM agents with a strong ability to solve complex tasks. However, existing RL methods normally use a single policy network, causing simplicity bias where simple tasks occupy most parameters and dominate gradient updates, leaving insufficient capacity for complex tasks. A plausible remedy could be employing the Mixture-of-Experts (MoE) architecture in the policy network, as MoE allows different parameters (experts) to specialize in different tasks, preventing simple tasks from dominating all parameters. However, a key limitation of traditional MoE is its token-level routing, where the router assigns each token to specialized experts, which fragments phase-consistent patterns into scattered expert assignments and thus undermines expert specialization. In this paper, we propose Phase-Aware Mixture of Experts (PA-MoE). It first features a lightweight phase router that learns latent phase boundaries directly from the RL objective without pre-defining phase categories. Then, the phase router allocates temporally consistent assignments to the same expert, allowing experts to preserve phase-specific expertise. Experimental results demonstrate the effectiveness of our proposed PA-MoE. Code is available at https://anonymous.4open.science/r/PA-MoE-576C/.

Applications · Chemistry, Physics, and Earth Sciences

Shreshth Malik, Tiarnan Doherty, Panagiotis Tigas, Muhammed Razzak, Stephen Roberts, Aron Walsh, Yarin Gal

Existing benchmarks for computational materials discovery primarily evaluate static predictive tasks or isolated computational sub-tasks. While valuable, these evaluations neglect the inherently iterative and adaptive nature of scientific discovery. We introduce MAterials Discovery Environments (MADE), a novel framework for benchmarking end-to-end autonomous materials discovery pipelines. MADE simulates closed-loop discovery campaigns in which an agent or algorithm proposes, evaluates, and refines candidate materials under a constrained oracle budget, capturing the sequential and resource-limited nature of real discovery workflows. We formalize discovery as a search for thermodynamically stable compounds relative to a given convex hull, and evaluate efficacy and efficiency via comparison to baseline algorithms. The framework is flexible; users can compose discovery agents from interchangeable components such as generative models, filters, and planners, enabling the study of arbitrary workflows ranging from fixed pipelines to fully agentic systems with tool use and adaptive decision making. We demonstrate this by conducting systematic experiments across a family of systems, enabling ablation of components in discovery pipelines, and comparison of how methods scale with system complexity.

Deep Learning · Generative Models and Autoencoders

Sangeek Hyun, MinKyu Lee, Jae-Pil Heo

Scalability has driven recent advances in generative modeling, yet it remains underexplored for adversarial learning. We study the scaling behavior of Generative Adversarial Networks through two design choices: training in a compact Variational Autoencoder latent space and using purely transformer-based generators and discriminators. While this setup is efficient and scales well with compute, naively scaling exposes failure modes; underutilization of early layers in the generator and increasing optimization instability. We address these issues with lightweight intermediate supervision and width-aware learning-rate adjustment. Our Generative Adversarial Transformers (GAT) train reliably from small (S) to extra-large (XL) model sizes, and GAT-XL model achieves state-of-the-art single-step class-conditional generation on ImageNet at 256×256 resolution (FID of 2.18) in 60 epochs, requiring 4x fewer epochs than strong baselines.

Social Aspects · Security

Qing Wen, Haohao Li, Zhongjie Ba, Peng Cheng, Miao He, Li Lu, Kui Ren

Advances in AIGC technologies have enabled the synthesis of highly realistic audio deepfakes capable of deceiving human auditory perception. Although numerous audio deepfake detection (ADD) methods have been developed, most rely on local temporal/spectral features or pairwise relations, overlooking high-order interactions (HOIs). HOIs capture discriminative patterns that emerge from multiple feature components beyond their individual contributions. We propose HyperPotter, a hypergraph-based framework that explicitly models these synergistic HOIs through clustering-based hyperedges with class-aware prototype initialization. Extensive experiments demonstrate that HyperPotter surpasses its baseline by an average relative gain of 22.15\% across 11 datasets and outperforms state-of-the-art methods by 13.96\% on 4 challenging cross-domain datasets, demonstrating superior generalization to diverse attacks and speakers.

Kieran Didi, Sarah Alamdari, Alex Lu, Bruce Wittmann, Kadina Johnston, Ava Amini, Ali Madani, Maya Czeneszew, Christian Dallago, Kevin Yang

Machine learning methods that predict protein fitness from sequence remain sensitive to changes in data distributions, limiting generalization across common conditions encountered in protein engineering. Practically, protein engineers are thus left wondering about the effective utility of ML tools. The FLIP benchmark established protocols for testing generalization under some domain shifts, but it was limited to measurements of stability, binding, and viral capsid viability. We introduce FLIP2, a protein fitness benchmark spanning seven new datasets, including enzymes, protein-protein interactions, and light-sensitive proteins, as well as splits that measure generalization relevant to real-world protein engineering campaigns. Evaluating a suite of benchmark models across these datasets and suites reveals that simpler models often matched or outperformed fine-tuned protein language models on \ourset, challenging the utility of existing transfer learning techniques. Provenance for all datasets has been recorded and we redistribute all data CC-BY 4.0 to facilitate continued progress.

Deep Learning · Large Language Models

Senyu Han, Yilu Cao, Kai Yu, Lu Chen

Large language model (LLM) exists a subset of attention heads that are highly responsible for long-context processing. Existing work has identified different long-context heads in models, but their detection methods mainly rely on model inference on actual long texts and do not analyze the inherent properties of the head parameters. In this paper, we use kernel methods to analyze static *frequency kernels* formed by different rotation frequency components of attention heads, and we design a Long-context Potential Score (LPS) to measure the potential of attention heads in processing long contexts. Kernels of heads with high LPS exhibit concentrated low-frequency energy and low effective rank, which allow them to effectively capture highly specialized information from distant contexts. Experiments and analysis on long-context tasks and model behaviors show that the LPS metrics can well reflect the actual capability of heads on long contexts. Furthermore, by simply amplifying low-frequency kernels of heads with high retrieval potential, we can further improve model's performance on long-context tasks. Our metrics and head enhancement methods are fully static and offline, and they can be quickly conducted under low-resource constraints.

Deep Learning · Large Language Models

Li, Yaming Guo, Shenghao Gao, Xinlong Chen, Zuhao Xu, Ying Sun, Chao Wang, Hui Xiong

While Large language models (LLMs) have strong abilities, they generally rely on fine-tuning to supplement downstream task-specific knowledge. Due to the prohibitive memory overhead of full fine-tuning (FT), existing parameter-efficient fine-tuning techniques, e.g., LoRA and Adapters, update parameters only in low-rank or restricted subspaces. However, they fail to approximate FT---the performative fine-tuner---and risk performance degradation in tough tasks. Therefore, we naturally raise a *Low-cost Full Fine-tuning* question: Can we approach standard full fine-tuning in theory, yet with much lower costs in practice? Our key insight is that performing selective updates at each step can, theoretically, recover FT asymptotically, while being cost-effective and ignoring no parameter direction. This motivates a new general fine-tuning paradigm (called *Think-Touch*): we first predict potentials of parameter groups (*think*) and then update only the selected (*touch*) in one step. Theoretically, we show that under a very weak sufficient condition---divergence of the cumulative coverage of the expected gradient norm---any selection strategy can converge in the full-parameter space to a stationary point at which the FT admits no further first-order improvement. Besides, we further derive the general convergence rate for our paradigm and identify a post-hoc greedy strategy that is rate-optimal. Unfortunately, this strategy cannot be directly applied in practice due to its reliance on full and accurate gradient information. Thus, we propose a bandit-based method to online approximate this ideal strategy in the long run with a rigorous regret guarantee. Extensive experimental results on various tasks demonstrate the potential of our paradigm, including much lower space overheads against FT and better performance than LoRAs.

Yuxin Zhang, Ju Fan, Meihao Fan, Shaolei Zhang, Xiaoyong Du

Advanced agents are increasingly demonstrating the potential to operate as autonomous engineers, creating a growing demand for evaluation benchmarks that capture the complexity of real-world development. Such environments typically involve both complex code and large-scale data (i.e., file system). However, existing benchmarks usually evaluate code-centric or data-centric capabilities in isolation, leaving a clear gap with real development scenarios. In this paper, we bridge this gap by introducing CoDA-Bench, the first benchmark to jointly evaluate code and data intelligence in a data-intensive environment. We construct a data-intensive Linux sandbox based on the Kaggle ecosystem (containing hundreds of datasets), where agents must actively explore complex file hierarchies to identify relevant resources and generate code for data-driven analytical tasks. CoDA-Bench comprises 1,202 tasks spanning 53 communities, with each task environment containing an average of 700 files, simulating realistic data scale and noise. Evaluations of advanced agents reveal that even top-performing systems struggle to effectively integrate data discovery with code execution, achieving a success rate of only 56.1%. These results highlight a substantial gap in current agentic capabilities for data-intensive tasks and point to promising directions for future research.

Kieran Didi, Sarah Alamdari, Alex Lu, Bruce Wittmann, Kadina Johnston, Ava Amini, Ali Madani, Maya Czeneszew, Christian Dallago, Kevin Yang

Machine learning methods that predict protein fitness from sequence remain sensitive to changes in data distributions, limiting generalization across common conditions encountered in protein engineering. Practically, protein engineers are thus left wondering about the effective utility of ML tools. The FLIP benchmark established protocols for testing generalization under some domain shifts, but it was limited to measurements of stability, binding, and viral capsid viability. We introduce FLIP2, a protein fitness benchmark spanning seven new datasets, including enzymes, protein-protein interactions, and light-sensitive proteins, as well as splits that measure generalization relevant to real-world protein engineering campaigns. Evaluating a suite of benchmark models across these datasets and suites reveals that simpler models often matched or outperformed fine-tuned protein language models on \ourset, challenging the utility of existing transfer learning techniques. Provenance for all datasets has been recorded and we redistribute all data CC-BY 4.0 to facilitate continued progress.

Shaolei Zhang, Ju Fan, Meihao Fan, Yizhe Liu, Yuxin Zhang, Xiaoyong Du

Autonomous data science on the structured data has been a long-standing challenge, and is now becoming feasible with the emergence of powerful large language models (LLMs). Recent workflowbased data agents have shown promising results on specific data tasks but remain fundamentally limited in achieving full autonomy due to their reliance on predefined workflows. In this paper, we introduce DeepAnalyze, the first agentic LLM for autonomous data science, capable of automatically completing the end-to-end data science from structured data to analyst-grade research reports. To tackle high-complexity data science tasks, we propose a curriculum-based agentic training paradigm that emulates the learning trajectory of human data scientists, enabling LLMs to progressively acquire and integrate multiple capabilities in real-world environments. Accordingly, we contribute a data-grounded trajectory synthesis framework to constructs high-quality data science training data. Through training in real-world environment, DeepAnalyze learns to perform a broad spectrum of data tasks, ranging from data question answering to open-ended data research. Experiments on 13 benchmarks demonstrate that, with only 8B parameters, DeepAnalyze outperforms previous workflow-based agents built on most advanced proprietary LLMs.