论文检索

Optimization · Non-Convex

Alexander Tyurin

Even for the gradient descent (GD) method applied to neural network training, understanding its optimization dynamics, including convergence rate, iterate trajectories, function value oscillations, and especially its implicit acceleration, remains a challenging problem. We analyze nonlinear models with the logistic loss and show that the steps of GD reduce to those of generalized perceptron algorithms (Rosenblatt, 1958), providing a new perspective on the dynamics. This reduction yields significantly simpler algorithmic steps, which we analyze using classical linear algebra tools. Using these tools, we demonstrate on a minimalistic example that the nonlinearity in a two-layer model can provably yield a faster iteration complexity $\tilde{\mathcal{O}}(\sqrt{d})$ compared to $\Omega(d)$ achieved by linear models, where $d$ is the number of features. This helps explain the optimization dynamics and the implicit acceleration phenomenon observed in neural networks. The theoretical results are supported by extensive numerical experiments. We believe that this alternative view will further advance research on the optimization of neural networks.

Theory · Probabilistic Methods

Weihao Li, Jun Liu

Logistic regression is widely used in applications; however, when the dimension scales with the sample size, theory reveals that the asymptotic behavior of common M-estimators depends on bias and variance scaling constants, which are functions of the signal strength. To leverage the theory to design statistical methodologies, it is essential to obtain accurate estimates of the signal strength. In this work, we utilize a data-splitting strategy to efficiently estimate the signal strength. To alleviate issues caused by separable data, we analyze the exact asymptotics of an M-estimator with a data-driven, non-decomposable regularizer that adapts to the true covariance structure. We justify the validity of our method through both theoretical analysis and numerical experiments.

Deep Learning · Generative Models and Autoencoders

huayang Huang, Ruoyu Wang, Jinhui Zhao, Wei Deng, Daiguo Zhou, Jian Luan, Yu Wu, Ye Zhu

Generative distillation significantly accelerates text-to-image (T2I) generation by compressing multi-step trajectories into few-step student models while preserving perceptual quality. However, existing distillation methods prioritize efficiency and output fidelity, often overlooking the preservation of critical properties inherent to the original trajectory. In this work, we identify a key lost property: sensitivity to initial noise, the absence of which impairs downstream control methods that rely on noise-based optimization and manipulation. We trace this deficiency to standard distillation objectives, which enforce pointwise output alignment. This inadvertently flattens the input-output landscape and suppresses the local geometric structure present in the teacher model. To address this, we propose Geometry-Aware Distillation (GAD), a sensitivity-preserving framework that explicitly aligns the local functional behavior of the teacher and student. GAD enforces geometric consistency by matching Jacobian-vector products with respect to input noise, ensuring the student faithfully reproduces the teacher’s differential response to perturbations. Extensive experiments across multiple T2I paradigms and noise-driven control tasks demonstrate that GAD significantly recovers sensitivity and improves diversity, while maintaining high visual fidelity.

Applications · Computer Vision

Ye Zhu, Kaleb Newman, Johannes Lutzeyer, Adriana Romero-Soriano, Michal Drozdzal, Olga Russakovsky

Despite high semantic alignment, modern text-to-image (T2I) generative models still struggle to synthesize diverse images from a given prompt. This lack of diversity not only restricts user choice, but also risks amplifying societal biases. In this work, we enhance the T2I diversity through a geometric lens. Unlike most existing methods that rely primarily on entropy-based guidance to increase sample dissimilarity, we introduce Geometry-Aware Spherical Sampling (GASS) to enhance diversity by explicitly controlling both prompt-dependent and prompt-independent sources of variation. Specifically, we decompose the diversity measure in CLIP embeddings using two orthogonal directions: the text embedding, which captures semantic variation related to the prompt, and an identified orthogonal direction that captures prompt-independent variation (e.g., backgrounds). Based on this decomposition, GASS increases the geometric projection spread of generated image embeddings along both axes and guides the T2I sampling process via expanded predictions along the generation trajectory. Our experiments on different frozen T2I backbones (U-Net and DiT, diffusion and flow) and benchmarks demonstrate the effectiveness of disentangled diversity enhancement with minimal impact on image fidelity and semantic alignment.

Theory · Optimization

Marek Elias, Fabrizio Grandoni, Adam Polak, Eleonora Vercesi

The Traveling Salesperson Problem (TSP) has long served as a benchmark for evaluating the strength of optimization techniques in the classical theory of algorithms. In recent efforts to apply ML to algorithmic problems, TSP has also become a natural testbed for the development of ML-based techniques. A common approach is to train a neural network to output a heatmap estimating the likelihood of each edge to be part of the optimal tour; however, converting such a heatmap into an actual tour remains a non-trivial and often computationally intensive step. In this work, we propose algorithms for transforming heatmaps into tours with theoretical guarantees linking the achieved approximation ratio to the quality of the provided heatmap. In the spirit of \emph{algorithms with predictions}, our results can be described as $(1+2\eta/OPT)$-approximation algorithms, where $\eta$ denotes the L1 distance between the prediction (heatmap) and an optimal solution (tour). Since the previous works lack such explicit guarantees, we compare our approach against them experimentally.

Applications · Computer Vision

Yin Tang, Jiawei Ma, Jinrui Zhang, Alex Jinpeng Wang, Deyu Zhang

Continuous prediction in complex environments is critical for Unmanned Aerial Vehicle (UAV). However, the existing Vision-Language Navigation (VLN) models follows the dead-reckoning, which iteratively predicts the next waypoint and updates its position, thereby constructing the complete trajectory. Then, such stepwise manner will inevitably lead to accumulated errors of position over time, resulting in misalignment between internal belief and objective coordinates, which is known as ``state drift'' and ultimately compromises the subsequent trajectory prediction. Drawing inspiration from classical control theory, we propose to correct for errors by formulating the continuous prediction as a recursive Bayesian state estimation problem. In this paper, we design NeuroKalman, a novel framework that decouples navigation into two complementary processes: a Prior Prediction, based on motion dynamic,s and a Likelihood Correction, from historical observation. We first mathematically associate Kernel Density Estimation of the measurement likelihood with the attention-based retrieval mechanism, which then allows the system to rectify the latent representation using retrieved historical anchors without gradient updates. Comprehensive experiments on TravelUAV benchmark demonstrate that, with only 10\% of the full training data fine-tuning, our method clearly outperforms strong baselines and regulates drift accumulation.

Applications · Language, Speech and Dialog

Linjuan Wu, Ruiqi Zhang, Xinze Lyu, Ye Guo, Daoxin Zhang, Zhe Xu, Yao Hu, Yixin Cao, Yongliang Shen, Weiming Lu

Social media platforms enable large-scale cross-lingual communication, yet translating user-generated content (UGC) remains challenging due to its informal style, culture-laden expressions, and interaction-driven nuances. While recent LLMs have advanced translation quality, existing benchmarks and metrics often overlook whether translations preserve intended meaning and cultural resonance in real-world contexts. In this work, we introduce **CULTURE-MT**, a benchmark for social media translation that explicitly emphasizes **CUL**tural **T**ransmission and **U**GC-specific emotion **RE**sonance. CULTURE-MT comprises 1,002 Chinese-to-English UGC notes spanning 14 domains, systematically categorized into four types based on culture-loaded symbols and linguistic styles. We also construct UGC-oriented training data to fine-tune Qwen3-8B and Qwen3-32B as strong baselines. We propose the **cultural effectiveness** criterion and train a related JUDGER model that jointly assesses expression accuracy and cultural adaptability. Evaluating 15 models, we find that standard automatic metrics are largely insensitive to cultural effectiveness. Our work establishes a comprehensive framework for evaluating and advancing UGC translation, and will provide an open evaluation platform to support future research in culturally effective UGC translation.

General Machine Learning · Causality

Zheng Li, Feng Xie, Shenglan Nie, Xichen Guo, Ruxin Wang, Hao Zhang

Constraint-based causal discovery is widely used for learning causal structures, but heavy reliance on conditional independence (CI) testing makes it computationally expensive in high-dimensional settings. To mitigate this limitation, many divide-and-conquer frameworks have been proposed, but most assume causal sufficiency, i.e., no latent variables. In this paper, we show that divide-and-conquer strategies can be theoretically generalized beyond causal sufficiency to settings with latent variables. Specifically, we propose a recursive decomposition framework, termed DiCoLa, that enables divide-and-conquer causal discovery in the presence of latent variables. It recursively decomposes the global learning task into smaller subproblems and integrates their solutions through a principled reconstruction step to recover the global structure. We theoretically establish the soundness and completeness of the proposed framework. Extensive experiments on synthetic data demonstrate that our approach significantly improves computational efficiency across a range of causal discovery algorithms, while experiments on a real-world dataset further illustrate its practical effectiveness.

General Machine Learning · Clustering

Zheming Xu, Aiyue Tang, Shidi Chen, Xuechao Zou, Congyan Lang, Rogelio A. Mancisidor, Michael Kampffmeyer

Incomplete multi-view clustering (IMVC) aims to uncover shared cluster structures from data with partially observed views. Although recent imputation-free methods based on variational inference demonstrate robustness to missing views, they commonly rely on a conditional independence assumption across views, which fails to capture the inherently structured and potentially correlated nature of multi-view data. In this paper, we propose a variational framework that explicitly goes beyond this assumption by introducing a learnable cross-view correlation structure. Specifically, we explicitly model and learn correlations between views by utilizing the covariance structure of posterior estimation errors. To facilitate robust and efficient learning, the correlation matrix is parameterized through a normalized Cholesky decomposition, ensuring positive definiteness and enabling the entire model to be trained jointly through a unified variational objective. Extensive experiments on multiple IMVC benchmarks demonstrate that our method consistently outperforms state-of-the-art approaches across a wide range of missing-view settings. These results highlight the effectiveness of adaptive correlation modeling in variational IMVC, demonstrating the need to go beyond the independence assumption in IMVC.

General Machine Learning · Clustering

Xiaohan Zhang, Chao Zhang, Chunlin Chen, Huaxiong Li

Image clustering is a fundamental unsupervised task in computer vision. Recent studies have explored incorporating external linguistic information to facilitate visual feature learning and thereby enhance clustering performance. Nevertheless, these methods typically rely on fixed vocabularies (e.g., WordNet) to generate language counterparts, leading to inter-modal semantic misalignment due to granularity discrepancy between visual and textual semantics. Moreover, they often overlook the issue of intra-modal semantic redundancy caused by task-irrelevant knowledge. To address these challenges, we propose a new Multi-grAnularity lanGuage-informed Image Clustering method, dubbed MAGIC. To reduce semantic misalignment, we first prompt the vision-language models to generate multi-granularity language descriptions that capture rich image semantics, which are then integrated for effective multi-modal alignment. To alleviate semantic redundancy, we design modality-specific semantic adapters that adaptively refine and compress the semantically dense features into clustering-friendly representations under task guidance. A consensus representation is obtained by fusing the refined visual and textual features, which acts as a teacher to guide image clustering through a robust contrastive learning framework. Extensive experiments on benchmarks demonstrate that MAGIC outperforms state-of-the-art methods.

Applications · Everything Else

Ruotong Zhao, Zhiyu Chen, Xurui Liu, Haidong Xue, Dong Liang, Jigao Fu, Wu YanBiao, Yuanyi Zhen, Fengli Xu, Yong Li

Literature reviews are essential to reflect the landscape of research fields. Large language models, especially deep research agents, have recently shown strong capabilities in automated literature review generation. However, it remains a challenging task to rigorously evaluate the scientific value of the generated reviews, since human expert annotations are difficult to scale up and LLM-as-a-judge approaches lack of a convincing criteria. To address this gap, we introduce LitReview Arena, a battle-style evaluation platform with a structured protocol tailored to literature review quality. Our protocol imitates academic peer review by recruiting domain experts with research paper-writing experience, and we match each query to reviewers within their expertise. Each battle is judged with dimension-wise outcomes over five literature-review-specific criteria, enabling reproducible and diagnostic comparisons across systems. We collect a large-scale human preference dataset of expert votes (4984 votes×5 dimensions) and systematically measure how far current models are from human drafts. Results show that the most advanced models win only 23.0\% of decisive matches against humans on overall utility, leaving substantial room for improvement. Meanwhile, agentic LLMs, such as Sonar Deep Research, substantially outperform base language models by over 60\%. We also find that existing LLM-as-a-judge evaluation methods are severely misaligned with human experts (Spearman's $\rho \approx 0.467$). Based on the collected preference data, we provide an expert-calibrated evaluator, \emph{LitJudge}, improving alignment to $\rho \approx 0.78$, comparable to inter-expert consistency. Codes and datasets are publicly available at https://anonymous.4open.science/r/LitReview-Arena-3B82/.

General Machine Learning · Clustering

Youqing Wang, Tianxiang Zhao, Mengyuan Xin, Ye Su, Jiapu Wang, Tengfei Liu, Junbin Gao, Jipeng Guo

Deep multi-view graph clustering (DMGC) typically leverages graph neural networks for representation learning, but most existing methods excessively depend on local and static graph structures and only utilize simplistic cross-view fusion strategies. To this end, this paper proposes **A**ttribute-aware Bidirectional Structural Refinement (ABSR) and **P**seudo-label Guided Multi-level Fusion (PGMF) for DM**GC**, termed **APGC**. Specifically, ABSR selectively strengthens high-quality connections and suppresses semantically conflicting relationships, achieving bidirectional refinement of the graph structure based on attribute similarity. It incorporates global attribute semantics into the graph structure, thereby promoting the homophilic connections for discriminative graph representation learning. Guided by reliable pseudo-labels, PGMF achieves adaptive weighted fusion at both the node-level and the view-level, effectively balancing the differentiated contributions of multi-view information. Experiments on six homophilic and heterophilic datasets demonstrate the superior clustering performance of the proposed APGC method.

Applications · Computer Vision

Muyu Liu, Xuanyu Tian, Chenhe Du, Qing Wu, Hongjiang Wei, Yuyao Zhang

Recovering radiometric fidelity from unknown dynamic range compression (UDRC), such as low-light enhancement and HDR reconstruction, is a challenging blind inverse problem, due to the unknown forward model and irreversible information loss introduced by compression. To address this challenge, we first identify monotonicity as the fundamental physical invariant shared across UDRC tasks. Leveraging this insight, we introduce the cascaded monotonic Bernstein (CaMB) operator to parameterize the unknown forward model. CaMB enforces monotonicity as a hard architectural inductive bias, constraining optimization to physically consistent mappings and enabling robust and stable operator estimation. We further integrate CaMB with a plug-and-play diffusion framework, proposing CaMB-Diff. Within this framework, the diffusion model serves as a powerful geometric prior for structural and semantic recovery, while CaMB explicitly models and corrects radiometric distortions through a physically grounded forward operator. Extensive experiments on a variety of zero-shot UDRC tasks, including low-light enhancement, low-field MRI enhancement, and HDR reconstruction, demonstrate that CaMB-Diff significantly outperforms state-of-the-art zero-shot baselines in terms of both signal fidelity and physical consistency. Moreover, we empirically validate the effectiveness of the proposed CaMB parameterization in accurately modeling the unknown forward operator.

Haowen Li, Tianxiang Li, Yi Yang, Boyu Cao, Qi Liu

The advancement of diffusion-based text-to-music generation has opened new avenues for zero-shot music editing. However, existing methods fail to achieve context-aware editing, which requires altering specific stems while strictly preserving the background accompaniment. This limitation severely hinders practical application, since real-world production necessitates precise manipulation of components within dense mixtures. Our key finding is that, while vanilla cross-attention captures semantic intent, it lacks the spectral resolution to strictly localize targets in dense mixtures, leading to boundary leakage. To resolve this dilemma, we propose Polyphonia, a training-free editing framework with Acoustic-Informed Attention Calibration. Rather than relying solely on diffuse semantic attention, Polyphonia leverages a probabilistic acoustic prior to establish coarse boundaries, enabling background context preserved precise semantic synthesis. For evaluation, we propose PolyEvalPrompts, a standardized prompt set with 1,170 context-aware music editing tasks. Specifically, Polyphonia achieves an increase of 15.5% in target alignment compared to baselines, while maintaining competitive music fidelity and background integrity.

Applications · Health / Medicine

Chenhe Du, Xuanyu Tian, Qing Wu, Muyu Liu, Jingyi Yu, Hongjiang Wei, Yuyao Zhang

Plug-and-Play diffusion prior (PnPDP) frameworks have emerged as a powerful paradigm for solving imaging inverse problems by treating pretrained generative models as modular priors. However, we identify a critical flaw in prevailing PnP solvers (e.g., based on HQS or Proximal Gradient): they function as memoryless operators, updating estimates solely based on instantaneous gradients. This lack of historical tracking inevitably leads to non-vanishing steady-state bias, where the reconstruction fails to strictly satisfy physical measurements under heavy corruption. To resolve this, we propose **Dual-Coupled PnP Diffusion**, which restores the classical dual variable to provide integral feedback, theoretically guaranteeing asymptotic convergence to the exact data manifold. However, this rigorous geometric coupling introduces a secondary challenge: the accumulated dual residuals exhibit spectrally colored, structured artifacts that violate the Additive White Gaussian Noise (AWGN) assumption of diffusion priors, causing severe hallucinations. To bridge this gap, we introduce Spectral Homogenization (SH), a frequency-domain adaptation mechanism that modulates these structured residuals into statistically compliant pseudo-AWGN inputs. This effectively aligns the solver's rigorous optimization trajectory with the denoiser's valid statistical manifold. Extensive experiments on CT and MRI reconstruction demonstrate that our approach resolves the bias-hallucination trade-off, achieving state-of-the-art fidelity with significantly accelerated convergence.

Deep Learning · Large Language Models

Chenhui Hu, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao

The fundamental representational units (FRUs) of large language models (LLMs) remain undefined, limiting further understanding of their underlying mechanisms. In this paper, we introduce ***Atom Theory*** to systematically define, evaluate, and identify such FRUs, which we term atoms. Building on the atomic inner product (AIP), a non-Euclidean metric that captures the underlying geometry of LLM representations, we formally define atoms and propose two key criteria for ideal atoms: faithfulness ( $ R^2 $ ) and stability ( $ q^{\star} $ ). We further prove that atoms are identifiable under threshold-activated sparse autoencoders (TSAEs). Empirically, we uncover a pervasive representation shift in LLMs and demonstrate that the AIP corrects this shift to capture the underlying representational geometry, thereby grounding Atom Theory. We find that two widely used units, neurons and features, fail to qualify as ideal atoms: neurons are faithful ( $ R^2=1 $ ) but unstable ( $ q^{\star}=0.5 $% ), while features are more stable ( $ q^{\star}=68.2 $% ) but unfaithful ( $ R^2=48.8 $% ). To find atoms of LLMs, leveraging atom identifiability under TSAEs, we show via large-scale experiments that reliable atom identification occurs only when the TSAE capacity matches the data scale. Guided by this insight, we identify FRUs with near-perfect faithfulness ( $ R^2=99.9 $% ) and stability ( $ q^{\star}=99.8 $% ) across layers of Gemma2-2B, Gemma2-9B, and Llama3.1-8B, satisfying the criteria of ideal atoms statistically. Further analysis confirms that these atoms align with theoretical expectations and exhibit substantially higher monosemanticity. Overall, we propose and validate Atom Theory as a foundation for understanding the internal representations of LLMs.

Deep Learning · Generative Models and Autoencoders

Senmao Li, Kai Wang, Salman Khan, Fahad Khan, jian Yang, Yaxing Wang

Visual Autoregressive (VAR) modeling departs from the next-token prediction paradigm of traditional Autoregressive (AR) models through next-scale prediction, enabling high-quality image generation. However, the VAR paradigm suffers from sharply increased computational complexity and running time at large-scale steps. Although existing acceleration methods reduce runtime for large-scale steps, but rely on manual step selection and overlook the varying importance of different stages in the generation process. To address this challenge, we present FasterVAR, a systematic study and stage-aware acceleration framework for VAR models. Our analysis shows that early steps are critical for preserving semantic and structural consistency and should remain intact, while later steps mainly refine details and can be pruned or approximated for acceleration. Building on these insights, FasterVAR introduces a plug-and-play acceleration strategy that exploits semantic irrelevance and low-rank properties in late-stage computations, without requiring additional training. Our proposed FasterVAR achieves up to 3.4× speedup with only a 0.01 drop on GenEval and a 0.26 decrease on DPG, consistently outperforming existing acceleration baselines. These results highlight stage-aware design as a powerful principle for efficient visual autoregressive image generation. Our codes will be open-sourced.

General Machine Learning · Clustering

Yuwei Bian, Shidong Wang, Haofeng Zhang

Generalized Category Discovery (GCD) aims to learn semantically structured representations for discovering novel categories in unlabeled data using supervision from known classes. Most existing methods rely on self-supervised contrastive learning (CL) with consistency and uniformity objectives. We identify an inherent optimization conflict between these objectives: while uniformity enforces global feature dispersion, it can hinder the formation of class-discriminative and semantically coherent structures. To address this issue, we propose a two-stage framework that decouples representation learning from self-contrastive regularization. The first stage learns category-anchored representations aligned with known class prototypes, while the second stage extends the representation space to novel categories via a consistency objective enhanced with unified semantic regularization. We further introduce a Semantic Exploration Energy mechanism to capture shared semantics across categories and mitigate information loss caused by prototype orthogonalization. The resulting framework, termed *Consistency-under-Unified Semantic Regularization*(**CURE**), achieves state-of-the-art performance on multiple benchmarks and substantially reduces the performance gap between known and novel categories.

Zhaoyang Li, Yanjun Li, Wangkai Li, Yujia Chen, Tianzhu Zhang

Vision-Language Models (VLMs) are costly at inference time because they must process long sequences of visual tokens. Existing token pruning methods often degrade under high compression by blindly discarding information, breaking spatial structure or collapsing diversity. We propose SpecFlow, a training-free framework that shifts the paradigm from destructive pruning to conservative condensation, strictly enforcing spatial coverage and statistical conservation to ensure stability. Treating visual tokens as nodes in a $k$NN graph, SpecFlow (i) computes a stable importance field via spectral heat flow to preserve structural coherence, (ii) allocates budgets via adaptive spatial partitioning to guarantee coverage, and (iii) aggregates discarded information into coreset sinks to maintain statistical conservation. The method is plug-and-play, requires no fine-tuning, and is compatible with FlashAttention. Experiments confirm that our SpecFlow outperforms SOTA methods across tasks, VLM architectures, and pruning ratios. Notably, LLaVA-1.5 with SpecFlow retains 95.6\% of original performance despite pruning 88.9\% of visual tokens, offering an exceptional efficiency-accuracy balance.

Reinforcement Learning · Batch/Offline

Ritam Majumdar, Finale Doshi-Velez, Sonali Parbhoo

Off-policy evaluation (OPE) is essential for deploying reinforcement learning in safety-critical settings, yet existing estimators such as importance sampling and doubly robust (DR) often exhibit prohibitively high variance when rewards are sparse. In this work, we introduce Reward-Shaping Control Variates, a new family of unbiased estimators that leverage potential-based reward shaping to construct additional zero-mean control variates. We prove that shaped estimators always yields valid variance reduction, and that combining shaping-based and Q-based control variates strictly expands the variance-reduction subspace beyond DR and its minimax variant MRDR. Empirically, we provide a systematic regime map across synthetic chains, a cancer simulator, 5 single-stock and 1 multi-stock DOW-30 trading environments and an ICU-sepsis benchmark showing that shaping-based OPE consistently outperforms DR in sparse-reward settings, while a hybrid estimator achieves state-of-the-art performance across sparse, noisy, and misspecified environments. Our results highlight reward shaping as a powerful and interpretable tool for robust OPE, offering both theoretical guarantees and practical improvements in domains where standard estimators fail.