论文检索

General Machine Learning · Unsupervised and Semi-supervised Learning

Xuan-Ha Nguyen, Vu Duong, VanHoi Nguyen, Kim-Hung Le, Nhien-An Le-Khac

Anomaly detection (AD) on real-world tabular data is challenged by scarce labels, diverse anomaly types, and high sensitivity to data-specific hyperparameter tuning. We propose Discriminant Vector Machine for Anomaly detection (DVM-AD), a robust one-class method derived from discriminant analysis without data-specific tuning. DVM-AD makes one-class discriminant learning effective via a deterministic reference point, then solves a generalized eigenproblem using a Moore–Penrose pseudo-inverse to remain stable under high-dimensional or rank-deficient settings. Moreover, it selects discriminant directions from both extremes of a bounded inverse-scatter spectrum, combining directions to tighten the normal class with structure-preserving directions that retain informative geometry, addressing the compactness–structure tradeoff. At inference, DVM-AD samples scores by nearest-neighbor distance in the learned space with a training-geometry normalization to yield bounded and threshold-ready scores. Across 47 ADBench tabular datasets and 10 NLP/CV embedding benchmarks against 28 baselines, DVM-AD achieves the best average AUROC (89.65\%, average rank 2.98) on tabular datasets and remains top-ranked across four anomaly types and on embedding tasks (average rank 1.60, AUROC 72.68\%).

Deep Learning · Large Language Models

Longhui Ma, Di Zhao, Siwei Wang, Zhao Lv, Miao Wang

GUI grounding maps natural language instructions to the correct interface elements, serving as the perception foundation for GUI agents. Existing approaches predominantly rely on fine-tuning multimodal large language models (MLLMs) using large-scale GUI datasets to predict target element coordinates, which is data-intensive and generalizes poorly to unseen interfaces. Recent attention-based alternatives exploit localization signals in MLLMs attention mechanisms without task-specific fine-tuning, but suffer from low reliability due to the lack of explicit and complementary spatial anchors in GUI images. To address this limitation, we propose Trifuse, an attention-based grounding framework that explicitly integrates complementary spatial anchors. Trifuse integrates attention, OCR-derived textual cues, and icon-level caption semantics via a Consensus-SinglePeak (CS) fusion strategy that enforces cross-modal agreement while retaining sharp localization peaks. Extensive evaluations on four grounding benchmarks demonstrate that Trifuse achieves strong performance without task-specific fine-tuning, substantially reducing the reliance on expensive annotated data. Moreover, ablation studies reveal that incorporating OCR and caption cues consistently improves attention-based grounding performance across different backbones, highlighting its effectiveness as a general framework for GUI grounding.

General Machine Learning · Sequential, Network, and Time Series Modeling

Junyao Hong, Zesheng Lai, Xinyi Xiao, Suyang Zhou, Aodong Shen, Youyong Kong

Knowledge Distillation (KD) promises to bridge the gap between the high computational costs of Transformer-based models and the expressiveness limitations of linear models in long-term time series forecasting. Existing time series distillation methods inherit the computer vision paradigm, constraining student models by minimizing point-wise prediction matching (output-level distillation) errors. However, blindly mimicking teacher predictions, which are often uncertain, can induce negative transfer. To address this, we propose Dynamic Structural Distillation (DSD), a robust framework that goes beyond the prediction-matching paradigm. First, we design LMP-Net, leveraging manifold expansion to project features into a high-dimensional latent space, alleviating the expressiveness bottleneck while preserving lightweight inference. Second, to address architectural mismatch, we propose Dual Manifold Alignment, employing Similarity-Preserving Knowledge Distillation (SPKD) and Optimal Transport (OT) to align features at the topological and geometric levels, respectively. Finally, we introduce Regime-Aware Adaptive Distillation (RAAD) to mitigate teacher misguidance via a dataset-level regime prior and a confidence-based adaptive gating mechanism. Extensive experiments on five benchmarks validate that DSD is compatible with diverse Transformer-based teachers, mitigating negative transfer while achieving a favorable accuracy--efficiency trade-off. An anonymized implementation is available at https://anonymous.4open.science/r/DSD-master-4B8F.

Deep Learning · Algorithms

Jiaxiao Wang, Dachun Kai, Huyue Zhu, Quanquan Hu, Zhenyang Xu, Xiaoyan Sun

Reflection removal is a highly challenging problem. Though remarkable progress has been made, current methods primarily exploit static image priors from a single frame. Due to the inherent ambiguity between layers, existing methods still suffer from severe residual artifacts. In this paper, we propose leveraging event signals to break this ambiguity. By employing event cameras to capture micro-dynamics, we reveal the differential motion between the reflection and background layers. We thereby present a novel event-driven reflection removal network, EvReflection, that utilizes these dynamic cues for layer separation. Specifically, we design a Micro-Dynamics Decoupler to disentangle layer-specific motions from event streams as priors, which then guide a Parallax-Attention Rectifier to cleanly remove artifacts from the RGB image. Furthermore, to address the data shortage, we develop a physics-based simulation pipeline and construct the EVR$^2$ benchmark, the first real-world dataset for this task. Extensive experiments demonstrate that EvReflection significantly outperforms existing methods, recovering clean images in challenging real-world scenarios.

General Machine Learning · Methodology

Rui Ai, Yuqi Pan, David Simchi-Levi, Milind Tambe, Haifeng Xu

With the rapid progress of multi-agent large language model (LLM) reasoning, how to effectively aggregate answers from multiple LLMs has emerged as a fundamental challenge. Standard majority voting treats all answers equally, failing to consider latent heterogeneity and correlation across models. In this work, we design two new aggregation algorithms called Optimal Weight (OW) and Inverse Surprising Popularity (ISP), leveraging both first-order and second-order information. Our theoretical analysis shows these methods provably mitigate the inherent limitations of majority voting under mild assumptions, leading to more reliable collective decisions. We empirically validate our algorithms on synthetic datasets, popular LLM fine-tuning benchmarks such as UltraFeedback and MMLU, and a real-world healthcare setting ARMMAN. Our algorithms consistently outperform standard baselines, establishing a robust, training-free framework for effective multi-agent LLM aggregation.

General Machine Learning · Clustering

Fangfang Li, Quanxue Gao, Xingyu Xue

Manifold clustering has demonstrated strong capability in capturing complex data structures and has been widely studied in cluster analysis. However, many existing methods mainly focus on combining K-means with manifold learning, while overlooking the consistency between data structures and clustering labels, and often suffer from high computational cost when handling large scale data. To address these issues, we propose a manifold balanced clustering method based on anchor induced distance(LMBC), grounded in the relationship between K-means clustering and manifold learning. Specifically, the LMBC uses label information to guide the construction of the manifold structure, thereby ensuring consistency between data structures and clustering labels. To enable large scale clustering, we introduce an anchor induced distance representation that models manifold structure in a compact anchor space, significantly reducing computational complexity while preserving essential structural information. Furthermore, to naturally maintain class balance during clustering, we maximize the Schatten-p norm of the label representation and provide theoretical analysis to support its effectiveness. Experimental results on several benchmark datasets demonstrate the effectiveness and scalability of the proposed method.

Shenyuan Gao, William Liang, Kaiyuan Zheng, Ayaan Malik, Seonghyeon Ye, Sihyun Yu, Wei-Cheng Tseng, Yuzhu Dong, Kaichun Mo, Chen-Hsuan Lin 等

Being able to simulate the outcomes of actions in varied environments will revolutionize the development of generalist agents at scale. However, modeling these world dynamics, especially for dexterous robotics tasks, poses significant challenges due to limited data coverage and scarce action labels. As an endeavor towards this end, we introduce DreamDojo, a foundation world model that learns diverse interactions and dexterous controls from 44k hours of egocentric human videos. Our data mixture represents the largest video dataset to date for world model pretraining, spanning a wide range of daily scenarios with diverse objects and skills. To address the scarcity of action labels, we introduce continuous latent actions as unified proxy actions, enhancing interaction knowledge transfer from unlabeled videos. After post-training on small-scale target robot data, DreamDojo demonstrates a strong understanding of physics and precise action controllability. We also devise a distillation pipeline that accelerates DreamDojo to a real-time speed of 10.93 FPS and further improves consistency to the context. Our work enables several important applications based on generative world models, including live teleoperation, policy evaluation, and model-based planning. Systematic evaluation on multiple challenging out-of-distribution (OOD) benchmarks verifies the significance of our method for simulating open-world, contact-rich tasks, paving the way for general-purpose robot world models.

Theory · Learning Theory

Marios Koulakis, Constantin Seibold

A significant gap exists between theory and practice in deep learning. Generalization and approximation error bounds are often derived for simplified models or are too loose to be informative. Many rely on the manifold hypothesis and on geometric regularity such as intrinsic dimension, curvature, and reach. Progress requires insight into data-manifold geometry and suitable benchmarks, yet existing options are polarized: analytic manifolds with known geometry but limited applicability, or real-world datasets where geometry is only coarsely estimable. We introduce a benchmarking framework for studying data geometry by repurposing and extending dSprites and COIL-20 with additional transformation dimensions and denser sampling, enabling accurate finite-difference estimates of curvature, reach, and volume that are otherwise difficult to estimate reliably and implement in practice. As applications, we assess bounds by Genovese et al. and Fefferman et al., and analyze how geometry evolves across network layers in $\beta$-VAEs, highlighting the behavior of current bounds and the value of controlled benchmarks for guiding and validating future theory. Code to reproduce the framework and experiments is included with the submission and will be released as open-source library upon publication.

Yuanchao Dai, Ximing Li, Wei Wang, Changchun Li, Gang Niu, Masashi Sugiyama

Positive-Unlabeled (PU) learning is a weakly-supervised paradigm that trains a binary classifier from labeled positive and unlabeled instances. In PU risk estimation, the empirical risk consists of an unlabeled term and a positive term. In this paper, we observe that when labeled positives are scarce, the risk deviation is dominated by the generalization bound of the positive term, which is composed of a complexity term governed by Rademacher complexity and a concentration term governed by the uniform range bound, leading to estimator instability. Based on this observation, we theoretically derive the minimal sufficient learning threshold, defined as the smallest number of labeled positives required to achieve a target excess risk with high probability, and reveal its explicit dependence on both components. Inspired by this insight, we propose ScalePU, which incorporates variance regularization to induce a restricted sub-hypothesis space with reduced Rademacher complexity, and geometric regularization to encourage compact clustering of positive samples with a tighter effective range. Theoretical analysis demonstrates that both mechanisms effectively lower the threshold through improvements to different components of the bound. Experiments on eight benchmark datasets validate the effectiveness of ScalePU, with significant improvements under extreme label scarcity.

General Machine Learning · Everything Else

Prayas Agrawal, Prateek Chanda, Ishita Khatri, Ganesh Ramakrishnan, Bamdev Mishra, Pratik Kumar Jawanpuria

Training Large Language Models (LLMs) on heterogeneous datasets requires optimizing domain representations to balance convergence speed and domain coverage. While recent methods reduce computational overhead by selecting high-quality data subsets, they typically perform selection independently per domain or rely on computationally expensive proxy models to determine continuous domain weights. In this paper, we propose a joint sample selection framework that learns better domain representations in the batch directly and formulate the selection objective as joint domain utility maximization subject to partition matroid constraints. This approach enforces domain-specific budgets while maximizing a validation-guided gradient matching utility across all domains simultaneously. Theoretically, we establish that this objective is weakly submodular, allowing us to employ a computationally efficient orthogonal matching pursuit algorithm with provable guarantees. Empirically, we demonstrate that our method significantly outperforms baselines that favor independent domain selection on mathematical reasoning and molecular generation benchmarks. Furthermore, our analysis show that our approach reduces the number of conflicting training gradient pairs significantly as compared to independent selection across domains or domain agnostic selection. Applied to Qwen2.5 and Llama-3 and trained on MethaMathQA and Mol-Instructions, our approach yields robust gains under multiple subset fractions, with improvements on nine math-reasoning and four molecule-generation benchmarks, highlighting cross-domain benefits of joint subset selection.

General Machine Learning · Kernel methods

Suleman Baloch, Kishlay Jha, Alberto Segre, Philip Polgreen, Bijaya Adhikari

Traditional comorbidity scores (e.g., Charlson and Elixhauser) are widely used for risk adjustment and patient stratification, but they have two key limitations: they are largely mortality-centric and do not align well with other outcomes, and their linear, rule-based structure cannot capture nonlinear, outcome-specific risk relationships. We propose a Machine-Learned Comorbidity Index (MLCI) that maps diagnosis codes to a single scalar by maximizing the normalized Hilbert–Schmidt Independence Criterion (nHSIC) between the score and multiple clinical outcomes. MLCI captures nonlinear risk–outcome dependence and is supported by a novel theory that characterizes when a unified, informative patient ordering can be achieved across outcomes. Empirical results on multiple benchmark electronic health record (EHR) datasets show that MLCI outperforms strong single-index baselines across multiple evaluation metrics.

Applications · Time Series

Dongyuan Li, Renhe Jiang, Shun Zheng, Zheng Dong, Haotian Gao, Ying Zhang, Jiang Bian

Time Series Foundation Models (TSFMs) have achieved strong performance in univariate time series forecasting. However, most TSFMs rely on channel-independent pre-training that models each variable separately, limiting their ability to leverage inter-channel information that is crucial in real-world multivariate systems. Motivated by this limitation, we propose Chada, a lightweight plug-and-play channel adapter that allows frozen TSFMs to leverage multivariate correlations in a zero-shot setting. Chada first builds a budgeted pre-training dataset to cover diverse heterogeneous inter-channel dependency patterns. It then uses data-derived domain descriptors to learn a dataset-conditioned inter-channel similarity measure that reduces cross-domain metric distortion. Finally, it injects sparse inter-channel information via gated refinement, leveraging multivariate information without degrading intra-channel temporal dynamics. Extensive experiments on nine benchmarks validate the effectiveness of Chada, demonstrating consistent zero-shot improvements over four best-performing TSFMs while maintaining scalable deployment. Code is available at https://anonymous.4open.science/r/CHADA-A6.

Deep Learning · Generative Models and Autoencoders

Parsa Rahimi, Sébastien Marcel

Synthetic data generation is increasingly used in machine learning for **training and data augmentation**. Yet, many current strategies rely on external foundation models or datasets, which can be restricted by policy or legal constraints, especially for sensitive modalities such as human face images and videos. We propose **ScoreMix**, a **self-contained data augmentation** method to boost recognition performance by leveraging score compositionality in class-conditioned diffusion models. ScoreMix mixes class-conditioned scores along reverse diffusion trajectories, yielding domain-specific hard augmentations without external resources. We systematically study class-selection strategies and find that mixing classes that are distant in the discriminator embedding space yields larger gains, providing **up to 3\% additional average improvement across benchmarks** over proximity-based selection. Interestingly, we observe that learned condition and embedding spaces are largely uncorrelated under standard alignment metrics, and that condition-space distances are weakly correlated to downstream gains. Across **8 public face recognition benchmarks**, ScoreMix improves accuracy by **up to 7 percentage points** without hyperparameter search, highlighting robustness and practicality. Code and dataset will be made publicly available.

Social Aspects · Safety

Xiaodan Li, Mengjie Wu, Yao Zhu, Yunna Lv, YueFeng Chen, Cen Chen, Jianmei Guo, Hui Xue'

Large models (LMs) are powerful content generators, yet their open‑ended nature can also introduce potential risks, such as generating harmful or biased content. Existing guardrails mostly perform post-hoc detection that may expose unsafe content before it is caught, and the latency constraints further push them toward lightweight models, limiting detection accuracy. In this work, we propose PlugGuard, a novel plug-in framework that enables streaming risk detection within the LM generation pipeline. PlugGuard leverages intermediate LM hidden states through a Streaming Latent Dynamics Head (SLD), which models the temporal evolution of risk across the generated sequence for more accurate real-time risk detection. To ensure reliable streaming moderation in real applications, we introduce an Anchored Temporal Consistency (ATC) loss to enforce monotonic harm predictions by embedding a benign-then-harmful temporal prior. Besides, for a rigorous evaluation of streaming guardrails, we also present StreamGuardBench—a model-grounded benchmark featuring on-the-fly responses from each protected model, reflecting real-world streaming scenarios in both text and vision–language tasks. Across diverse models and datasets, PlugGuard consistently outperforms state-of-the-art post-hoc guardrails and prior plug-in probes (15.61% higher average F1), while using only 20M parameters and adding less than 0.5 ms of per-token latency. The code and StreamGuardBench are released at PlugGuard to facilitate research on streaming guardrails.

Applications · Computer Vision

Jingtong Dou, Chuancheng Shi, Yemin Wang, Shiming Guo, Anqi Yi, Wenhua Wu, Li Zhang, Fei Shen, Tat-Seng Chua

As generative AI achieves hyper-realism, superficial artifact detection has become obsolete. While prevailing methods rely on resource-intensive fine-tuning of black-box backbones, we propose that forgery detection capability is already encoded within pre-trained models rather than requiring end-to-end retraining. To elicit this intrinsic capability, we propose the discriminative neural anchors (DNA) framework, which employs a coarse-to-fine excavation mechanism. First, by analyzing feature decoupling and attention distribution shifts, we pinpoint critical intermediate layers where the focus of the model logically transitions from global semantics to local anomalies. Subsequently, we introduce a triadic fusion scoring metric paired with a curvature-truncation strategy to strip away semantic redundancy, precisely isolating the forgery-discriminative units (FDUs) inherently imprinted with sensitivity to forgery traces. Moreover, we introduce HIFI-Gen, a high-fidelity synthetic benchmark built upon the very latest models, to address the lag in existing datasets. Experiments demonstrate that by solely relying on these anchors, DNA achieves superior detection performance even under few-shot conditions. Furthermore, it exhibits remarkable robustness across diverse architectures and against unseen generative models, validating that waking up latent neurons is more effective than extensive fine-tuning.

Deep Learning · Foundation Models

Guri Zabërgja, Rafiq Kamel, Arlind Kadra, Christian Frey, Josif Grabocka

The long-standing dominance of gradient-boosted decision trees for tabular data has recently been challenged by in-context learning tabular foundation models. In-context learning methods fit and predict in one forward pass without parameter updates by leveraging the training data as context for predicting on query test points. While recent tabular foundation models achieve state-of-the-art performance, their transformer architecture based on the attention mechanism has quadratic complexity regarding dataset size, which in turn increases the overhead on training and inference time, and limits the capacity of the models to handle large-scale datasets. In this work, we propose TACO, an end-to-end tabular compression model that compresses the training dataset in a latent space. We test our method on the TabArena benchmark, where our proposed method is up to 94x faster in inference time, while consuming up to 97% less memory compared to the state-of-the-art tabular Transformer architecture, all while retaining performance without significant degradation. Lastly, our method not only scales better with increased dataset sizes, but it also achieves better performance compared to other baselines.

Deep Learning · Sequential Models, Time series

Yubin Kim, Viresh Pati, Jevon Twitty, Vinh Pham, Shihao Yang, Jiecheng Lu

Transformer architectures have established strong baselines in time series forecasting, yet they typically rely on positional encodings that assume uniform, index-based temporal progression. However, real-world systems, from shifting financial cycles to elastic biological rhythms, frequently exhibit ``time-warped'' dynamics where the effective flow of time decouples from the sampling index. In this work, we first formalize this misalignment and prove that rotary position embedding (RoPE) is mathematically incapable of representing non-affine temporal warping. To address this, we propose Symplectic Positional Embeddings (SyPE), a learnable encoding framework derived from Hamiltonian mechanics. SyPE strictly generalizes RoPE by extending the rotation group $\mathrm{SO}(2)$ to the symplectic group $\mathrm{Sp}(2,\mathbb{R})$, modulated by a novel input-dependent adaptive warp module. By allowing the attention mechanism to adaptively dilate or contract temporal coordinates end-to-end, our approach captures locally varying periodicities without requiring pre-defined warping functions. We implement this mechanism in StretchTime, a multivariate forecasting architecture that achieves state-of-the-art performance on standard benchmarks, demonstrating superior robustness on datasets exhibiting non-stationary temporal dynamics.

Applications · Computer Vision

Keyun Zhao, Guangchang Li, Yunpeng Bai, shao jiang, Ying Li

Hyperspectral image change detection (HSI-CD) suffers from severe annotation scarcity and complex change patterns, which fundamentally limit the effectiveness of directly fine-tuning pre-trained foundation models. Although semi-supervised learning provides a promising direction, existing approaches mainly rely on confidence-based pseudo-label selection, leading to limited data diversity or severe error propagation. In this paper, we propose Influence-Aware Semi-supervised Fine-tuning (IA-SFT), a novel framework that evaluates the influence of pseudo-labels on model decision behavior to identify truly valuable supervision signals. Instead of confidence-based selection, IA-SFT evaluates each low-confidence pseudo-label by measuring its impact on labeled data, enabling reliable filtering of high-value pseudo-labels with minimal noise. To further adapt foundation models to HSI-CD, we design an Adaptive Fusion Change Decoder (AFCD) that jointly models global semantic consistency and local change details. Extensive experiments on three benchmark datasets demonstrate that IA-SFT consistently improves pseudo-label quality and detection performance, achieving superior accuracy compared to state-of-the-art methods. Additional analyses validate the transferability of IA-SFT when integrated into different frameworks in a plug-and-play manner. Code will be released.

General Machine Learning · Online Learning, Active Learning and Bandits

Pascal Janetzky, Dr. Tobias Schlagenhauf, Stefan Feuerriegel

Continual learning (CL) aims to train models sequentially over multiple domains without forgetting previously learned knowledge. However, existing CL methods optimize for in-domain performance and are therefore prone to learning spurious, domain-specific cues (``shortcut learning''), which limits generalization to unseen domains after deployment. In this paper, we address this limitation through *continual learning of domain-invariant representation*. We introduce a broad class of CL methods that sequentially learn representations capturing invariant structures across domains. Our methods are motivated by the observation that such invariant structures often preserve the underlying causal mechanisms, which can reduce the risk of overfitting to domain-specific cues and thus offer better out-of-domain generalization. Our proposed CL methods combine replay-based training with a tailored sequential invariance alignment to learn---and preserve---invariant structures over time. We evaluate our methods under a deployment-oriented protocol that measures performance on unseen target domains. Across six benchmark and real-world datasets spanning vision, medicine, manufacturing, and ecology, our methods consistently outperform existing CL baselines in terms of generalization to unseen target domains. As an ablation, we further show that na\"ive extensions of sequential training with existing domain-invariant representation learning (DIRL) methods provide only limited benefits. To the best of our knowledge, this is the first work to develop domain-invariant representation methods for CL.

Deep Learning · Large Language Models

Hippolyte Pilchen, Romain Fabre, Franck SIGNE TALLA, Patrick Perez, Edouard Grave

Large language models (LLMs) are typically trained on shuffled corpora, yielding models whose knowledge is frozen at training time and whose temporal grounding remains poorly understood. In this work, we study the impact of pretraining dynamics on the acquisition of time-sensitive factual knowledge, focusing specifically on data ordering. Our main contributions are twofold. First, we introduce a comprehensive benchmark of over 7,000 temporally grounded questions and an evaluation protocol that enables analysis of whether models correctly associate facts with their corresponding time periods. Second, we pretrain 6B-parameter language models on temporally ordered Common Crawl snapshots and compare them against standard shuffled pretraining. Our results show that sequentially trained models match shuffled baselines on general language understanding and common knowledge while consistently exhibiting more up-to-date and temporally precise knowledge. Temporally ordered pretraining yields improved factual freshness, while shuffled pretraining peaks on older data, possibly due to increased factual repetition. These findings, along with the release of our checkpoints and datasets, provide a foundation for future research on continual learning for large language models.