论文检索

General Machine Learning · Transfer, Multitask and Meta-learning

Chengfeng Wu, Jingge Wang, Yanru Wu, Tao Zou

Multi-task learning (MTL) aims to construct a joint model for multiple tasks by sharing a common representation across domains. To achieve this goal, existing optimization-centric methods either balance task gradients or modify the shared architecture. However, as these approaches remain agnostic to the content of the shared representation, they fail to disentangle task-relevant structure from spurious context, leading to negative transfer and poor generalization. To overcome this limitation, we propose Causal Orthogonal Representations for Multi-Task Learning (CORE-MTL) , a representation-centric framework that structurally disentangles the shared representation into semantic and residual streams, concentrating task-relevant structure in the semantic stream while relegating nuisance variation to the residual stream. We instantiate this framework in the visual domain by leveraging physical priors for structured scenes and statistical constraints for attributes. Theoretically, our method enjoys a tighter out-of-distribution generalization bound than optimization-centric methods and reduces task gradient interference without explicit gradient projection or reweighting. Empirically, CORE-MTL consistently outperforms existing methods on visual multi-task benchmarks in both in-distribution and out-of-distribution settings.

General Machine Learning · Transfer, Multitask and Meta-learning

Yuta Kobayashi, Zilin Jing, Jiayu Yao, Hongseok Namkoong, Shalmali Joshi

Active feature acquisition (AFA) is a sequential decision-making problem where the goal is to improve model performance for test instances by adaptively selecting which features to acquire. In practice, AFA methods often learn from retrospective data with systematic missingness in the features and limited task-specific labels. To address this limitation, we introduce Learning-to-Measure (L2M), which consists of i) reliable uncertainty quantification over unseen tasks, and ii) an uncertainty-guided feature acquisition agent that maximizes conditional mutual information. We demonstrate an autoregressive pre-training approach that underpins reliable uncertainty quantification and acquisition for tasks with arbitrary missingness. L2M operates directly on datasets with retrospective missingness and performs the task in-context, eliminating per-task retraining. Across synthetic and real-world tabular benchmarks, L2M matches or surpasses task-specific baselines, particularly under scarce labels and high missingness.

General Machine Learning · Transfer, Multitask and Meta-learning

Zilin Du, Junqi Zhao, Albert Boyang Li

Synthetic data are increasingly used to train image classifiers, yet distributional mismatch with real data limits their effectiveness when used indiscriminately. A common strategy is to learn data weights via bi-level optimization, which we refer to as Meta-learning for Training-data Selection (MTS). Interestingly, in practice, MTS often performs below expectation. We identify two obstacles in properly training MTS: a poor gradient signal-to-noise ratio (GSNR), which causes optimization difficulties, and lack of informative features that correlates with data quality. We present a thorough mathematical analysis of MTS, which reveals the dynamics of normalized data weights and the relation between disparate data quality and poor GSNR. The analysis suggests a a simple yet effective solution: increasing the batch size. Further, we propose a set of informative features that capture the positions of training data in their distributions and training dynamics. Experiments across four benchmarks show consistent improvements, achieving average gains of 5.49\% over training without selection and 2.89\% over the strongest baseline.

General Machine Learning · Transfer, Multitask and Meta-learning

Zimo Zhai, Manjie Xu, Wei Liang

Large language models have shown strong reasoning abilities and are increasingly explored as high-level coordinators for multi-agent systems. However, directly deploying LLMs for coordination remains challenging, as effective policies often fail to reliably emerge at the low-level control level, and inference costs limit scalability. We propose SynCoord (Synthetic Coordination Distillation), a self-supervised pipeline that distills task-level decision-making for cooperation from high-capacity reasoning models into lightweight agent policies. Our approach does not rely on explicit supervision or handcrafted coordination rules. Instead, we define a set of task-level tool interfaces that constrain LLM interaction and enable the collection of interaction trajectories, which are then used to train compact coordinated policies. This distillation process transfers coordination behaviors that are difficult to elicit through prompting alone, while substantially reducing inference overhead at execution time. We evaluate our method in the multi-agent cooperation benchmark Overcooked-AI with varying team sizes and environment layouts. Experimental results show that the distilled policies achieve success rates and efficiency comparable to reinforcement learning–based methods, while exhibiting fewer erroneous or redundant actions and generalizing across team sizes without retraining.

General Machine Learning · Transfer, Multitask and Meta-learning

Zhiming Zhang, Rong Yin, Xiaoshuai Hao, Hansong Zhang, Hao Peng, Yong Liu, Can Ma, Dan Meng

Merging multiple low-rank adapters (LoRA) provides a practical route to scaling multi-task learning and deployment more efficiently than full-model weight merging, while avoiding reliance on task-specific training data. However, most existing approaches either treat LoRA updates as dense weight deltas or depend on expensive subspace factorizations, making the merge step a primary latency bottleneck. To address this issue, this paper establishes a theoretically positive relationship between merging quality and the effective rank of the matrices being merged. Motivated by this insight, we propose **CircuMerge**, a sketch-based framework for low-rank model merging built on shared circulant transforms. Especially, this approach treats each adapter as a pair of low-rank matrices and applies a shared circulant transform to align all tasks in a common coordinate system. This alignment enables more efficient sampling, allowing us to generate compact sketches that effectively summarize the interactions between tasks. These compact sketches enable applying the merging rules directly to them and reconstructing a standard low-rank adapter, preserving the essential information while significantly reducing computational overhead. Across a broad multi-task LoRA benchmarks covering both vision and language settings, extensive empirical results demonstrate that CircuMerge reduces the overall merging time by at least 44\% compared to the state-of-the-art approaches, with accuracy matching or exceeding the optimal level.

General Machine Learning · Transfer, Multitask and Meta-learning

Junpeng Ren, Carlos Misael Madrid Padilla, Yanzhen Chen, OSCAR HERNAN MADRID PADILLA

This paper develops a general transfer learning framework for nonparametric regression with heterogeneous data consisting of multiple groups. Under the assumption that groups share a common structure along with group-specific deviations in additive form, the proposed method employs a two-stage offset learning procedure: the first stage pools data from all groups to estimate an overall mean function, and the second stage estimates offsets for each group, yielding final group-level estimators through additive combination. Non-asymptotic upper bounds are established for the proposed framework, covering a broad class of nonparametric estimators under mild complexity and noise conditions. When instantiated with deep ReLU networks, explicit convergence rates are derived under hierarchical composition models, demonstrating the ability to overcome the curse of dimensionality. Conditions that enable positive transfer with faster rates are considered, including learning with simpler functions and data augmentation through pooling samples across groups. Various simulations and real-data experiments further validate the effectiveness of the proposed method.

General Machine Learning · Transfer, Multitask and Meta-learning

Yavuz Yarici, Ghassan AlRegib

Deploying multimodal models in real-world scenarios requires generalization to new environments where recording conditions differ from training, a challenge known as multimodal domain generalization (MMDG). Standard architectures employ separate encoders for each modality and a fusion module, training the system end-to-end by optimizing on the fused features. In this paper, we identify that such joint optimization causes encoders to exploit cross-modal co-occurrences, statistical relationships between modalities that arise from source-specific recording conditions, rather than learning domain-invariant features. We term this failure mode Fusion Overfitting. To address this, we propose Modality-Entropy Regularization for Domain Generalization (MER-DG), which maximizes the entropy of each encoder's feature distribution to preserve feature diversity. MER-DG is architecture-agnostic and integrates into existing multimodal frameworks as an additive loss term. Extensive experiments on EPIC-Kitchens and HAC benchmarks demonstrate average improvements of ${\sim}5\%$ over standard fusion and ${\sim}2\%$ over state-of-the-art methods.

General Machine Learning · Transfer, Multitask and Meta-learning

Hao Gu, Mao-Lin Luo, Zi-Hao Zhou, Han-Chen Zhang, Min-Ling Zhang, Tong Wei

Parameter-efficient continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge. Most existing approaches treat continual learning as avoiding interference with past updates, rather than considering what properties make the current task-specific update naturally preserve previously acquired knowledge. From a knowledge-decomposition perspective, we observe that low-rank adaptations exhibit highly imbalanced singular value spectra: a few dominant components absorb most of the adaptation energy, thereby (i) more likely to disrupt previously acquired knowledge and (ii) making the update more vulnerable to interference from subsequent tasks. To enable explicit balance among components, we decouple the *magnitude* of the task update from its *directional structure* and formulate it as a constrained optimization problem on a restricted Stiefel manifold. We address this problem using a projected first-order method compatible with standard deep-learning optimizers used in vision-language models. Our method mitigates both backward and forward forgetting, consistently outperforming continual learning baselines. Source code is available in supplementary material.

General Machine Learning · Transfer, Multitask and Meta-learning

Romain Cosentino

We develop a continual learning method for pretrained models that \emph{requires no access to old-task data}, addressing a practical barrier in foundation model adaptation where pretraining distributions are often unavailable. Our key observation is that pretrained networks exhibit substantial \emph{geometric redundancy}, and that this redundancy can be exploited in two complementary ways. First, redundant neurons provide a proxy for dominant pretraining-era feature directions, enabling the construction of approximately protected update subspaces directly from pretrained weights. Second, redundancy offers a natural bias for \emph{where} to place plasticity: by restricting updates to a subset of redundant neurons and constraining the remaining degrees of freedom, we obtain update families with reduced functional drift on the old-data distribution and improved worst-case retention guarantees. These insights lead to \textsc{PLATE} (\textbf{Pla}sticity-\textbf{T}unable \textbf{E}fficient Adapters), a continual learning method requiring no past-task data that provides explicit control over the plasticity-retention trade-off. PLATE parameterizes each layer with a structured low-rank update $\Delta W = B A Q^\top$, where $B$ and $Q$ are computed once from pretrained weights and kept frozen, and only $A$ is trained on the new task.

General Machine Learning · Transfer, Multitask and Meta-learning

Zhenyi Wang, Yixuan Sun, Yue Wang, Zhong Chen, Heng Huang

Continual learning (CL) aims to acquire new knowledge from a non-stationary data stream while retaining performance on previously learned tasks. Memory-based replay methods mitigate catastrophic forgetting by storing and revisiting past samples, but their effectiveness is fundamentally constrained by limited memory capacity, as each stored example represents only a single data instance. In this work, we propose data reassembly for CL, a new paradigm that significantly increases memory efficiency by reassembling composite replay samples from existing training data. Instead of storing raw training examples, we partition the current task training data into elementary patches and dynamically reassemble them into coherent replay instances through an energy-based optimization framework. The proposed objective jointly enforces semantic compatibility with target labels and global consistency among assembled patches. To make this optimization tractable, we derive an efficient variational inference algorithm that constructs a compact yet diverse set of reassembled samples for replay. Extensive theoretical analysis and experiments across multiple CL benchmarks demonstrate that data reassembly consistently outperforms existing memory-based approaches, achieving stronger retention of past knowledge while maintaining competitive computational efficiency.

General Machine Learning · Transfer, Multitask and Meta-learning

Binh-Nguyen Nguyen, Khang Tran, Hai Phan, Issa Khalil

Many organizations lack computational resources to fine-tune large language models (LLMs) on private (unshareable) data for better utility, while fine-tuning tiny language models (TinyLMs) alone performs poorly. To address this bottleneck, we propose a data-free knowledge distillation framework that generates LLM update vectors based on TinyLMs fine-tuned on private data. An update vector is a vector of parameter changes from an initial model to its fine-tuned version on a dataset, capturing the effect of cumulative gradient steps during fine-tuning. The key idea of our framework is a novel **Gradient Transformer** that transforms TinyLM's update vectors into LLM's update vectors. As derived from shadow datasets, $\texttt{Grad-Transformer}$ captures the correlation between TinyLM and LLM update vectors, enabling third-party providers to generate LLM update vectors given the organization's TinyLM update vectors without accessing the organization's private data. The framework supports multi-organization collaboration to jointly update LLMs, improving performance and cost-efficiency. Extensive experiments across language modeling and reasoning tasks show that $\texttt{Grad-Transformer}$ remarkably outperforms state-of-the-art knowledge distillation baselines, even under strict differential privacy protection.

General Machine Learning · Transfer, Multitask and Meta-learning

Thomas Sommariva, Francesca Morandi, Simone Calderara, Angelo Porrello

Task vector composition has emerged as a promising paradigm for editing pre-trained models, enabling model merging via addition and task removal via subtraction. Fine-tuning in the tangent space of a pre-trained model (*linearized fine-tuning*) has proven particularly effective in this setting, as it yields task vectors that are naturally disentangled and less prone to interference. However, linearized models suffer from reduced expressivity during training and increased computational cost at inference time, limiting their practical applicability. In this work, we propose to bridge linearized and standard non-linear fine-tuning through knowledge distillation. Specifically, we distill hidden representations from a linearized, curvature-regularized teacher into a non-linear student trained with conventional fine-tuning. By doing so, the goal is to transfer the disentanglement properties of the linearized regime, biasing optimization toward solutions that are composable by design. We show that the resulting task vectors can be composed using naïve Task Arithmetic, achieving strong results across vision and language benchmarks without incurring any inference-time overhead.

General Machine Learning · Transfer, Multitask and Meta-learning

Shaoran Lv, Xinyao Li, Jingjing Li

Test-time adaptation (TTA) adapts pretrained models to test data on-the-fly. Current TTA methods have focused on what to adapt: lightweight domain-aware components (prompts, normalization statistics) updated with consistency-aware self-supervised losses. This work investigates the more fundamental yet underexplored optimization process, providing insights and guidelines on how to appropriately update models for TTA. By analyzing the optimization error during TTA, we identify a pivotal stability-plasticity trade-off: the model should adapt to novel distributions while retaining learned knowledge, which motivates our design of a CONfidence-and-Gradient-Aware scheduler (CONGA) to constrain model learning rate (LR) within an adaptive exploration interval. For each iteration, the lower bound encourages model exploration on informative confident samples, while the upper bound prevents aggressive overfitting to noisy optimization gradients. Based on our theoretical findings, an adaptation-progress-conditioned cosine decay function decides the specific LR within the interval. As an LR scheduler, CONGA is naturally applicable on existing TTA methods as a plug-in module, introducing little computation overheads. Extensive experiments and analysis demonstrate the superiority and validness of CONGA.

General Machine Learning · Transfer, Multitask and Meta-learning

Wenju Sun, Qingyong Li, Tiancheng Li, Yangliao Geng, Albert Boyang Li

Model merging offers an efficient solution for integrating task-specific knowledge from multiple fine-tuned models. Most existing approaches focus on manipulating the difference vectors between fine-tuned and pre-trained weights, often overlooking the generalization capabilities inherent in the pretrained parameters themselves. In this work, we revisit the role of pretrained weights in model merging and investigate their efficacy from a subspace perspective. We find that the components of pretrained weights residing in the core subspace—defined by the dominant singular vectors—are essential for maintaining generalization across diverse tasks. Specifically, we present empirical evidence that pretrained weights are nearly first-order stationary and exhibit predominantly non-negative curvature within this core subspace with respect to multi-task loss landscapes, indicating near-optimality. These findings suggest that task-specific adaptations should be injected primarily into the orthogonal complement of the core subspace, thereby preserving the generalization properties of the pretrained model. Extensive experiments on vision and vision-language tasks show that this subspace-aware strategy consistently yields improvements over state-of-the-art training-free merging methods, including Task Arithmetic, LOT Merging, ISO, and TSV.

General Machine Learning · Transfer, Multitask and Meta-learning

Ziang Song, Zhou Zhidan, Zijun Zhang

Domain generalization aims to improve model performance on unseen, out-of-distribution (OOD) domains, yet existing methods often overlook the crucial aspect of uncertainty quantification in their predictions. While ensemble learning combined with knowledge distillation offers a promising avenue, naively combining these techniques is non-trivial and remains largely unexplored in the context of domain generalization. In this work, we systematically investigate different ensemble and distillation strategies for domain generalization tasks and design a tailored data allocation scheme. This approach trains teacher models on distinct subsets of domains and performs distillation on complementary (unseen) subsets, thereby fostering model diversity and training efficiency. Moreover, our theoretical analysis demonstrates that distilling from teachers on unseen domains effectively filters out domain-specific spurious correlations. To address the accuracy degradation often observed with standard uncertainty distillation, we further develop a novel technique that decouples uncertainty distillation from the standard distillation process, enabling accurate uncertainty estimation without compromising model accuracy. Our proposed method, *Domain-aware Ensemble Distillation* (DomED), is extensively evaluated against state-of-the-art domain generalization and ensemble distillation techniques across multiple benchmarks, achieving competitive accuracies and substantially improved uncertainty estimates.

General Machine Learning · Transfer, Multitask and Meta-learning

Hongyou Zhou, Zihan Ye, Ling Shao, Marc Toussaint

Despite strong zero-shot performance, SAM is unreliable under domain shift due to Mask-level Confidence Confusion (MCC), where a single IoU-based mask score fails to reflect pixel-wise reliability near boundaries. Motivated by the contrast between texture-biased shortcuts in neural networks and shape-centric processing in human vision, we model out-of-domain variation as appearance shifts and non-rigid deformations that jointly stress calibration. We propose Segment Anything with Robust Uncertainty-Accuracy Correlation (RUAC) for robust pixel-wise uncertainty estimation under appearance and deformation shifts. RUAC adds a lightweight uncertainty head, trains it with a collaborative style-deformation attack that jointly perturbs texture and geometry, and applies Uncertainty-Error Alignment to ensure uncertainty consistently highlights erroneous pixels even under adversarial perturbations. Across 23 zero-shot domains, RUAC improves segmentation quality and yields more faithful uncertainty with stronger uncertainty-accuracy correlation.

General Machine Learning · Transfer, Multitask and Meta-learning

Md Abdullah Al Forhad, Yuansheng Zhu Zhu., Abhinab Acharya, Xumin Liu, Qi Yu, Weishi Shi

The Mixture of Experts (MoE) framework is widely used in continual learning to mitigate catastrophic forgetting. MoEs typically combine a small inter-task shared parameter space with largely independent expert parameters. However, as the number of tasks increases, the shared space becomes a bottleneck, reintroducing forgetting, while fully independent experts require explicit task ID predictors (e.g., routers), adding complexity. In this work, we eliminate the inter-task shared parameter space and the need for a task ID predictor by enabling expert communication and allowing knowledge to be shared dynamically, akin to human collaboration. We bridge the inter-expert knowledge sharing by leveraging the open-set learning capabilities of a multimodal foundation model (e.g., CLIP), thereby providing “expert priors” that bolster each expert’s task-specific representations. Guided by these priors, experts learn calibrated inter-task posteriors. Additionally, Multivariate Gaussians over the learned posteriors promote complementary specialization among experts. We propose new evaluation benchmarks that simulate realistic continual learning scenarios, and our prior-conditioned strategy consistently outperforms existing methods across diverse settings without relying on reference datasets or replay memory.

General Machine Learning · Transfer, Multitask and Meta-learning

Ali Vahedifar, Abhisek Ray, Qi Zhang

Continual learning enables neural networks to learn tasks sequentially without forgetting previously acquired knowledge. However, catastrophic forgetting, where performance on earlier tasks degrades sharply when learning new ones, remains a fundamental challenge. We address this problem with Shapley Neuron Valuation (SNV), a principled framework grounded in cooperative game theory that quantifies Neuron importance in continual learning. By selectively freezing important Neurons while keeping others plastic, SNV enables memory-free continual learning without architectural expansion. Extensive experiments show that SNV delivers substantial gains over memory-free baselines, achieving +19.50% accuracy on CIFAR-100 and +17.20% on TinyImageNet in the Class-IL setting. In Task-IL scenarios, SNV consistently surpasses existing memory-free approaches by large margins, reaching up to 9.08% higher accuracy on CIFAR-100 compared to the second-best memory-free method, while remaining competitive in comparison with memory-based methods.

General Machine Learning · Transfer, Multitask and Meta-learning

Zhiyuan Yu, Enneng Yang, Hao Jiang, Guojie Zhu, Feihong He, Peng Wang, Li Shen

Upcycling, a strategy that initializes Mixture-of-Experts (MoE) by replicating pre-trained feed-forward or MoE networks to expand model capacity, has become a popular method in continual learning due to its effectiveness in mitigating catastrophic forgetting. However, existing paradigms rely on indiscriminate expansion prioritize performance at the cost of extreme inefficiency, introducing parameter redundancy without exploiting the structural heterogeneity essential for counteracting forgetting with architectural economy. To address this, we investigate the determinants of forgetting in training dynamics using Centered Kernel Alignment (CKA) and loss landscape flatness to analyze the behavior of pre- and post-expansion MoE layers, uncovering instability in deep-layer representations and heterogeneous expert sensitivity to new tasks, thereby demonstrating the potential of selective upcycling to eliminate redundancy. Consequently, we propose a dynamic bilevel optimization framework to guide adaptive upcycling, featuring an outer loop employing a Gumbel-Softmax differentiable mask to perform Neural Architecture Search (NAS) for adaptive growth, while an inner loop optimizes weight updates via task objectives and CKA-regularized replay. Experiments on TRACE benchmark demonstrate that our proposed method achieves better average accuracy with 80\% forgetting reduction, while effectively eliminating 60\% of redundant parameter expansion that standard upcycling would introduce.

General Machine Learning · Transfer, Multitask and Meta-learning

Selim Furkan Tekin, Gaowen Liu, Ramana Kompella, Ling Liu

The advancement of LLMs and their accessibility have triggered renewed interest in multi-agent reinforcement learning as robust and adaptive frameworks for dynamically changing environments. This paper introduces RL-Focal, a two-stage RL agent framework that routes and ensembles LLMs. First, we develop the Decider RL-agent, which learns to dynamically select an ensemble of small size ($m_i$) among $N$ LLMs ($m_i \ll N$) for incoming queries from a user-defined downstream task $i$, by maximizing both error-diversity and reasoning-performance of the selected ensemble through iterative updates of task-adaptive rewards and policy. Second, to enable effective fusion of dynamically selected LLMs, we develop the stage-2 Fusion RL-agent, which learns to resolve reasoning conflicts from different LLMs and dynamically adapt to different ensemble teams composed by the Decider Agent for different downstream tasks. {\em Third}, we introduce the focal diversity metric to better model the error correlations among multiple LLMs further improving the generalization performance of the Decider Agent, which actively prunes the ensemble combinations. By focal diversity, we enhance performance across tasks by effectively promoting reward-aware and policy-adaptive ensemble selection and inference fusion. Extensive evaluations on five benchmarks show that RL-Focal achieves the performance improvement of 8.48\% with an ensemble of small size compared to the best individual LLM in a pool and offers stronger robustness. Code is available at \url{https://anonymous.4open.science/r/rl-focal-8DCF/}