Federated Learning (FL) often suffers from a trade-off between global model performance and client-level fairness due to data heterogeneity, which often leads to inconsistent performance of the globally trained models, resulting in unfair outcomes among users. Existing fair FL algorithms face a trade-off: they either sacrifice global model performance to promote fairness or fall short of achieving optimal fairness. In this paper, we propose a novel framework that bridge this trade-off by integrating information-theoretic principles with model alignment. Specifically, we leverage the Maximum Entropy Principle to derive an analytic, closed-form solution for fair aggregation weights, ensuring significant fairness enhancements with minimal computational overhead. To maintain the global model performance, we further employ a step-wise model alignment strategy that synchronizes gradient directions across heterogeneous clients, effectively mitigating the drift induced by local updates. Theoretical analysis proves that our method guarantees convergence even in non-convex settings. Importantly, we push the theoretical frontier of federated fairness by extending performance variance analysis to generalized regression, providing broader guarantees. Extensive experiments on five datasets demonstrate that our approach consistently outperforms state-of-the-art methods, achieving superior fairness without sacrificing global accuracy.
论文检索
输入标题、作者或关键词，从 13,033 篇学术成果中精准定位
Deep Learning · Foundation Models
Sparse Mixture-of-Experts architectures are essential for scaling model capacity efficiently, yet the standard Top-$k$ routing imposes a rigid sparsity pattern that ignores the intrinsic variance in token difficulty and layer-specific computational needs. While Top-$p$ routing offers a flexible alternative, we demonstrate that existing naive Top-$p$ implementations with fixed global probability thresholds provide only marginal gains over Top-$k$, suffer from hyperparameter sensitivity, and result in uncontrolled computational costs. In this paper, we propose $\texttt{DTop-}p$, a sparsity-controllable dynamic routing mechanism. To overcome the non-differentiability of the MoE sparsity level - the Top-$p$ threshold, we utilize a Proportional-Integral controller that dynamically learns the Top-$p$ probability threshold to align the running sparsity with a user-defined budget. Furthermore, we introduce dynamic routing normalization to adaptively rescale logits, enabling distinct expert selection patterns across layers under a global sparsity constraint. Extensive experiments on Large Language Models and Diffusion Transformers demonstrate that $\texttt{DTop-}p$ consistently outperforms both Top-$k$ and fixed Top-$p$ baselines while matching the average FLOPs of Top-$k$ MoE. Our analysis confirms that $\texttt{DTop-}p$ exhibits strong scaling properties across expert granularity, total expert capacity, model size, and dataset size, offering a robust and efficient MoE framework for foundation model pre-training.
Deep Learning · Large Language Models
Prompt optimization is a key way to steer large language models when fine-tuning is impractical. However, instruction optimization (IO) and in-context learning (ICL) demonstration selection are often optimized separately and combined post hoc, implicitly assuming that a "best'' instruction and a "best" demonstration set compose well. In practice, their interactions are strong, making such decoupled pipelines brittle. We propose SMILE, an efficient method that *jointly* selects instructions and demonstrations. Our key observation is that the ICL performance exhibits consistent diminishing returns across diverse instructions. Leveraging this structure, SMILE learns an instruction-conditioned surrogate aligned with LLM feedback and instantiates it as an Extended Deep Submodular Function that captures sample--sample coverage, sample--query relevance, and sample--instruction compatibility. SMILE then performs greedy, query-adaptive selection of the instruction--demonstration pair. Experiments on six datasets and multiple LLM backbones show that SMILE consistently outperforms IO-only, ICL-only, and existing joint baselines, supporting a context engineering view of prompting: jointly optimizing interacting components rather than tuning them in isolation.
Applications · Computer Vision
While Diffusion Transformers (DiTs) have revolutionized high-fidelity image synthesis, the prohibitive computational costs of training at ultra-high resolutions necessitate robust inference-time extrapolation. Existing extrapolation methods typically operate under a *scale-agnostic* assumption, treating the denoising dynamics identically across resolutions. In this work, we identify a critical oversight in this paradigm: the spectral evolution of the diffusion process, transitioning from low-frequency structural construction to high-frequency texture refinement, is inherently scale-dependent. Consequently, applying a uniform strategy across scales causes a spectral misalignment, manifesting as *structural collapse* or *textural degradation*. To bridge this gap, we introduce **SigMa ($\sigma$)**, a training-free framework that utilizes Sigmoid Modulation for *scale-adaptive* calibration of the extrapolation process. SigMa orchestrates the spectral evolution via a parameterized schedule with two core mechanisms: *Decoupled Geometric Center Alignment*, which synchronizes the transition timing to secure global structure, and *Iso-Variance Rate Adaptation*, which scales the transition velocity to ensure a smooth feature handover. Extensive experiments demonstrate that SigMa effectively rectifies spectral deviations, enabling training-free extrapolation up to 16 megapixels and achieving state-of-the-art performance on standard benchmarks.
Applications · Language, Speech and Dialog
Generative AI agents equate *understanding* with resolving explicit queries, an assumption that confines interaction to what users can articulate. This assumption breaks down when users themselves lack awareness of what is missing, risky, or worth considering. In such conditions, proactivity is not merely an efficiency enhancement, but an epistemic necessity. We refer to this condition as *epistemic incompleteness*: where progress depends on engaging with unknown *unknowns* for effective partnership. Existing approaches to proactivity remain narrowly anticipatory, extrapolating from past behavior and presuming that goals are already well defined, thereby failing to support users meaningfully. However, surfacing possibilities beyond a user’s current awareness is not inherently beneficial. Unconstrained proactive interventions can misdirect attention, overwhelm users, or introduce harm. Proactive agents, therefore, require *behavioral grounding*: principled constraints on *when, how*, and to *what extent* an agent should intervene. We advance the position that **generative proactivity must be grounded both epistemically and behaviorally**. Drawing on the *philosophy of ignorance* and *research on proactive behavior*, we argue that these theories offer critical guidance for designing agents that can engage responsibly and foster meaningful partnerships.
Deep Learning · Everything Else
Transformer-based autoregressive models excel in data generation but are inherently constrained by their reliance on discretized tokens, which limits their ability to represent continuous values with high precision. We analyze the scalability limitations of existing discretization-based approaches for generating hybrid discrete-continuous sequences, particularly in high-precision domains such as logos, layouts, and semiconductor circuit designs, where precision loss potentially leads to visual artifacts, aesthetic degradation, and even functional failure. To address the challenge, we propose a novel unified framework that jointly models discrete and continuous values for variable-length sequences. Our approach employs a hybrid approach that combines categorical prediction for discrete values with diffusion-based modeling for continuous values, incorporating two key technical components: an end-of-sequence (EOS) logit adjustment mechanism that uses an MLP to dynamically adjust EOS token logits based on sequence context, and a length regularization term integrated into the loss function. Additionally, we present ContLayNet, a large-scale benchmark comprising 334K high-precision semiconductor layout samples with specialized evaluation metrics that capture functional correctness, where precision errors significantly impact performance. Experiments on semiconductor layouts (ContLayNet), graphic layouts, and SVGs demonstrate that our approach achieves higher-fidelity hybrid vector representations than discretization-based and fixed-schema baselines, while scaling to high-precision generation across multiple domains.
Applications · Computer Vision
Multi-modal large language models (MLLMs) depend on in-context learning (ICL) for rapid task adaptation, but their scalability is severely limited by finite context windows and the growing cost of key–value (KV) caches in long multi-modal sequences. Existing memory compression approaches typically rely on rigid token removal or sample-dependent importance estimation, which introduces bias, disrupts semantic structure—particularly for visual representations—and yields static memories that cannot adapt to new queries. We introduce TASM (Task-Aware Structured Memory), a training-free framework that addresses these limitations through task-aware, structure-preserving, and dynamically accessible memory construction. TASM employs Task-Vector Guided Compression to replace sample-specific signals with a task-level direction that captures shared relevance across demonstrations. To preserve the underlying information manifold, it further applies Semantics-Aware Token Merging, formulating compression as a Bipartite Graph Matching problem that merges tokens without destructive pruning. Finally, TASM organizes compressed representations into a multi-resolution hierarchy consisting of a compact Core Memory and a Latent Bank, enabling Query-Adaptive Dynamic Activation and Dynamic Retrieval at inference time. Empirical evaluations show that TASM sustains strong multi-modal ICL performance under high compression ratios, demonstrating an effective balance between efficiency, adaptability, and semantic fidelity.
Optimization · Large Scale, Parallel and Distributed
We analyze the Accelerated Noisy Power Method, an algorithm for Principal Component Analysis in the setting where only inexact matrix-vector products are available, which can arise for instance in decentralized PCA. While previous works have established that acceleration can improve convergence rates compared to the standard Noisy Power Method, these guarantees require overly restrictive upper bounds on the magnitude of the perturbations, limiting their practical applicability. We provide an improved analysis of this algorithm, which preserves the accelerated convergence rate under much milder conditions on the perturbations. We show that our new analysis is worst-case optimal, in the sense that the convergence rate cannot be improved, and that the noise conditions we derive cannot be relaxed without sacrificing convergence guarantees. We demonstrate the practical relevance of our results by deriving an accelerated algorithm for decentralized PCA, which has similar communication costs to non-accelerated methods. To our knowledge, this is the first decentralized algorithm for PCA with provably accelerated convergence.
Optimization · Zero-order and Black-box Optimization
In batch multi-objective Bayesian optimization (MOBO), it is often desirable to identify the whole Pareto optimal set, especially when considering the complicated interplay between different design criteria and constraints. This poses unique challenges in acquiring batches of both high quality and diversity to cover the Pareto front. We propose a novel acquisition strategy, Probability of Matching (POM), which evaluates both batch candidate quality and diversity by explicitly capturing the likelihood that all batch points are Pareto optimal, and the probability that they collectively cover the full Pareto set. To estimate the coverage probability and promote diversity, we incorporate non-replacement sampling principles, resulting in our new POM-guided batch MOBO method. Across synthetic benchmarks and real-world tasks, our method consistently outperforms state-of-the-art baselines on standard MOBO metrics as well as a new design-space coverage metric, Expected Minimum Distance (EMD), with comparable computational efficiency.
Deep Learning · Theory
Sparse Mixture-of-Experts (SMoE) architectures are now widely deployed in state-of-the-art language and vision models, where conditional routing allows scaling to very large networks. However, this very Top-$k$ expert selection that enables conditional routing also renders the SMoE map inherently discontinuous. In the vicinity of these discontinuity surfaces, even inputs that are arbitrarily close may activate substantially different sets of experts resulting in significantly different outputs. In this work we give a rigorous geometric and stochastic analysis of these discontinuities. We first classify them by order, determined by the number of tied experts at a switching event. Using measure-theoretic slicing arguments, we establish asymptotic volume estimates for the thickened discontinuity surfaces, showing that lower-order discontinuity sets dominate, whereas higher-order ones occupy a vanishingly small relative volume. Next, modeling random perturbations in the input space via a diffusion process, we prove that the path eventually encounter a discontinuity, and moreover that the first hit almost surely occurs on an order-1 discontinuity with explicit finite-time probability bounds. We further derive occupation-time bounds that quantify the duration the random path spend in the neighborhoods of each discontinuity order. These theoretical results imply that inputs are more likely to lie near lower order discontinuities. Motivated by this insight, we propose a simple smoothing mechanism that can be directly applied to existing SMoEs, softly incorporating experts near discontinuities; our analysis guarantees that the added computational overhead remains small while providing localized smoothing near discontinuities, and experiments across language and vision tasks show that smoothing not only enforces continuity of the SMoE map but also enhances empirical performance.
General Machine Learning · Clustering
We study the problem of dynamic facility location with non-uniform costs. Facility location is a central problem in unsupervised learning and in recent years the dynamic version of the problem has been extensively studied. In this paper, we study the setting where clients are added and deleted in real-time and one is interested in maintaining efficiently a stable and high-quality solution. Interestingly, we are able to show that on High Dimensional Euclidean metrics it is possible to obtain efficient algorithms for this problem. More formally, we obtain a randomized algorithm for dynamic facility location in $d$-dimensional Euclidean spaces with $\gamma$ approximation ratio, $O(\log m)$ amortized recourse and $\text{poly}(d) \cdot (m+n)^{O(1/\gamma)}$ amortized update time, for every sufficiently large constant $\gamma \geq 1$. Our result is the first efficient dynamic algorithm for the \emph{non-uniform} dynamic facility location problem in high-dimensional Euclidean spaces. It also provides a stronger recourse bound than the existing solutions.
Deep Learning · Large Language Models
Standard mixed-precision training of neural networks requires many bytes of accelerator memory for each model parameter. These bytes reflect not just the parameter itself, but also its gradient and one or more optimizer state variables. With each of these values typically requiring 4 bytes, training even a 7 billion parameter model can be impractical for researchers with less than 100GB of accelerator memory. We introduce FlashOptim, a suite of optimizations that reduces per-parameter memory by over 50% while preserving model quality and API compatibility. Our approach introduces two key techniques. First, we improve master weight splitting by finding and exploiting a tight bound on its quantization error. Second, we design companding functions that greatly reduce the error in 8-bit optimizer state quantization. Together with 16-bit gradients, these techniques reduce AdamW memory from 16 bytes to 7 bytes per parameter, or 5 bytes with gradient release. They also cut model checkpoint sizes by more than half. Experiments with FlashOptim applied to SGD, AdamW, and Lion show no measurable quality degradation on any task from a collection of standard vision and language benchmarks, including Llama-3.1-8B finetuning.
Deep Learning · Large Language Models
Layer dropout (a.k.a.\ stochastic depth) has been shown to enable faster training, higher accuracy, and robustness to zero-shot layer pruning in both language and vision transformers. However, as models and datasets have scaled, dropout---particularly layer dropout---has largely disappeared from LLM pre-training recipes. While some prior work has reported that dropout can degrade accuracy, no comprehensive study has quantified, let alone mitigated, this effect. In this study, we show that layer dropout **should** be used in state-of-the-art LLM training, establishing best practices and scaling analysis for both training and post-training benefits. Concretely, with optimal layer distribution, time schedule, and optimizer hyperparameters, a 3.9B-parameter LLM can achieve \textbf{lower validation loss} while saving 20\% of training FLOPs. Moreover, layer dropout enables significant post-training optimizations, such as early exit, intermediate-layer skipping, and self-speculative decoding, yielding up to 1.7x inference speedup with negligible accuracy loss. Across more than 2400 training experiments, spanning models from 271M to 3.9B parameters and datasets up to 116B tokens, we demonstrate that these findings extend reliably to large-scale training regimes.
Video editing has evolved toward In-Context Learning (ICL) paradigms, yet the resulting quadratic attention costs create a critical computational bottleneck. In this work, we propose **I**n-context **S**parse **A**ttention (**ISA**), the first experimentally lossless sparse framework tailored for ICL video editing. Our design is grounded in two key insights: __**first**__, context tokens exhibit significantly lower saliency than source tokens; __**second**__, we theoretically prove and empirically validate that Query sharpness correlates with approximation error. Motivated by these findings, ISA implements an efficient pre-selection strategy to prune redundant context, followed by a dynamic query grouping mechanism that routes high-error queries to full attention and low-error ones to a computationally efficient 0-th order Taylor sparse attention. Furthermore, we construct a scalable pipeline to curate a 1M-sample dataset and train __**LIVEditor**__, a novel lightning video editing model via ISA. Extensive experiments demonstrate that LIVEditor achieves a ~60% reduction in latency while surpassing state-of-the-art methods across EditVerseBench, IVE-Bench, and VIE-Bench, delivering experimentally lossless acceleration without compromising visual fidelity.
Theory · Online Learning and Bandits
We study the problem of neural logistic bandits, where the main task is to learn an unknown reward function within a logistic link function using a neural network. Existing approaches either exhibit unfavorable dependencies on $\kappa$, where $1/\kappa$ represents the minimum variance of reward distributions, or suffer from direct dependence on the feature dimension $d$, which can be huge in neural network–based settings. In this work, we introduce a novel Bernstein-type inequality for self-normalized vector-valued martingales that is designed to bypass a direct dependence on the ambient dimension. This lets us deduce a regret upper bound that grows with the effective dimension $\widetilde{d}$, not the feature dimension, while keeping a minimal dependence on $\kappa$. Based on the concentration inequality, we propose two algorithms, NeuralLog-UCB-1 and NeuralLog-UCB-2, that guarantee regret upper bounds of order $\widetilde{O}(\widetilde{d}\sqrt{\kappa T})$ and $\widetilde{O}(\widetilde{d}\sqrt{T/\kappa})$, respectively, improving on the existing results. Lastly, we report numerical results on both synthetic and real datasets to validate our theoretical findings.
Applications · Computer Vision
Recovering surface normals from single-view polarization images in complex media remains challenging. This paper focuses on ice as a representative complex medium, where intricate light–matter interactions lead to a nonlinear mapping between polarization observations and surface normals. To address this, a structure-aware polarization prior based on autocorrelation functions is proposed to capture the local spatial consistency of AoLP. Building on this, a dual-branch network (IceSfP) is designed to integrate raw polarization features with priors via cross-modal attention and multi-scale feature fusion, enabling accurate surface normal estimation under complex media conditions. To evaluate the method, the first real-world ice SfP dataset is constructed. Experimental results show that the method outperforms existing approaches across all metrics, achieving a MAE of 16.01$^\circ$, which is 2.74$^\circ$ lower than the second-best method. The framework provides a generalizable solution for high-precision geometric perception in complex media.
Deep Learning · Everything Else
Malware detection remains largely reactive: machine learning models trained on known samples degrade as threats evolve. Understanding evolutionary relationships among malware families can inform proactive defense, but traditional reverse engineering can take months to years to uncover such lineage relationships. We propose MalTree, a framework that applies bioinformatics-inspired phylogenetic techniques (UPGMA and Neighbor-Joining) at scale to model malware evolution automatically using structural, behavioral, and image-based features. We introduce temporal validation using VirusTotal timestamps to assess whether inferred trees reflect actual evolutionary order. MalTree achieves 87% temporal consistency, indicating that inferred evolutionary relationships closely align with real-world emergence timelines. Our analysis shows that some families mutate over 10 times faster than others, suggesting that detection strategies should be tailored to family-specific evolutionary tempos. Case studies, including the Mirai botnet, confirm that inferred relationships from our phylogenetic tree align with documented threat intelligence. Our framework provides a foundation for shifting malware analysis from sample-by-sample classification toward lineage-aware evolutionary modeling.
General Machine Learning · Online Learning, Active Learning and Bandits
We study linear dueling bandits in volatile environments characterized by the simultaneous presence of post-serving contexts, delayed feedback, and adversarial corruption. Feedback is subject to unknown stochastic or adversarial delays and a cumulative corruption budget $\mathcal{C}$. To address these challenges, we propose \term, which integrates a learned approximator that predicts post-serving contexts from pre-serving information. It further employs an adaptive weighting strategy that clips feature vectors to mitigate the impact of corrupted and delayed observations simultaneously. Under standard regularity conditions and a parametric post-serving mapping, we rigorously establish that our algorithm is delay-regime-agnostic, achieving a regret upper bound of $\widetilde{\mathcal{O}}(d(\sqrt{T} + \mathcal{C} + \mathcal{D}))$, where $d$ is the total feature dimension and $\mathcal{D}$ encapsulates the delay complexity. Crucially, our analysis reveals an additive cost structure between corruption and delay, avoiding the multiplicative degradation typical of prior works. We further establish lower bounds that nearly match our upper bounds up to a $\sqrt{d}$ factor for adversarial delays in the absence of post-serving contexts.
Deep Learning · Everything Else
WiFi-based human pose estimation offers privacy-preserving and occlusion-robust sensing, but current Transformer-based approaches suffer from quadratic complexity and lack explicit inductive biases for Channel State Information structure. We propose WiFi-Mamba, the first State Space Model architecture for WiFi-based 3D multi-person pose estimation. Our approach introduces three key contributions: (1) a Dual-Stream Selective State Space Model that processes amplitude and phase through parallel pathways with cross-stream state coupling to respect their distinct physical properties, (2) Selective State Attention for pose query decoding with SSM-derived sequential context, and (3) Persistent SSM Memory for temporal consistency across frames without recurrent memory explosion. Extensive experiments on the Person-in-WiFi 3D dataset, covering both single-person and multi-person scenarios, demonstrate 16-27% MPJPE reduction across varying numbers of persons while using only 4.4% of baseline parameters (2.14M vs. 48.2M), achieving superior efficiency-accuracy trade-offs particularly beneficial for edge deployment in privacy-sensitive continuous monitoring scenarios.
Applications · Language, Speech and Dialog
Speculative Decoding (SD) has emerged as a critical technique for accelerating Large Language Model (LLM) inference. Unlike deterministic system optimizations, SD performance is inherently data-dependent, meaning that diverse and representative workloads are essential for accurately measuring its effectiveness. Existing benchmarks suffer from limited task diversity, inadequate support for throughput-oriented evaluation, and a reliance on high-level implementations that fail to reflect production environments. To address this, we introduce **SPEED-Bench**, a comprehensive suite designed to standardize SD evaluation across diverse semantic domains and realistic serving regimes. SPEED-Bench offers a carefully curated *Qualitative* data split, selected by prioritizing semantic diversity across the data samples. Additionally, it includes a *Throughput* data split, allowing speedup evaluation across a range of concurrencies, from latency-sensitive low-batch settings to throughput-oriented high-load scenarios. By integrating with production engines like vLLM and TensorRT-LLM, SPEED-Bench allows practitioners to analyze system behaviors often masked by other benchmarks. We highlight this by quantifying how synthetic inputs overestimate real-world throughput, identifying batch-size dependent optimal draft lengths and biases in low-diversity data, and analyzing the caveats of vocabulary pruning in state-of-the-art drafters. We release SPEED-Bench to establish a unified evaluation standard for practical comparisons of SD algorithms.