论文检索

Deep Learning · Generative Models and Autoencoders

Zepeng Zhang, Aref Einizade, Jhony H. Giraldo, Olga Fink

Missing data is a common challenge in spatiotemporal systems, arising in applications such as air quality monitoring and urban traffic management. Traditional machine learning approaches, like recurrent and graph neural networks, rely on iterative propagation, which tends to accumulate errors over time and space. Recent diffusion-based methods mitigate error propagation but require iterative sampling and often depend on problem-agnostic Gaussian priors, limiting both efficiency and effectiveness. To address these limitations, we propose GiFlow, a Graph-Informed Flow Matching framework for spatiotemporal imputation. GiFlow replaces the typical Gaussian prior with a graph-informed prior constructed via spatiotemporal filtering of observable signals, which better aligns the source distribution to the target and thereby simplifies the generation trajectory. The flow field is parameterized by a hybrid vector field model that integrates spatial attention, temporal attention, and spatiotemporal propagation, enabling joint modeling of spatial and temporal dependencies. Unlike diffusion models, GiFlow is trained via direct regression and supports deterministic, few-step generation at inference. Extensive experiments on both synthetic and real-world datasets with different missing patterns and missing rates demonstrate that the proposed GiFlow outperforms the state-of-the-art approaches in spatiotemporal imputation.

Deep Learning · Generative Models and Autoencoders

Xianglu Wang, Bangxian Han, Hu Ding

Diffusion models have achieved state-of-the-art performance in image generation tasks. However, training powerful diffusion models remains time-consuming, which limits their practical deployment. In this paper, we revisit the learning dynamics of diffusion models through the lens of *spectral bias*, a phenomenon in which deep neural networks prioritize learning low-frequency modes. Through an empirical analysis of diffusion training, we observe that diffusion models exhibit a **dual** spectral bias. First, over training iterations, they fit low-frequency components earlier than high-frequency details. Second, along the diffusion timesteps, early denoising steps mainly reconstruct coarse low-frequency content, while high-frequency details emerge in later steps. Motivated by this observation, we propose Frequency-aware Patch Selection **(FaPS)**, a general and fast training method for diffusion models that can be applied to both UNet and DiT backbones. Specifically, FaPS introduces a *frequency-aware gating* that adaptively selects image patches based on their frequency information and focuses computation only on the selected patches. Since the selection decisions are discrete and thus non-differentiable, we model the gating as a stochastic policy network and optimize it end-to-end using a policy gradient method. Our experiments demonstrate that FaPS achieves up to $\mathbf{3}\times$ faster training while maintaining comparable or superior generation quality, and improves the performance of diffusion models in limited-data settings.

Deep Learning · Generative Models and Autoencoders

Subham Sekhar Sahoo, Zhihan Yang, Yash Akhauri, Johnna Liu, Deepansha Singh, Zhoujun Cheng, Zhengzhong Liu, Eric Xing, John Thickstun, Arash Vahdat

Diffusion-based language models offer a compelling alternative to autoregressive (AR) models by enabling parallel and controllable generation. Within this family, Masked Diffusion Models (MDMs) currently perform best but still underperform AR models in perplexity and lack key inference-time efficiency features, most notably KV caching. We introduce Eso-LMs, a new family of models that fuses AR and MDM paradigms, smoothly interpolating between their perplexities while overcoming their respective limitations. Unlike prior work, which uses transformers with bidirectional attention as MDM denoisers, we exploit the connection between MDMs and Any-Order autoregressive models and adopt causal attention. This design lets us compute the exact likelihood of MDMs for the first time and, crucially, enables us to introduce KV caching for MDMs while preserving parallel generation for the first time, significantly improving inference efficiency. Combined with an optimized sampling schedule, Eso-LMs achieves a new state of the art on the speed-quality Pareto frontier for unconditional generation. On long contexts, it yields 14−65× faster inference than standard MDMs and 3−4× faster inference than prior semi-autoregressive approaches.

Deep Learning · Generative Models and Autoencoders

Desong Yang, Mang Ye

With recent advancements in large-scale pre-trained text-to-image (T2I) models, training-free image editing methods have demonstrated remarkable success. Typically, these methods involve adding noise to a clean image via an inversion process, followed by separate denoising steps for the reconstruction and editing paths during the forward process. However, since the reconstruction path is approximated using noisy latents from mismatched timesteps, existing methods inevitably suffer from accumulated drift, which fundamentally limits reconstruction fidelity. To address this challenge, we systematically analyze the inversion process within the flow transformer and propose DirectEdit, a simple yet efficient editing method that eliminates the inherent reconstruction error without introducing additional neural function evaluations (NFEs). Unlike most prior works that attempt to rectify the inversion path, DirectEdit focuses on directly aligning the forward paths, enabling precise reconstruction and reliable feature sharing. Furthermore, we introduce a preservation mechanism based on attention feature injection and multi-branch mask-guided noise blending, which effectively balances fidelity and editability. Extensive experiments across diverse scenarios demonstrate that DirectEdit achieves efficient and accurate image editing, delivering superior performance that outperforms state-of-the-art methods. Code will be released.

Deep Learning · Generative Models and Autoencoders

Mashrur M. Morshed, Vishnu Boddeti

The physical world is fundamentally compositional, yet empirical data are often fragmented across decentralized silos that cannot be aggregated due to privacy, legal, or economic constraints. Such scenarios pose a fundamental challenge for generative modeling: learning models that collectively cover the union of these sources while enabling compositional generalization when the factors required for composition are distributed across isolated data sources. We introduce Decentralized Compositional Flow Matching (DCFM), a framework for learning generative models from decentralized private data without exchanging raw samples. DCFM enforces structural constraints that induce conditional independence across the global set of generative factors. As a result, DCFM allows novel combinations to emerge through interactions across peers, even when no single data source contains sufficient information to support composition on its own. Empirically, DCFM substantially outperforms federated learning and mixture-of-experts baselines across conditional image generation, robotic spatial planning and medical attribute co-occurrence modeling.

Deep Learning · Generative Models and Autoencoders

Zixuan Zhang, Hengyu Fu, Zhuoran Yang, Mengdi Wang, Tuo Zhao, Minshuo Chen

Discrete diffusion models have recently emerged as a compelling alternative for language generation, enabling efficient non-autoregressive sampling while achieving strong empirical performance. A key design choice in discrete diffusion---absent in most continuous diffusion formulations---is the forward corruption kernel, with masked/absorbing corruption now dominating practice. Despite this empirical preference, there is limited statistical theory explaining when and why masking should outperform alternative kernels such as uniform replacement. In this paper, we take a step toward closing this gap from a statistical learning perspective. Our analysis establishes generalization bounds and, through an explicit comparison across different forward corruption kernels, reveals a central advantage of masking: it scales with the effective data support rather than the full ambient state space, thereby mitigating the curse of state space cardinality. We further derive structure-aware refinements that capture how concentration and sparsity in real sequential data sharpen the sample complexities. Together, these results offer a principled explanation for the empirical strength of masked diffusion and provide guidance for forward-kernel design in discrete generative modeling.

Deep Learning · Generative Models and Autoencoders

Ying Li, Zefang Wang, Zhaode Wang, Zhiwen Chen, chengfei lv, Huan Wang

Scaling up visual autoregressive models improves generation quality but incurs substantial inference costs. Mixture-of-Experts (MoE) architectures mitigate this issue through sparse activation and have proven effective in large language models. However, training MoE models from scratch remains prohibitively expensive, and dense-to-MoE conversion for visual autoregressive models is still underexplored. To enable *low-cost and high-quality dense-to-MoE conversion*, we propose **Prism-MoE**, an efficient framework for transforming pretrained dense visual autoregressive models into sparse MoE models. Prism-MoE consists of two key components. First, we introduce trajectory-consistent Initialization, which formulates expert initialization as a principled decomposition problem and preserves the generation trajectory of pretrained models. Second, we propose a confidence-adaptive sparse fine-tuning framework that aligns expert specialization with the information density of visual tokens via confidence-aware routing supervision. Experiments show that Prism-MoE achieves dense-to-MoE conversion with less than **10\%** of the standard training budget, while maintaining generation quality comparable to dense baselines with only **37.5\%** active parameters.

Deep Learning · Generative Models and Autoencoders

Ying Li, Zhaode Wang, Zhiwen Chen, chengfei lv, Huan Wang

As larger language models deliver stronger capabilities, their autoregressive inference becomes increasingly expensive. *Speculative decoding* accelerates generation by letting a fast draft propose tokens that the target model verifies in parallel. Yet under sampling ($T>0$), observed speedups consistently lag behind those under greedy decoding, as the classical lossless verification rule tends to over-reject low-risk drafts, leading to lower acceptance rates and limited acceleration. To address this gap, we propose **ARC-Decode** (**A**cceptance with **R**isk **C**ontrol), a training-free method that augments speculative decoding without extra forward passes. ARC-Decode enables \textbf{relaxed} acceptance by identifying drafts whose acceptance preserves the output distribution of the target model, under a risk-controlled criterion based on Jensen--Shannon divergence. It combines confidence-based pre-verification filtering with a risk-bounded acceptance criterion derived from an analytic upper bound on the potential distributional deviation. Integrated into the state-of-the-art EAGLE-3 pipeline, ARC-Decode increases accept length per cycle and reduces verification compute, achieving up to **1.6**$\times$ end-to-end speedup over EAGLE-3 under sampling with negligible quality change across benchmarks.

Deep Learning · Generative Models and Autoencoders

Wenda Chu, Bingliang Zhang, Jiaqi Han, Yizhuo Li, Linjie Yang, Yisong Yue, Qiushan Guo

Autoregressive image modeling relies on visual tokenizers to compress images into compact latent representations. We design an end-to-end training pipeline that jointly optimizes reconstruction and generation, enabling direct supervision from generation results to the tokenizer. This contrasts with prior two-stage approaches that train tokenizers and generative models separately. We further investigate leveraging vision foundation models to improve 1D tokenizers for autoregressive modeling. Our autoregressive generative model achieves strong empirical results, including a state-of-the-art FID score of 1.48 without guidance on ImageNet 256×256 generation.

Deep Learning · Generative Models and Autoencoders

Qianxin Xia, Zhiyong Shu, Wenbo Jiang, Jiawei Du, Jielei Wang, Guoming Lu

Dataset distillation aims to synthesize a compact proxy dataset that is unreadable or non-raw from the original dataset for privacy protection and highly efficient learning. However, previous approaches typically adopt a single-stage distillation paradigm, which suffers from learning specific patterns that overfit on a prior architecture, consequently suppressing the expression of semantics and leading to performance degradation across heterogeneous architectures. To address this issue, we propose a novel dual-stage distillation framework called ${\textbf{DIVER}}$, which leverages the pre-trained diffusion model to dive deeper into $\textbf{DI}$stilled data $\textbf{V}$ia $\textbf{E}$xpressive semantic $\textbf{R}$ecovery, a process of semantic inheritance, guidance, and fusion. Semantic inheritance distills high-level semantic knowledge of abstract distilled images into the latent space to filter out architecture-specific ``noise" and retain the intrinsic semantics. Furthermore, semantic guidance improves the preservation of the original semantics by directing the reverse procedure. Ultimately, \textcolor{red}{semantic fusion is designed to provide semantic guidance only during the concrete phase of the reverse process, preventing semantic ambiguity and artifacts while maintaining the guidance information.} Extensive experiments validate the effectiveness and efficiency of our method in improving classical distillation techniques and significantly improving cross-architecture generalization, requiring processing time comparable to raw DiT on ImageNet (256$\times$256) with only 4 GB of GPU memory usage.

Deep Learning · Generative Models and Autoencoders

Andrei Atanov, Jesse Allardice, Roman Bachmann, Oğuzhan Fatih Kar, R Devon Hjelm, David Griffiths, Peter Fu, Amir Zamir, Afshin Dehghan

Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling, e.g., conditional video generation. Beyond compression, tokenizers define what information is preserved and how it is organized. A _de facto_ standard approach is to represent a video with a spatiotemporal 3D grid of tokens, each corresponding to a local patch in the original signal. This requires a downstream model, e.g., a text-to-video model, to learn to predict all low-level details ``pixel-by-pixel'' irrespective of the video's inherent complexity, resulting in high computational cost during training. We present VideoFlexTok, a tokenizer that represents videos with a _variable-length sequence of tokens structured in a coarse-to-fine manner_, where the first tokens capture abstract information like semantics and motion and later tokens provide fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the tokens count to particular downstream needs and encode videos longer than the 3D grid approach under the same budget. We evaluate VideoFlexTok on class-to-video and text-to-video generative tasks and show that it leads to more efficient training compared to 3D grid tokens, e.g., achieving comparable generation quality (gFVD and ViCLIP Score) with a 10x smaller model (0.4B vs 3.6B). Finally, we demonstrate how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.

Theory · Learning Theory

Futoshi Futami, Masahiro Fujisawa

Despite their remarkable success, a rigorous theoretical understanding of how latent variables (LVs) govern the generalization performance of Variational Autoencoders (VAEs) remains largely elusive. Existing theoretical analyses are confined to supervised learning or models with discrete latent spaces, leaving their role in standard VAEs with continuous LVs poorly understood. This paper establishes the first information-theoretic analysis for VAEs by adapting a theoretical framework from supervised learning---the leave-one-out conditional mutual information framework---to the unsupervised, continuous latent space of these models. Our analysis reveals that their generalization error is bounded solely by the information complexity of the encoder and LVs, independent of the decoder. The versatility of our framework is demonstrated through its extension to both hierarchical VAEs, for which we provide layer-wise bounds, and data generation, where we link our information-theoretic principles to a novel bound on the 2-Wasserstein distance between true and generated distributions.

Deep Learning · Generative Models and Autoencoders

Gabriel Moreira, Manuel Marques, Joao Costeira, Chenyan Xiong

We introduce Spectral Guidance, a framework for controlling diffusion models by leveraging the intrinsic geometry of the generative process. As data is progressively corrupted by noise, only a small number of features remain informative for control. We characterize them as the singular functions of a conditional expectation operator and show that they can be learned via a self-supervised objective. Once recovered, this basis enables the projection of arbitrary guidance signals, such as labels, CLIP embeddings, or masks, directly onto the sampling trajectory. This approach allows for stable, high-fidelity control without retraining or denoiser backpropagation during sampling. Empirically, we improve conditional accuracy on CIFAR-10 by 37 percentage points over the strongest training-free baseline while offering $4\times$ faster sampling. Moreover, the same representations that support label and CLIP guidance also enable spatial control, such as mask-based guidance, without auxiliary models. Finally, our framework reveals a phase transition in the generative process, pinpointing the optimal time window for effective guidance.

Deep Learning · Generative Models and Autoencoders

Zhiqi Li, Yuchen Sun, Duowen Chen, Jinjin He, Bo Zhu

We propose \emph{Euler Mean Flows (EMF)}, a flow-based generative framework for one-step and few-step generation that enforces long-range trajectory consistency with minimal sampling cost. The key idea of EMF is to replace the trajectory consistency constraint, which is difficult to supervise and optimize over long time scales, with a principled linear surrogate that enables direct data supervision for long-horizon flow-map compositions. We derive this approximation from the semigroup formulation of flow-based models and show that, under mild regularity assumptions, it faithfully approximates the original consistency objective while being substantially easier to optimize. This formulation leads to a unified, JVP-free training framework that supports both $u$-prediction and $x_1$-prediction variants, avoiding explicit Jacobian computations and significantly reducing memory and computational overhead. Experiments on image synthesis, particle-based geometry generation, and functional generation demonstrate improved optimization stability and sample quality under fixed sampling budgets, together with approximately $50\%$ reductions in training time and memory consumption compared to existing one-step methods for image generation.

Deep Learning · Generative Models and Autoencoders

Hongxu CHEN, Hongxiang Li, Zhen Wang, Long Chen

Flow Matching (FM) models have emerged as a leading paradigm for high-fidelity synthesis. However, their reliance on iterative Ordinary Differential Equation (ODE) solving creates a significant latency bottleneck. Existing solutions face a dichotomy: training-free solvers suffer from significant performance degradation at low Neural Function Evaluations (NFEs), while training-based methods incur prohibitive training costs and lack plug-and-play versatility. To bridge this gap, we propose the Bi-Anchor Interpolation Solver (BA-solver). BA-solver retains the versatility of standard training-free solvers while achieving significant acceleration by introducing a lightweight SideNet (1-2% backbone size) alongside the frozen backbone. Specifically, our method is founded on two synergistic components: 1) Bidirectional Temporal Perception, where the SideNet learns to approximate both future and historical velocities without retraining the heavy backbone; and 2) Bi-Anchor Velocity Integration, which utilizes the SideNet with two anchor velocities to efficiently approximate intermediate velocities for batched high-order integration. By utilizing the backbone to establish high-precision “anchors” and the SideNet to densify the trajectory, BA-solver enables large step sizes with minimized error. Empirical results on ImageNet-256 demonstrate that BA-solver achieves generation quality comparable to 100+ NFEs Euler solver in just 10 NFEs and maintains high fidelity in as few as 5 NFEs, incurring negligible training costs. Furthermore, BA-solver ensures seamless integration with existing generative pipelines, facilitating downstream tasks such as image editing.

Deep Learning · Generative Models and Autoencoders

Gwangho Kim, Sungyoon Lee

Diffusion models can unintentionally memorize training samples, raising concerns about privacy and copyright. While recent methods can detect memorization, they often rely on global or model-specific signals and provide limited insight into where memorization appears within a generated image. We provide a geometric characterization of local memorization as a coordinate-wise variance collapse. However, such collapse can also arise from intrinsic data constraints rather than overfitting. To isolate overfitting-driven memorization, we propose curvature-difference methods that subtract the curvature of an underfitted baseline, either the unconditional model or a less-trained version of itself. We further derive a score-difference surrogate that provides a geometric explanation for the widely used score-difference-based detection metric. Experiments on Stable Diffusion, evaluated against ground-truth memorization masks, show that our method outperforms the prior attention-based localization method.

Deep Learning · Generative Models and Autoencoders

Thomas Sesmat, Gabriel Meseguer-Brocal, Geoffroy Peeters

Understanding what generative models retain from training data remains challenging, with implications for copyright and privacy. This question becomes particularly relevant as Rectified Flows power increasingly deployed systems. We analyze the interpolation path $X_\lambda = (1-\lambda)X_0 + \lambda X_1$ that defines Rectified Flow training. We show that train-test distinguishability follows a bell-shaped curve over $\lambda$, with a maximum whose location we derive in closed form under Gaussian assumptions. This signal accumulates during training while validation metrics remain stable. We validate these predictions on both audio and images, and show that the bell-shaped structure is universal while the peak prediction holds when our assumptions are satisfied. As a proof of concept, we implement a membership inference attack achieving 0.91 AUC by exploiting this $\lambda$-resolved structure.

Deep Learning · Generative Models and Autoencoders

Chenyang Xu, Dezhen Wang, Lin Chen, Kepeng Lin, Hao Wang

Accelerating diffusion models via feature caching has evolved from static reuse to polynomial extrapolation, yet current "cache-then-forecast" strategies remain limited by rigid, hand-crafted approximation families (e.g., Taylor or Hermite bases) that often misalign with the complex, layer-specific non-stationarity of generative feature dynamics. This paper introduces EigenCache, a theoretically grounded framework that re-frames acceleration as a problem of covariance learning and experimental design. By modeling feature trajectories as time-indexed stochastic processes governed by learnable temporal kernels, we demonstrate that the statistically optimal feature predictor (Minimum Mean Squared Error) is the Gaussian Process posterior mean (Kriging), which strictly generalizes and outperforms previous fixed-basis expansions. Crucially, this probabilistic formulation couples prediction with uncertainty quantification via a closed-form variance certificate. Leveraging this, we derive an information-theoretic scheduling algorithm that selects computation anchors by maximizing the log-determinant of the posterior covariance—a submodular objective that admits a provably near-optimal greedy solution. EigenCache thus provides a unified, training-free foundation for efficient inference, offering not only superior reconstruction accuracy but also a rigorous mechanism for robust, uncertainty-aware compute allocation.

Deep Learning · Generative Models and Autoencoders

Cong Geng, Xue Han, Ye Yuan, Qiang Hu, Xin Huang, Ruiqiao Bai, JUNLAN FENG

Energy-based models (EBMs) provide a flexible framework for generative models with strong distribution modeling capabilities. Nevertheless, their broader adoption has been limited by the difficulty of stable and efficient training. In this paper, we propose a unified and efficient latent-guided cooperative EBM that leverages informative target latent variables to guide the joint energy in capturing both data distribution and semantic structure, along with a cooperative generator designed for effective MCMC initialization. Our joint space optimization only requires MCMC sampling in the data space, and allows the energy to learn semantic data–latent relationships directly from real data. Experiments show our method improves generation quality and training stability with fewer resources, and performs effectively across multiple downstream tasks.

Applications · Robotics

Lukas Lao Beyer, Sertac Karaman

Deep learning methods have vastly expanded the capabilities of motion planning in robotics applications, as learning priors from large-scale data has been shown to be essential in capturing the highly complex behavior required for solving tasks such as manipulation or navigation for autonomous vehicles. At the same time, model-based planning algorithms based on search or optimization remain an essential tool due to their flexibility, efficiency, and the ability to incorporate domain knowledge via expert-designed algorithms and objective functions. We propose a new generative framework to unify these two paradigms. First, we learn an autoencoder with a high compression ratio and a latent space of hierarchically ordered, discrete-valued tokens. Leveraging both the dimensionality reduction and the hierarchical coarse-to-fine structure learned by this autoencoder, we then perform motion planning by directly searching in the latent space of tokens. This search can optimize arbitrary objective functions specified at test time, providing a large degree of flexibility while maintaining efficiency and producing realistic solutions by relying on the generative capabilities of the highly compressed autoencoder. We evaluate our method on nuPlan and the Waymo Open Motion Dataset, showing how latent space search can be used for a variety of guided behavior generation tasks, achieving strong performance for closed-loop motion planning and multi-agent guided scenario synthesis without requiring any task-specific training.