论文检索

Deep Learning · Generative Models and Autoencoders

Zehua Chen, Yucheng Yang, Binjie Yuan, Kaiwen Zheng, Jun Liu, Jun Zhu

Guidance methods, e.g., classifier-free guidance (CFG) and auto-guidance (AG), have distinctively improved noise-to-data diffusion generation results. Recently, bridge models have been proposed, which present a data-to-data sampling process to exploit instructive information from clean prior representation, showing advantages on the tasks such as image-to-image translation. In this work, we design a custom guidance method for bridge models, named prior guidance (PG). Different from highlighting condition alignment (CFG) or score accuracy (AG), we training-freely construct an additional weak prior for the pre-trained bridge models, and extrapolate the estimation results to further encourage prior exploitation. Then, we analyze the underlying mechanism of prior exploitation in bridge process and design frequency-modulated prior guidance (FMPG), which tailors the guidance scale to low- and high-frequency bands coherent with bridge generative dynamics. Finally, considering the challenge of bridge models on image in-painting, we develop a cascaded guidance framework, CFG-FMPG, that first generates a coarse prior under global semantic condition and then refines it with FMPG, naturally fulfilling their complementary advantages along sampling trajectory. Experiments conducted on strong pre-trained bridge models, DDBM and DBIM, valid the consistent improvement achieved by our training-free design.

Deep Learning · Generative Models and Autoencoders

Ganggui Ding, Xiaogang Xu, Hao Chen, Chunhua Shen

Generative video diffusion models have shown strong robustness to large motion and occlusions for video frame interpolation (VFI). However, their inference efficiency lags significantly behind learning-based methods due to the structural redundancy of pairwise inference and the procedural latency of multi-step iterative denoising. To address these limitations, we propose SpeedVFI, a one-step diffusion framework that achieves dual efficiency improvements by interpolating the entire video sequence in a single forward pass to eliminate pairwise overhead, and distilling the generation trajectory into a one-step denoising process to bypass iterative latency. To support this high-efficiency architecture, we introduce temporal RoPE alignment to ensure temporal consistency across the unified sequence, and noise-centric partial attention to reduce computational overhead while preserving global context. Extensive experiments demonstrate that SpeedVFI accelerates diffusion-based VFI by orders of magnitude while maintaining competitive quantitative and visual quality.

Deep Learning · Large Language Models

Paul Jünger, Justin Lovelace, Linxi Zhao, Dongyoung Go, Kilian Weinberger

Diffusion language models offer fast, parallel decoding via non-autoregressive generation and uncertainty-aware denoising, yet these properties remain underexplored for retrieval. We propose *Self-Augmenting Retrieval for Diffusion Language Models*, a dynamic framework that uses intermediate diffusion states to refine retrieval throughout the denoising trajectory. At each iteration, we query an external corpus with the partially denoised text, retrieve additional evidence, and condition subsequent denoising steps on the updated context. This tightly couples retrieval to the diffusion process: high-confidence tokens guide retrieval early, while uncertain spans are completed after new evidence is incorporated. Experiments with DREAM-7B, a discrete diffusion language model, on open-domain question answering benchmarks show significant improvements in answer accuracy over static question-only retrieval, while achieving 2--6$\times$ higher throughput than autoregressive baselines, demonstrating that diffusion decoding offers a compelling paradigm for efficient, high-quality retrieval-augmented generation.

Deep Learning · Generative Models and Autoencoders

Rajalaxmi Rajagopalan, Debottam Dutta, Yu-Lin Wei, Romit Roy Choudhury

Imagine Alice has a specific image $x^\ast$ in her mind, say, the view of the street in which she grew up during her childhood. To generate that exact image, she guides a generative model with multiple rounds of prompting and arrives at an image $x^{p*}$. Although $x^{p*}$ is reasonably close to $x^\ast$, Alice finds it difficult to close that gap using language prompts. This paper aims to narrow this gap by observing that even after language has reached its limits, humans can still tell when a new image $x^+$ is closer to $x^\ast$ than $x^{p*}$. Leveraging this observation, we develop **MultiBO** (Multi-Choice Preferential Bayesian Optimization) that carefully generates $K$ new images as a function of $x^{p*}$, gets preferential feedback from the user, uses the feedback to guide the diffusion model, and ultimately generates a new set of $K$ images. We show that within $B$ rounds of user feedback, it is possible to arrive much closer to $x^\ast$, even though the generative model has no information about $x^\ast$. Qualitative scores from $30$ users, combined with quantitative metrics compared across $5$ baselines, show promising results, suggesting that multi-choice feedback from humans can be effectively harnessed for personalized image generation.

Sen Ye, Jianning Pei, Mengde Xu, Shuyang Gu, Chunyu Wang, Liwei Wang, Han Hu

Most visual generative models compress images into a latent space before applying diffusion or autoregressive modelling. Yet, existing approaches such as VAEs and foundation model aligned encoders implicitly constrain the latent space without explicitly shaping its distribution, making it unclear which types of distributions are optimal for modeling. We introduce **Distribution-Matching VAE** (**DMVAE**), which explicitly aligns the encoder’s latent distribution with an arbitrary reference distribution via a distribution matching constraint. This generalizes beyond the Gaussian prior of conventional VAEs, enabling alignment with distributions derived from self-supervised features, diffusion noise, or other prior distributions. With DMVAE, we can systematically investigate which latent distributions are more conducive to modeling, and we find that SSL-derived distributions provide an excellent balance between reconstruction fidelity and modeling efficiency, reaching a gFID of 3.2 on ImageNet with only 64 training epochs. Our results suggest that choosing a suitable latent distribution structure (achieved via distribution-level alignment), rather than relying on fixed priors, is key to bridging the gap between easy-to-model latents and high-fidelity image synthesis.

Qi Li, Runpeng Yu, Haiquan Lu, Xinchao Wang

Discrete Diffusion Large Language Models (dLLMs) have recently emerged as a promising non-autoregressive paradigm, offering faster inference while achieving strong performance in code generation and mathematical reasoning tasks. In this work, we show that dLLMs’ decoding mechanism not only improves utility but also enables effective model attribution: by analyzing a response’s decoding trajectory, we can identify its source model and help mitigate risks from model misuse. A key challenge is the diversity of attribution scenarios, ranging from distinguishing different models to identifying different checkpoints or backups of the same model. To ensure broad applicability, we focus on two core questions: what information to extract from the decoding trajectory, and how to use it effectively. We first observe that per-step model confidence is ineffective, as the bidirectional nature of dLLMs causes mutual influence among decoded tokens, leading to highly redundant confidence signals that obscure structural information about decoding order and dependencies. To overcome this, we propose a novel information extraction scheme called the \textit{Directed Decoding Map (DDM)}, which captures structural relationships between decoding steps and reveals model-specific behaviors. Furthermore, to fully leverage the extracted structure, we propose \textit{Gaussian-Trajectory Attribution (GTA)}, which fits a cell-wise Gaussian distribution at each decoding position for each model and uses log-likelihood differences between trajectories as the attribution score. Extensive experiments across diverse models, datasets and different model access assumptions validate the effectiveness of our approach.

Applications · Computer Vision

Liangsi Lu, Minzhe Guo, Xuhang Chen, Yang Shi

Despite the generative capabilities of diffusion models, real-image editing remains constrained by a persistent trade-off between semantic editability and structural fidelity. We identify a primary cause of this limitation as the implicit coupling of editing progress with noise scale in existing paradigms. This coupling creates a budget misallocation: achieving stronger semantic changes often necessitates initializing from high-noise states, which can consume computation on disrupting global layout before semantic modification begins. To address this, we introduce NaviEdit, a training-free framework that decouples the editing trajectory from the denoising schedule via a strict Time-Axis Consistency principle. By reformulating editing as controlled vector field navigation on a distinct task axis, NaviEdit strategically concentrates the computational budget within semantically responsive intermediate scales while reducing exposure to destructive high-noise regimes. Experiments show that NaviEdit outperforms strong state-of-the-art baselines across PIE-Bench, achieving larger semantic edits with better structure preservation under comparable compute budgets, without requiring model tuning.

Tianhao Zhao, Youjia Zhang, Hang Long, Jinshen Zhang, Wenbing Li, Yang Yang, Gongbo Zhang, Jozef Hladký, Matthias Nießner, Wei Yang

In this paper, we introduce LATO, a novel topology-preserving latent representation that enables scalable, flow matching-based synthesis of explicit 3D meshes. LATO represents a mesh as a Vertex Displacement Field (VDF) anchored on surface, incorporating a sparse voxel Variational Autoencoder (VAE) to compress this explicit signal into a structured, topology-aware voxel latent. To decapsulate the mesh, the VAE decoder progressively subdivides and prunes latent voxels to instantiate precise vertex locations. In the end, a dedicated connection head queries the voxel latent to predict edge connectivity between vertex pairs directly, allowing mesh topology to be recovered without isosurface extraction or heuristic meshing. For generative modeling, LATO adopts a two-stage flow matching process, first synthesizing the structure voxels and subsequently refining the voxel-wise topology features. Compared to prior isosurface/triangle-based diffusion models and autoregressive generation approaches, LATO generates meshes with complex geometry, well-formed topology while being highly efficient in inference.

Deep Learning · Large Language Models

Julie Kallini, Artidoro Pagnoni, Tomasz Limisiewicz, Gargi Ghosh, Luke Zettlemoyer, Christopher Potts, Xiaochuang Han, Srinivasan Iyer

Recent byte-level language models (LMs) match the performance of token-level models without relying on subword vocabularies, yet their practical deployment is limited by slow inference. In this work, we enhance the Byte Latent Transformer (BLT) using new training and inference techniques. First, we introduce **BLT Diffusion (BLT-D)**, a new model and our fastest BLT variant. BLT-D is trained with an auxiliary block-wise diffusion objective over byte blocks alongside the standard next-byte prediction loss. This enables an inference procedure that generates multiple bytes in parallel per decoding step, substantially improving decoding efficiency. Second, we propose two extensions inspired by speculative decoding that trade some speed for improved quality: **BLT Self-speculation (BLT-S)**, a faster generation method for BLT in which it speculates bytes beyond its normal patch boundaries and verifies its own generations; and **BLT Diffusion+Verification (BLT-DV)**, which enhances BLT-D by adding an autoregressive verification step after diffusion-based generation. Each approach offers its own unique advantages, and together, they overcome key barriers to large-scale deployment of byte-level LMs.

Social Aspects · Security

Pyo Min Hong, Albert No

We propose dgMARK, a decoding-guided watermarking method for discrete diffusion language models (dLLMs). Unlike autoregressive models, dLLMs can generate tokens in arbitrary order. While an ideal conditional predictor would be invariant to this order, practical dLLMs exhibit strong sensitivity to the unmasking order, creating a new channel for watermarking. dgMARK steers the unmasking order toward positions whose high-reward candidate tokens satisfy a simple parity constraint induced by a binary hash, without explicitly reweighting the model’s learned probabilities. The method is plug-and-play with common decoding strategies (e.g., confidence, entropy, and margin-based ordering) and can be strengthened with a one-step lookahead variant. Watermarks are detected via elevated parity-matching statistics, and a sliding-window detector ensures robustness under post-editing operations including insertion, deletion, substitution, and paraphrasing.

Deep Learning · Generative Models and Autoencoders

Binxu Wang, Jacob A Zavatone-Veth, Cengiz Pehlevan

Diffusion models trained on different, non-overlapping subsets of a dataset often produce strikingly similar outputs when given the same noise seed. We trace this consistency to a simple linear effect: the shared Gaussian statistics across splits already predict much of the generated images. To formalize this, we develop a random matrix theory (RMT) framework that quantifies how finite datasets shape the expectation and variance of the learned denoiser and sampling map in the linear setting. For expectations, sampling variability acts as a renormalization of the noise level through a self-consistent relation $\sigma^2\to\kappa(\sigma^2)$, explaining why limited data overshrink low-variance directions and pull samples toward the dataset mean. For fluctuations, our variance formulas reveal three key factors behind cross-split disagreement: \textit{anisotropy} across eigenmodes, \textit{inhomogeneity} across inputs, and overall scaling with dataset size. Extending deterministic-equivalence tools to fractional matrix powers further allows us to analyze entire sampling trajectories. The theory sharply predicts the behavior of linear diffusion models, and we validate its predictions on UNet and DiT architectures in their non-memorization regime, identifying where and how samples deviates across training data split. This provides a principled baseline for reproducibility in diffusion training, linking spectral properties of data to the stability of generative outputs.

Applications · Computer Vision

RuiQiang Zhang, Hengyi Wang, Chang Liu, Guanjie Wang, Zehua Ma, Weiming Zhang

Large-scale text-to-image (T2I) diffusion models excel at open-domain synthesis but still struggle with precise text rendering, especially for multi-line layouts, dense typography, and long-tailed scripts such as Chinese. Prior solutions typically necessitate costly retraining or impose rigid external layout constraints, often compromising aesthetic quality and flexibility. We propose **FreeText**, a training-free, plug-and-play framework that improves text rendering by leveraging intrinsic mechanisms of *Diffusion Transformer (DiT)* models. **FreeText** decomposes the problem into *where to write* and *what to write*. For the former, we localize writing regions by extracting token-wise spatial attribution from image-to-text attention, using sink-like tokens as stable spatial anchors and topology-aware refinement to produce high-confidence masks. For the latter, we introduce Spectral-Modulated Glyph Injection (SGMI), which injects a noise-aligned glyph prior with frequency-domain band-pass modulation to strengthen glyph structure and mitigate semantic leakage (rendering the concept instead of the word). Extensive experiments on Qwen-Image, FLUX.1-dev, and SD3 variants across longText-Benchmark, CVTG, and our CLT-Bench show consistent gains in text readability while maintaining semantic alignment and aesthetic quality, with modest inference overhead.

Probabilistic Methods · Monte Carlo and Sampling Methods

Denis Blessing, Lorenz Richter, Julius Berner, Egor Malitskiy, Gerhard Neumann

Sampling from unnormalized densities using diffusion models has emerged as a powerful paradigm. However, while recent approaches that use least-squares `matching' objectives have improved scalability, they often necessitate significant trade-offs, such as restricting prior distributions or relying on unstable optimization schemes. By generalizing these methods as special forms of fixed-point iterations rooted in Nelson's relation, we develop a new method that addresses these limitations. Our approach enables learning a stochastic transport map between arbitrary prior and target distributions with a single, scalable, and stable objective. Furthermore, we introduce a damped variant of this iteration that incorporates a regularization term to mitigate mode collapse. Empirically, we demonstrate that our method enables sampling at unprecedented scales while preserving mode diversity, achieving state-of-the-art results on complex synthetic densities and high-dimensional molecular benchmarks.

Applications · Computer Vision

Xiaotong Fu, Wenchao Meng, Qihang Zhou, Qian Liu, Qinmin Yang, Shibo He

Diffusion-based camouflaged object detection (COD) has recently shown great potential. In contrast to existing approaches that rely on multiple sample steps to refine the predicted masks, we propose CODiff, which reformulates the diffusion process to enable one-step mask prediction while maintaining competitive accuracy. Specifically, we first establish the theoretical feasibility of one-step sampling for COD. Based on this, we design a dedicated network for one-step inference with a global semantic guidance mechanism to guide the denoising process globally and hierarchical condition integration blocks to provide fine-grained structural semantics. In addition, we design a straight-forward regularization to learn better intermediate features by bridging the representation gap between the condition backbone and the diffusion model. Extensive experiments demonstrate that CODiff achieves state-of-the-art performance across multiple benchmarks, improving MAE by over 22\% on the challenging COD10K dataset. The code will be released upon publication.

Probabilistic Methods · Everything Else

Matteo Gätzner, Johannes Kirschner

We present a principled framework for uncertainty quantification in computed tomography (CT) reconstruction. Based on the sequential likelihood mixing framework (Kirschner et al., 2025), we establish the first confidence regions with theoretical coverage guarantees for deep learning-based CT reconstructions. In particular, we consider a realistic forward model following the Beer-Lambert law, i.e., a log-linear forward model with Poisson noise, closely reflecting clinical and scientific imaging conditions. The framework is general and applies to both classical algorithms and deep learning reconstruction methods, including U-Nets, U-Net ensembles, and generative Diffusion models. Empirically, we demonstrate that deep reconstruction methods yield substantially tighter confidence regions than classical reconstructions, without sacrificing theoretical coverage guarantees. Our approach allows the detection of hallucinations in reconstructed images and provides interpretable visualizations of confidence regions. This establishes deep models not only as powerful estimators, but also as reliable tools for uncertainty-aware medical imaging.

Deep Learning · Generative Models and Autoencoders

Brett Levac, Jon Tamir, Marcelo Pereyra, Julián Tachella

Diffusion models (DMs) are a powerful framework for image generation and restoration. However, existing DMs are primarily trained in a supervised manner by using a large corpus of clean images. This poses fundamental challenges in many real-world scenarios, where acquiring noise-free data is hard or infeasible. While some methods are capable of training DMs using noisy data, they are effective only when the amount of noise is very mild or when additional noise-free data is available. In addition, existing methods for training DMs from incomplete measurements require access to multiple complementary acquisition processes, a significant practical limitation. Here we introduce the first approach for learning DMs for image restoration using only noisy measurement data from a single operator. First, we show that DMs, and more broadly minimum mean squared error denoisers, exhibit a weak form of scale equivariance linking rescaling in signal amplitude to changes in noise intensity. We then leverage this theoretical insight to develop a denoising score-matching strategy that generalizes robustly to noise levels below the training data, thereby enabling the learning of DMs from noisy measurements. For problems involving measurements both noisy and incomplete, we integrate our method with equivariant imaging, a complementary self-supervised learning framework that exploits the inherent invariants of imaging problems. This allows training DMs for image restoration from single-operator noisy measurements. We validate the effectiveness of our approach through extensive experiments on image denoising, demosaicing, inpainting, and MRI reconstruction along with comparisons with the state of the art.

Deep Learning · Large Language Models

Jingwei Zhang, Haoyu LEI, Zijin Feng, Jiacheng Sun, Farzan Farnia

Although diffusion models have revolutionized continuous domains like image synthesis through high quality generations and controllable guidance mechanisms, bringing this controllability to the discrete, sequential nature of text remains an open challenge. Meanwhile, current sampling strategies and guidance methods adjust token likelihoods without capturing the broader semantic landscape, leading to a suboptimal balance between fidelity and diversity. In this work, we introduce a novel training-free Semantic-Aware Kernel Entropy (SAKE) guidance method. Our method computes the order-2 Rényi entropy over a kernel Gram matrix that captures both cross-token semantic interactions and relative token positions. By linearizing this objective in the embedding space, we derive a tractable guidance signal that dynamically adjusts the sampling distribution—flattening it to encourage exploration during redundancy and sharpening it for fidelity when diverse. Empirical experiments demonstrate that our approach achieves a superior Pareto frontier between fidelity and diversity, and improves multi-sample performance on reasoning-intensive tasks, such as code and mathematics generation, compared to temperature scaling and discrete guidance baselines.

Reinforcement Learning · Deep RL

Calvin Luo, Chen Sun, shuran song

A promising recipe towards intelligent robotic decision-making is the finetuning of pretrained generative control policies, which can summarize offline experience effectively through behavior cloning, with reinforcement learning techniques to adapt them to online experience. In this work we present Diffusion Filtered Exploration via Ensembles (DF-ExpEnse), an exploration technique that meaningfully improves the quality of online experience collection, thus increasing the sample efficiency of the finetuning procedure. DF-ExpEnse first leverages the multimodal modeling capability of the generative control policy to create an expressive and tractably evaluatable candidate set. Then, it utilizes an ensemble of critics to identify an action with high exploration interest that best balances quality with uncertainty. When instantiated in a parallelized fleet, DF-ExpEnse further utilizes cross-agent communication to facilitate collaborative exploration as a group. As it is only used for online experience collection, DF-ExpEnse can be seamlessly integrated on top of existing techniques that seek to finetune pretrained generative control policies via reinforcement learning. We experimentally validate consistent sample-efficiency benefits when using DF-ExpEnse for exploration over both manipulation and locomotion tasks, compared to default finetuning and alternative action selection schemes.

Social Aspects · Security

Ji Guo, xiaolong qin, Wenbo Jiang, Cencen Liu, Jielei Wang, Jierun Chen

Vision-Language Models (VLMs) have achieved remarkable success in tasks such as image captioning and visual question answering (VQA). However, as their applications become increasingly widespread, recent studies have revealed that VLMs are vulnerable to backdoor attacks. Existing backdoor attacks on VLMs primarily rely on data poisoning by adding visual triggers and modifying text labels, where the induced image–text mismatch makes poisoned samples easy to detect. To address this limitation, we propose the Clean-Label Backdoor Attack on VLMs via Diffusion Models (CBV), which leverages diffusion models to generate natural poisoned examples via score matching. Specifically, CBV modifies the score during the reverse generation process of the diffusion model to guide the generation of poisoned samples that contain triggered image features. To further enhance the effectiveness of the attack, we incorporate the textual information of the triggered images as multimodal guidance during generation. Moreover, to enhance stealthiness, we introduce a GradCAM-guided Mask (GM) that restricts modifications to only the most semantically important regions, rather than the entire image. We evaluate our method on MSCOCO and VQA v2 with four representative VLMs, achieving over 80\% ASR while preserving normal functionality.

Bohdan Turbal, Blossom Metevier, Max Springer, Aleksandra Korolova

Although there is a rich literature on adversarial attacks on large language models, their current practical impact is limited. Gradient-based attacks such as Greedy Coordinate Gradient (Zou et al., 2023) typically produce high-perplexity, incoherent suffixes that are easily detectable and thus easy to guard against, especially in combination with other defense-in-depth techniques (Bengio et al., 2024). On the other hand, attacks that aim to produce coherent prompts often alter the semantic intent of the original query. When the model complies with such altered query, it often produces a response that is not actually useful for the original query, thus incurring the so-called "jailbreak tax". In this work, we introduce a novel framework that can efficiently generate adversarial attacks against safety-aligned models while maintaining low perplexity and high semantic adherence to the adversary's original intent. The framework, Greedy Coordinate Diffusion (GCD), leverages the generative priors of discrete diffusion language models to guide the search for adversarial suffixes that achieve semantic coherence and adherence. Furthermore, unlike GCG, GCD does not require direct gradient access, allowing it to operate in a gray-box setting. We empirically demonstrate the power of GCD by showing it achieves state-of-the-art attack success rates against aligned models, and its adversarial prompts are not detected by semantic filters such as llama-guard-3.