论文检索

Deep Learning · Generative Models and Autoencoders

Yutian Zhao, Chao Du, Xiaosen Zheng, Tianyu Pang, Min Lin

Data attribution for generative models seeks to quantify the influence of individual training examples on model outputs. Existing methods for diffusion models typically require access to model gradients or retraining, limiting their applicability in proprietary or large-scale settings. We propose a *nonparametric* attribution method that operates entirely on data, measuring influence via patch-level similarity between generated and training images. Our approach is grounded in the analytical form of the optimal score function and naturally extends to multiscale representations, while remaining computationally efficient through convolution-based acceleration. In addition to producing spatially interpretable attributions, our framework uncovers patterns that reflect intrinsic relationships between training data and outputs, independent of any specific model. Experiments demonstrate that our method achieves strong attribution performance, closely matching gradient-based approaches and substantially outperforming existing nonparametric baselines.

Deep Learning · Generative Models and Autoencoders

Binhao Wang, Shihao Zhao, Bo Cheng, Qiuyu Ji, YuhangMa, Liebucha Wu, Shanyuan Liu, Dawei Leng, Yuhui Yin

Recent diffusion-based approaches have made substantial progress in image layer decomposition. However, accurately decomposing complex natural images remains challenging due to difficulties in occlusion completion, robust layer disentanglement, and precise foreground boundaries. Moreover, the scarcity of high-quality multi-layer natural image datasets limits advancement. To address these challenges, we propose **RevealLayer**, a diffusion-based framework that decomposes an RGB image into multiple RGBA layers, enabling precise layer separation and reliable recovery of occluded content in natural images. RevealLayer incorporates three key components: (1) a **Region-Aware Attention** module to disentangle hidden and visible layers; (2) an **Occlusion-Guided Adapter** to leverage contextual information to enhance overlapping regions; and (3) a **composite loss** to enforce sharp alpha boundaries and suppress residual artifacts. To support training and evaluation, we introduce **RevealLayer-100K**, a high-quality multi-layer natural image constructed through a collaboration between automated algorithms and human annotation, and further establish **RevealLayerBench** for benchmarking layer decomposition in general natural scenes. Extensive experiments demonstrate that RevealLayer consistently outperforms existing approaches in layer decomposition.

Deep Learning · Generative Models and Autoencoders

Jiajun He, Zongyu Guo, Zhaoyang Jia, Xiaoyi Zhang, Jiahao Li, Xiao Li, Bin Li, Jose Miguel Hernandez-Lobato, Yan Lu

Modern visual generative models acquire rich visual knowledge through large-scale training, yet existing visual representations (such as pixels, latents, or tokens) remain external to the model and cannot directly exploit this knowledge for compact storage or reuse. In this work, we introduce a new visual representation framework that encodes a signal as a function, which is parametrized by low-rank adaptations attached to a frozen visual generative model. Such implicit representations are learned by compressing the visual signal, e.g., an 81-frame video, into a single compact vector, achieving strong perceptual video compression at extremely low bitrates. Beyond basic compression, the functional nature of this representation enables inference-time scaling and control, allowing additional refinement on the compression performance. More broadly, as the implicit representations directly act as a function of the generation process, this suggests a unified framework bridging visual compression and generation.

Social Aspects · Safety

Yunqi Xue, Zhijiang Li, Phil Torr, Jindong Gu

Unlike diffusion-based models that operate in continuous latent spaces, autoregressive unified multimodal models produce images by sequentially predicting discretized visual tokens. These tokens are derived from a codebook that maps embeddings to quantized visual patterns. The language-like architecture enables unified multimodal models to effectively capture text conditional information for generation, making them promising for text-to-image tasks. This also raises an interesting question: how safe are the images generated in such an autoregressive way? Existing methods that ensure safe generation by operating on diffusion continuous representations fail to generalize well to discrete representations. In this work, we propose iterative self-improving codebooks for safe autoregressive generation. We leverage the understanding and judgment capabilities of the unified multimodal model itself to identify unsafe generated images without human annotation. Subsequently, the inherent representations in the codebook are fixed to eliminate harmful mappings. Our method comprises two steps: first, we use the unified model to identify unsafe generations and construct corresponding harmful and safe image-text pairs. These pairs are used to construct the Harmful Space and guide updates to the codebook, thereby eliminating harmful outputs. Second, we perform adaptive fine-tuning on the codebook within the harmless space using safe image-text pairs to ensure the quality of generated images. These two steps are repeated until no further improvement is observed, producing a safety-enhanced model codebook. Extensive experiments are conducted to verify the effectiveness of our method on five unified multimodal models. Without additional external feedback, the safety of models is improved iteratively.

Deep Learning · Generative Models and Autoencoders

Hongyang Du, Hongyang Du, Xiaoyan Cong, Runhao Li, Jingcheng Ni, Aman Agarwal, Zeqi Zhou, Zekun Li, Randall Balestriero, Yue Wang

While recent video diffusion models (VDMs) produce visually impressive results, they fundamentally struggle to maintain 3D structural consistency, often resulting in object deformation or spatial drift. We hypothesize that these failures arise because standard denoising objectives lack explicit incentives for geometric coherence. To address this, we introduce VideoGPA (Video Geometric Preference Alignment), a data-efficient self-supervised framework that leverages a geometry foundation model to automatically derive dense preference signals that guide VDMs via Direct Preference Optimization (DPO). This approach effectively steers the generative distribution toward inherent 3D consistency without requiring human annotations. VideoGPA significantly enhances temporal stability, physical plausibility, and motion coherence using minimal preference pairs, consistently outperforming state-of-the-art baselines in extensive experiments.

Deep Learning · Generative Models and Autoencoders

Marco Pegoraro, Evan Atherton, Bruno Roy, Aliasghar Khani, Arianna Rampini

Learning natural body motion remains challenging due to the strong coupling between spatial geometry and temporal dynamics. Embedding motion in phase manifolds, latent spaces that capture local periodicity, has proven effective for motion prediction; however, existing approaches lack scalability and remain confined to specific settings. We introduce FunPhase, a functional periodic autoencoder that learns a phase manifold for motion and replaces discrete temporal decoding with a function-space formulation, enabling smooth trajectories that can be sampled at arbitrary temporal resolutions. FunPhase unifies motion prediction and generation within a single interpretable phase manifold, enabling motion generation via latent diffusion, generalizes across skeletons and datasets, and supports downstream tasks such as motion super-resolution and partial-body completion. Our model achieves substantially lower reconstruction error than prior periodic autoencoder baselines, achieving uniform improvements of at least 45% across all metrics, while enabling a broader range of applications and performing on par with state-of-the-art motion generation methods.

Deep Learning · Generative Models and Autoencoders

Yida Xiong, Jiameng Chen, Xiuwen Gong, Jia Wu, Shirui Pan, wenbin Hu

Graph generation aims to sample discrete node and edge attributes while satisfying coupled structural constraints. Diffusion models for graphs often adopt largely factorized forward-noising, and many flow-matching methods start from factorized reference noise and coordinate-wise interpolation, so node–edge coupling is not encoded by the generative geometry and must be recovered implicitly by the core network, which can be brittle after discrete decoding. Bayesian Flow Networks (BFNs) evolve distribution parameters and naturally support discrete generation. But classical BFNs typically rely on factorized beliefs and independent channels, which limit geometric evidence fusion. We propose Variational Bayesian Flow Network (VBFN), which performs a variational lifting to a tractable joint Gaussian variational belief family governed by structured precisions. Each Bayesian update reduces to solving a symmetric positive definite linear system, enabling coupled node and edge updates within a single fusion step. We construct sample-agnostic sparse precisions from a representation-induced dependency graph, thereby avoiding label leakage while enforcing node–edge consistency. On synthetic and molecular graph datasets, VBFN improves fidelity and diversity, and surpasses baseline methods. The reproducible code is available at https://anonymous.4open.science/r/VBFN-D6B0/.

Deep Learning · Generative Models and Autoencoders

Dengsheng Chen, Yangming Shi, Enhua Wu

Conventional autoregressive models achieve causality through next-token prediction, but suffer from prohibitive latency and typically under-perform non-causal alternatives such as masked prediction and diffusion. We propose self-token prediction, which enables predicting a flexible number of tokens per step, and introduce AGARIC, the first spatially causal image generator built on this paradigm. AGARIC delivers markedly faster inference and consistently outperforms prior autoregressive baselines, matching the performance of state-of-the-art non-causal models. Our findings point to self-token prediction as a key step toward unified and efficient multimodal autoregressive modeling.

Theory · Everything Else

Naïl B. Khelifa, Richard E Turner, Ramji Venkataramanan

Machine learning models are increasingly trained or fine-tuned on synthetic data. Recursively training on such data has been observed to significantly degrade performance in a wide range of tasks, often characterized by a progressive drift away from the target distribution. In this work, we theoretically analyze this phenomenon in the setting of score-based diffusion models. For a realistic pipeline where each training round uses a combination of synthetic data and fresh samples from the target distribution, we obtain upper and lower bounds on the accumulated divergence between the generated and target distributions. This allows us to characterize different regimes of drift, depending on the score estimation error and the proportion of fresh data used in each generation. We also provide empirical results on synthetic data and images to illustrate the theory.

Applications · Computer Vision

Tianyi Xie, Zhiyuan Yu, kaihong huang, Beilun Wang, Zhaoyang Wang, Dian Shen

Diffusion-based text-to-image (T2I) models have demonstrated remarkable advancements in generating high-quality images. However, while real-world applications like product packaging and logo design necessitate synthesis within irregular geometries, existing methods struggle to handle such constraints. Therefore, generating complete pictures that conform to arbitrary-shaped canvas constraints while maintaining semantic integrity remains a significant challenge. To address this, we introduce AnyCanvas, a training-free framework that leverages a Mask-to-Potential Field paradigm to convert binary masks into a differentiable potential field, which guides content to naturally converge within target regions. Extensive experiments demonstrate that AnyCanvas achieves 4.23\% higher spatial adherence to user-specified constraints while maintaining 99.45\% of the semantic fidelity measured by CLIP score, leading to a superior harmonic mean of spatial and semantic metrics. AnyCanvas also exhibits robust generalizability across different model backbones and versatile spatial control objectives.

Deep Learning · Generative Models and Autoencoders

Yuxuan Yao, Yuxuan Chen, Hui Li, Kaihui Cheng, Qipeng Guo, Yuwei Sun, Zilong Dong, Jingdong Wang, Siyu Zhu

Multimodal Diffusion Transformers (MMDiTs) for text-to-image generation maintain separate text and image branches, with bidirectional information flow between text tokens and visual latents throughout denoising. In this setting, we observe a prompt forgetting phenomenon: the semantics of the prompt representation in the text branch is progressively forgotten as depth increases. We further verify this effect on three representative MMDiTs—SD3, SD3.5, and FLUX.1 by probing linguistic attributes of the representations over the layers in the text branch. Motivated by these findings, we introduce a training-free approach, prompt reinjection, which reinjects prompt representations from early layers into later layers to alleviate this forgetting. Experiments on GenEval, DPG, and T2I-CompBench++ show consistent gains in instruction-following capability, along with improvements on metrics capturing preference, aesthetics, and overall text--image generation quality.

Social Aspects · Accountability, Transparency, and Interpretability

MingYu Lu, Soham Gadgil, Chris Lin, Chanwoo Kim, Su-In Lee

As Text-to-Image (T2I) diffusion models are increasingly used in real-world creative workflows, a principled framework for valuing contributors who provide a collection of data is essential for fair compensation and sustainable data marketplaces. While the Shapley value offers a theoretically grounded approach to attribution, it faces a dual computational bottleneck: (i) the prohibitive cost of exhaustive model retraining for each sampled subset of players (i.e., data contributors) and (ii) the combinatorial number of subsets needed to estimate marginal contributions due to contributor interactions. To this end, we propose **SurrogateSHAP**, a retraining-free framework that approximates the expensive retraining game through inference from a pretrained model. To further improve efficiency, we employ a gradient-boosted tree to approximate the utility function and derive Shapley values analytically from the tree-based model. We evaluate SurrogateSHAP across three diverse attribution tasks: (i) image quality for DDPM-CFG on CIFAR-20, (ii) aesthetics for Stable Diffusion on Post-Impressionist artworks, and (iii) product diversity for FLUX.1 on Fashion-Product data. Across settings, SurrogateSHAP outperforms prior methods while substantially reducing computational overhead, consistently identifying influential contributors across multiple utility metrics. Finally, we demonstrate that SurrogateSHAP effectively localizes data sources responsible for spurious correlations in clinical images, providing a scalable path toward auditing safety-critical generative models. Code is available at https://anonymous.4open.science/r/CFG-Attribution-15DD/

General Machine Learning · Everything Else

Chunxu Zhang, Weipeng Zhang, Guodong Long, Zhiheng Xue, Bo Yang

Most federated recommender systems represent each user with a single embedding learned from local interaction data, implicitly assuming that user preferences are fixed and precisely identifiable. In federated settings, however, each client observes only a limited and fragmentary view of user behavior, rendering such point estimates inherently brittle. To address this mismatch, we model user preferences as distributions rather than points, allowing multiple compatible preference representations to coexist. Rather than collapsing evidence into a single embedding, our approach preserves uncertainty and diversity in user representations, providing a richer basis for preference modeling. We instantiate this idea with a diffusion-based generative framework that produces diverse user embeddings and derives recommendation scores by aggregating predictions across them. This distributional formulation yields more stable ranking behavior and improved robustness under ambiguous feedback. Extensive experiments on federated recommendation benchmark datasets demonstrate consistent and significant improvements over baselines. Our code is available.

Deep Learning · Generative Models and Autoencoders

Yunhan Yang, Chunshi Wang, Junliang Ye, YANG LI, Zanxin Chen, Zehuan Huang, Yao Mu, Zhuo Chen, Chunchao Guo, Xihui Liu

Synthesizing physics-grounded 3D assets is a critical bottleneck for interactive virtual worlds and embodied AI. Existing methods predominantly focus on static geometry, overlooking the functional properties essential for interaction. We propose that interactive asset generation must be rooted in functional logic and hierarchical physics. To bridge this gap, we introduce PhysForge, a decoupled two-stage framework supported by PhysDB, a large-scale dataset of 150,000 assets with four-tier physical annotations. First, a VLM acts as a physical architect to plan a Hierarchical Physical Blueprint defining material, functional, and kinematic constraints. Second, a physics-grounded diffusion model realizes this blueprint by synthesizing high-fidelity geometry alongside precise kinematic parameters via a novel KineVoxel Injection (KVI) mechanism. Experiments demonstrate that PhysForge produces functionally plausible, simulation-ready assets, providing a robust data engine for interactive 3D content and embodied agents.

Applications · Computer Vision

Qingdong He, Xueqin Chen, Chaoyi Wang, Yanjie Pan, Xiaobin Hu, Zhenye Gan, Chengjie Wang, Xiangtai Li, Jiangning Zhang, Yabiao Wang

Instruction-based image editing (IIE) has advanced rapidly with the success of diffusion models. However, existing efforts primarily focus on simple and explicit instructions to execute editing operations such as adding, deleting, moving, or swapping objects. They struggle to handle more complex implicit hypothetical instructions that require deeper reasoning to infer plausible visual changes and user intent. Additionally, current datasets provide limited support for training and evaluating reasoning-aware editing capabilities. Architecturally, these methods also lack mechanisms for fine-grained detail extraction that support such reasoning. To address these limitations, we propose Reason50K, a large-scale dataset specifically curated for training and evaluating hypothetical instruction–reasoning image editing, along with ReasonBrain, a novel framework designed to reason over and execute implicit hypothetical instructions across diverse scenarios. Reason50K includes over 50K samples spanning four key reasoning scenarios: Physical, Temporal, Causal, and Story reasoning. ReasonBrain leverages Multimodal Large Language Models (MLLMs) for editing guidance generation and a diffusion model for image synthesis, incorporating a Fine-grained Reasoning Cue Extraction (FRCE) module to capture detailed visual and textual semantics essential for supporting instruction reasoning. To mitigate the semantic loss, we further introduce a Cross-Modal Enhancer (CME) that enables rich interactions between the fine-grained cues and MLLM-derived features. Extensive experiments demonstrate that ReasonBrain consistently outperforms state-of-the-art baselines on reasoning scenarios while exhibiting strong zero-shot generalization to conventional IIE tasks. Our dataset and code will be released publicly.

Deep Learning · Generative Models and Autoencoders

Kaizhen Zhu, Mokai Pan, Zhechuan Yu, Jingya Wang, Jingyi Yu, Ye Shi

Diffusion Bridge and Flow Matching have both demonstrated compelling empirical performance in transformation between arbitrary distributions. However, there remains confusion about which approach is generally preferable, and the substantial discrepancies in their modeling assumptions and practical implementations have hindered a unified theoretical account of their relative merits. We have, for the first time, provided a unified theoretical and experimental validation of these two models. We recast their frameworks through the lens of Stochastic Optimal Control and prove that the cost function of the Diffusion Bridge is lower, guiding the system toward more stable and natural trajectories. Simultaneously, from the perspective of Optimal Transport, interpolation coefficients $t$ and $1-t$ of Flow Matching become increasingly ineffective when the training data size is reduced. To corroborate these theoretical claims, we propose a novel, powerful architecture for Diffusion Bridge built on a latent Transformer, and implement a Flow Matching model with the same structure to enable a fair performance comparison in various experiments. Comprehensive experiments are conducted across Image Restoration, Translation, and Style Transfer tasks, systematically varying both the distributional discrepancy (different difficulty) and the training data size. Extensive empirical results align perfectly with our theoretical predictions and allow us to delineate the respective advantages and disadvantages of these two models. Our code is available at \url{https://anonymous.4open.science/r/DBFM-3E8E/}.

Theory · Probabilistic Methods

Dario Rancati, Jan Maas, Francesco Locatello

Diffusion-based models on continuous spaces have seen substantial recent progress through the mathematical framework of gradient flows, leveraging the Wasserstein-2 (${W}_2$) metric via the Jordan-Kinderlehrer-Otto (JKO) scheme. Despite the increasing popularity of diffusion models on discrete spaces using continuous-time Markov chains, a parallel theoretical framework based on gradient flows has remained elusive due to intrinsic challenges in translating the ${W}_2$ distance directly into these settings. In this work, we propose the first successful approach addressing these challenges, leveraging an appropriate metric $W_K$ on the simplex of probability distributions, which enables us to interpret widely used discrete diffusion paths, such as the discrete heat equation, as gradient flows of specific free-energy functionals. Through this theoretical insight, we introduce a novel methodology for learning diffusion dynamics over discrete spaces, which recovers the underlying functional directly by leveraging first-order optimality conditions for the JKO scheme. The resulting method optimizes a simple quadratic loss, trains extremely fast, does not require individual sample trajectories, and only needs a numerical preprocessing computing $W_K$-geodesics. We validate our method through extensive numerical experiments on synthetic data, showing that we can recover the underlying functional for a variety of graph classes.

Theory · Optimization

Anming Gu, Juno Kim

The mean-field Langevin dynamics (MFLD) minimizes an entropy-regularized nonlinear convex functional on the Wasserstein space over $\mathbb{R}^d$, and has gained attention recently as a model for the gradient descent dynamics of interacting particle systems such as infinite-width two-layer neural networks. However, many problems of interest have constrained domains, which are not solved by existing mean-field algorithms due to the global diffusion term. We study the optimization of probability measures constrained to a convex subset of $\mathbb{R}^d$ by proposing the \emph{mirror mean-field Langevin dynamics} (MMFLD), an extension of MFLD to the mirror Langevin framework. We obtain linear convergence guarantees for the continuous MMFLD via a uniform log-Sobolev inequality, and uniform-in-time propagation of chaos results for its time- and particle-discretized counterpart.

Deep Learning · Generative Models and Autoencoders

Bo Yin, Xiaobin Hu, Xingyu Zhou, Yu HE, Peng-Tao Jiang, Yue Liao, Junwei Zhu, Jiangning Zhang, Ying Tai, Shuicheng YAN

Diffusion models have achieved remarkable success in generative modeling, yet how to effectively adapting large pretrained models to new tasks remains challenging. We revisit the reconstruction behavior of diffusion models during denoising to unveil the underlying frequency–energy mechanism governing this process. Building upon this observation, we propose \textbf{FeRA}, a frequency-driven fine-tuning framework that aligns parameter updates with the intrinsic frequency–energy progression of diffusion. FeRA establishes a comprehensive frequency–energy framework for effective diffusion adaptation fine-tuning, comprising three synergistic components: \textit{(i)} a compact frequency–energy indicator that characterizes the latent’s bandwise energy distribution, \textit{(ii)} a soft frequency router that adaptively fuses multiple frequency-specific adapter experts, and \textit{(iii)} a frequency–energy consistency regularization that stabilizes diffusion optimization and ensures coherent adaptation across bands. Routing operates in both training and inference, with inference-time routing dynamically determined by the latent frequency energy. It integrates seamlessly with adapter-based tuning schemes and generalizes well across diffusion backbones and resolutions. By aligning adaptation with the frequency–energy mechanism, \textbf{FeRA} provides a simple, stable, and compatible paradigm for effective and robust diffusion model adaptation. Codes will be made publicly available.

General Machine Learning · Scalable Algorithms

Celine Lee, Jing Nathan Yan, Chen Liang, Jiaxin Shi, Yin Zhang, Jeremiah Zhe Liu, Pengcheng Yin, Ed Chi, Fernando Pereira, Derek Cheng 等

Modern language models typically rely on two design choices: subword tokenization and autoregressive (AR) ordering. To achieve more universal modeling, the field is advancing toward byte-level modeling to bypass domain-specific vocabularies and masked diffusion models (MDM) to enable parallel non-sequential generation. Intuitively, the intersection of these paradigms represents a generative ideal: a modality-agnostic system capable of fine-grained any-order generation. However, the computational interaction between these granular representations and non-sequential objectives remains under-explored. In this work, we investigate the viability of this combination through a compute-matched scaling study. We observe a structural dichotomy: AR models on bytes effectively amortize the cost of tokenization, naturally rediscovering sub-word segmentation at scale. In contrast, byte-level MDMs suffer a non-convergent efficiency collapse. We attribute this disparity to the masking objective, which shatters the local contiguity required to resolve sub-word semantics from bytes, whereas AR's stable causal history preserves these essential local dependencies. Our findings inform the community of a critical efficiency tradeoff, suggesting that future modality-agnostic designs should address this context fragility to maintain efficient scaling.