论文检索

Deep Learning · Generative Models and Autoencoders

Piotr Wójcik, Maksym Petrenko, Wojciech Gromski, Przemysław Spurek, Maciej Zieba

Recent advances in large-scale diffusion models have intensified concerns about their potential misuse, particularly in generating realistic yet harmful or socially disruptive content. This challenge has spurred growing interest in effective machine unlearning, the process of selectively removing specific knowledge or concepts from a model without compromising its overall generative capabilities. Among various approaches, Low-Rank Adaptation (LoRA) has emerged as an effective and efficient method for fine-tuning models toward targeted unlearning. However, LoRA-based methods often exhibit limited adaptability to concept semantics and struggle to balance removing closely related concepts with maintaining generalization across broader meanings. Moreover, these methods face scalability challenges when multiple concepts must be erased simultaneously. To address these limitations, we introduce UnHype, a framework that incorporates hypernetworks into single- and multi-concept LoRA training. The proposed architecture can be directly plugged into Stable Diffusion as well as modern flow-based text-to-image models, where it demonstrates stable training behavior and effective concept control. During inference, the hypernetwork dynamically generates adaptive LoRA weights based on the CLIP embedding, enabling more context-aware, scalable unlearning. We evaluate UnHype across several challenging tasks, including object erasure, celebrity erasure, and explicit content removal, demonstrating its effectiveness and versatility.

Deep Learning · Generative Models and Autoencoders

Parsa Rahimi, Sébastien Marcel

Synthetic data generation is increasingly used in machine learning for **training and data augmentation**. Yet, many current strategies rely on external foundation models or datasets, which can be restricted by policy or legal constraints, especially for sensitive modalities such as human face images and videos. We propose **ScoreMix**, a **self-contained data augmentation** method to boost recognition performance by leveraging score compositionality in class-conditioned diffusion models. ScoreMix mixes class-conditioned scores along reverse diffusion trajectories, yielding domain-specific hard augmentations without external resources. We systematically study class-selection strategies and find that mixing classes that are distant in the discriminator embedding space yields larger gains, providing **up to 3\% additional average improvement across benchmarks** over proximity-based selection. Interestingly, we observe that learned condition and embedding spaces are largely uncorrelated under standard alignment metrics, and that condition-space distances are weakly correlated to downstream gains. Across **8 public face recognition benchmarks**, ScoreMix improves accuracy by **up to 7 percentage points** without hyperparameter search, highlighting robustness and practicality. Code and dataset will be made publicly available.

Deep Learning · Generative Models and Autoencoders

Sizhuang He, Yangtian Zhang, Shiyang Zhang, David van Dijk

The finite symmetric group $S_n$ provides a natural domain for permutations, yet learning probability distributions on $S_n$ is challenging due to its factorially growing size and discrete, non-Euclidean structure. Recent permutation diffusion methods define forward noising via shuffle-based random walks (e.g., riffle shuffles) and learn reverse transitions with Plackett–Luce (PL) variants, but the resulting trajectories can be abrupt and increasingly hard to denoise as $n$ grows. We propose *Soft-Rank Diffusion*, a discrete diffusion framework that replaces shuffle-based corruption with a structured soft-rank forward process: we lift permutations to a continuous latent representation of order by relaxing discrete ranks into soft ranks, yielding smoother and more tractable trajectories. For the reverse process, we introduce *contextualized generalized Plackett–Luce (cGPL)* denoisers that generalize prior PL-style parameterizations and improve expressivity for sequential decision structures. Experiments on sorting and combinatorial optimization benchmarks show that Soft-Rank Diffusion consistently outperforms prior diffusion baselines, with particularly strong gains in long-sequence and intrinsically sequential settings.

Deep Learning · Generative Models and Autoencoders

Yuwei Niu, Munan Ning, Mengren Zheng, Weiyang Jin, Bin Lin, Peng Jin, Jiaqi Liao, Chaoran Feng, Fanqing Meng, Kun-Peng Ning 等

Text-to-Image (T2I) models are capable of generating high-quality artistic creations and visual content. However, existing research and evaluation standards predominantly focus on image realism and shallow text-image alignment, lacking a comprehensive assessment of complex semantic understanding and world knowledge integration in text-to-image generation. To address this challenge, we propose WISE, the first benchmark specifically designed for World Knowledge-Informed Semantic Evaluation. WISE moves beyond simple word-pixel mapping by challenging models with 1000 meticulously crafted prompts across 25 subdomains in cultural common sense, spatio-temporal reasoning, and natural science. To overcome the limitations of traditional CLIP metric, we introduce WiScore, a novel quantitative metric for assessing knowledge-image alignment. Through comprehensive testing of 20 models (10 dedicated T2I models and 10 unified multimodal models) using 1,000 structured prompts spanning 25 subdomains, our findings reveal significant limitations in their ability to effectively integrate and apply world knowledge during image generation, highlighting critical pathways for enhancing knowledge incorporation and application in next-generation T2I models. Code and data will be available.

Deep Learning · Generative Models and Autoencoders

Donghao Zhou, Guisheng Liu, Hao Yang, Jiatong Li, Jingyu Lin, Xiaohu Huang, Yichen Liu, Xin Gao, Cunjian Chen, Shilei Wen 等

In this work, we study **Human-Object Interaction Video Generation (HOIVG)**, which aims to synthesize high-quality HOI videos via text, reference image, audio, and pose conditions. To address the challenges of harmonious multimodal injection and heterogeneous data utility, we present **OmniShow**, an end-to-end framework tailored for HOIVG. We introduce *Unified Channel-wise Conditioning* to efficiently inject image and pose cues, *Gated Local-Context Attention* to ensure precise audio-visual synchronization, and a *Decoupled-then-Joint Training strategy* to effectively harness heterogeneous data. Extensive experiments on the proposed *HOIVG-Bench* demonstrate that OmniShow achieves state-of-the-art performance.

Deep Learning · Generative Models and Autoencoders

Tatiana Gaintseva, Andrew Stepanov, Ziquan Liu, Martin Benning, Gregory Slabaugh, Jiankang Deng, Ismail Elezi

Steering intermediate representations has emerged as a powerful strategy for controlling generative models. However, despite its empirical success, it currently lacks a comprehensive theoretical framework. In this paper, we bridge this gap by formalizing the theory of concept steering. First, we establish a link between steering and affine concept erasure, proving that the standard approach for removing unwanted behaviors is a special case of LEACE (a closed-form method for affine erasure). Next, we formulate a principled theoretical framework for concept switching, LEACE-Switch, and characterize the assumptions under which it provides an optimal affine solution. Building on this analysis, we then introduce MidSteer (Minimal Disturbance concept Steering), a more general affine framework for concept manipulation that relaxes these assumptions and enables directed, minimal-disturbance transformations. We empirically demonstrate that MidSteer performs favorably across a range of tasks, modalities, and architectures, including vision diffusion models and large language models.

Applications · Health / Medicine

Dexiong Chen, Andrei Manolache, Mathias Niepert, Karsten Borgwardt

Classifying protein topology is essential for deciphering biological function, but progress is held back by the lack of large-scale benchmarks that avoid duplicates and by models that do not scale well. We introduce TEDBench, a large-scale, non-redundant benchmark for protein fold classification constructed from the Encyclopedia of Domains (TED) and Foldseek-clustered AlphaFold structures. We show that on TEDBench, current protein representation learning methods either require very large models or fail to deliver strong performance. To address this challenge, we propose Masked Invariant Autoencoders (MiAE), a self-supervised framework for protein structure representation learning. MiAE uses an extremely high masking ratio of up to 90% with an $\mathrm{SE(3)}$-invariant encoder and a lightweight decoder that reconstructs backbone coordinates from the latent representation and mask tokens. MiAE scales well and outperforms supervised counterparts and state-of-the-art baselines on TEDBench, establishing a strong recipe for protein fold classification. To test transfer beyond AlphaFold structures, we further benchmark on a curated dataset from experimental structures of CATH 4.4. We will release TEDBench and model checkpoints.

Deep Learning · Generative Models and Autoencoders

Raghav Kansal, David Crair, Nghia Nguyen, Scott Pope, Bradley Parry

Flow matching (FM) has emerged as a powerful framework for learning dynamic transport maps between two empirical distributions. However, less explored is the setting with intermediate observed marginals that can help constrain the flows between the endpoints. This "multimarginal" regime is central to modeling temporal evolution in dynamical systems in many scientific domains that can sample sequential distributions, such as biology and meteorology. We tackle this problem with a novel approach that leverages the connection between FM and dynamic optimal transport (OT), introducing time-dependent potential terms in the dynamic OT action that can steer the flow towards the intermediate marginals. By extending the conditional FM learning target to incorporate these potentials, we derive an efficient, simulation-free algorithm for multimarginal FM that offers considerable flexibility in the spatiotemporal dynamics of the learned flows. We demonstrate state-of-the-art performance of OT-potential FM (OTP-FM) on diverse scientific datasets.

Applications · Neuroscience, Cognitive Science

KIEREN YU, Ziyang Liu, Chang Huang, Kaishun WU

EEG foundation models aim to learn transferable representations, yet EEG recordings are dominated by high-frequency noise and large cross-subject variability. Existing pretraining strategies such as masked autoencoding or autoregressive modeling often treat waveform reconstruction as the learning signal, making the objective sensitive to stochastic fluctuations rather than consistent neurophysiological structure. To address this overlap, we propose \textbf{PATCHCODE}, a region-aware discrete predictive learning framework that keeps the encoder input continuous while introducing region-aware discrete codes as stable supervision targets. We pretrain a masked predictive encoder on continuous EEG patches with dual-granularity learning: it predicts missing patch-level representations to preserve fine spatiotemporal structure, while aligning them to discretized code targets from a frozen tokenizer to anchor robust semantics. Extensive Experiments across ten downstream datasets spanning emotion recognition, motor imagery, sleep staging, and seizure detection demonstrate that PATCHCODE achieves competitive performance compared to state-of-the-art baselines, with notable gains in data efficiency under limited labels. Our code is available at https://anonymous.4open.science/r/PATCHCODE-323D/.

Deep Learning · Generative Models and Autoencoders

Yuqi Wang, Jianwei Niu, Xinghao Wu, Xuefeng Liu, Xin Hao

One emerging approach to mitigating data heterogeneity in Federated Learning (FL) is to employ diffusion models to generate synthetic data for clients, thereby aligning local data distributions with the global distribution. Prior work has primarily focused on balance-oriented augmentation, which assumes a balanced global class distribution and thus generates samples of rare classes to rebalance each client's local dataset. However, in practice, global data distributions are often inherently imbalanced. Moreover, privacy constraints in FL hinder the server’s ability to accurately estimate the global distribution, rendering balance-oriented augmentation suboptimal. This raises a key, underexplored challenge: How can synthetic data be generated and selected to align local distributions with the true, yet unknown, global distribution? Our key insight is that a model’s performance implicitly reflects the data distribution it has been trained on. Based on this observation, we use the performance discrepancy between local and global models to identify the regions where each client’s local dataset is lacking, and generate corresponding samples for clients. Furthermore, we adapt the diffusion model via preference optimization, enabling it to generate data that better aligns with the true global distribution. Extensive experiments on multiple benchmarks demonstrate that FedPDG outperforms state-of-the-art methods, achieving up to 3.82\% improvement.

Deep Learning · Generative Models and Autoencoders

Lei Tong, Zhihua Liu, Chaochao Lu, Dino Oglic, Tom Diethe, Philip Teare, Sotirios Tsaftaris, Chen Jin

We present Causal-Adapter, a modular framework that adapts frozen text-to-image diffusion backbones for counterfactual image generation. Our method enables causal interventions on target attributes while preserving all other aspects of the image, including the core identity. In contrast to prior approaches that rely on prompt engineering without explicit causal structure, Causal-Adapter leverages structural causal modeling augmented with two attribute regularization strategies: prompt-aligned injection, which aligns causal attributes with textual embeddings for precise semantic control, and a conditioned token contrastive loss to disentangle attribute factors and reduce spurious correlations. Causal-Adapter achieves state-of-the-art results on synthetic and real-world datasets, outperforming other baselines in effectiveness, composition, realism, and minimality. These results demonstrate that the approach enables efficient, robust, and generalizable counterfactual image editing with faithful attribute modification and strong preservation of core identity.

Sijie Zhao, Feng Liu, Xueliang Zhang, Hao Chen, Tao Han, JUNCHAO GONG, Ran Tao, Pengfeng Xiao, Xinyu Gu, LEI BAI

The increasing impact of climate change and extreme weather events has spurred growing interest in deep learning for weather research. However, existing studies often rely on weather data in pixel space, which presents several challenges such as smooth outputs in model outputs, limited applicability to a single pressure-variable subset (PVS), and high data storage and computational costs. To address these challenges, we propose a novel Weather Latent Autoencoder (WLA) that transforms weather data from pixel space to latent space, enabling efficient data representation. By decoupling weather reconstruction from downstream tasks, WLA improves the accuracy and sharpness of weather task model results. The incorporated Pressure-Variable Unified Module transforms multiple PVS into a unified representation, enhancing the adaptability of the model in multiple weather scenarios. Furthermore, weather tasks can be performed in a low-storage latent space of WLA rather than a high-storage pixel space, thus significantly reducing data storage and computational costs. Through extensive experimentation, we demonstrate its superior compression and reconstruction performance, enabling the creation of the ERA5-Latent dataset with unified representations of multiple PVS from ERA5 data. The compressed full PVS in the ERA5-Latent dataset reduces the original 244.34 TB of data to 0.43 TB. The downstream task further demonstrates that task models can apply to multiple PVS with low data costs in latent space and achieve superior performance compared to models in pixel space.

Deep Learning · Theory

Vicente Mendes, Lorenzo Bardone, Cédric Koller, Jorge Medina Moreira, Vittorio Erba, Emanuele Troiani, Lenka Zdeborova

Many real-world datasets contain hidden structure that cannot be detected by simple linear correlations between input features. For example, latent factors may influence the data in a coordinated way, even though their effect is invisible to covariance-based methods such as PCA. In practice, nonlinear neural networks often succeed in extracting such hidden structure in unsupervised and self-supervised learning. However, constructing a minimal high-dimensional model where this advantage can be rigorously analyzed has remained an open theoretical challenge. We introduce a tractable high-dimensional spiked model with two latent factors: one visible to covariance, and one statistically dependent yet uncorrelated, appearing only in higher-order moments. PCA and linear autoencoders fail to recover the latter, while a minimal nonlinear autoencoder provably extracts both. We analyze both the population risk, and empirical risk minimization. Our model also provides a tractable example where self-supervised test loss is poorly aligned with representation quality: nonlinear autoencoders recover latent structure that linear methods miss, even though their reconstruction loss is higher.

Deep Learning · Generative Models and Autoencoders

Hongyu Zhang, Yufan Deng, Shenghai Yuan, Yian Zhao, Peng Jin, Xuehan Hou, Chang Liu, Jie Chen

In the domain of text-to-video (T2V) generation, reliably synthesizing compositional content involving multiple subjects with intricate relations is still underexplored. The main challenges are twofold: 1) Subject presence, where not all subjects can be presented in the video; 2) Inter-subject relations, where the interaction and spatial relationship between subjects are misaligned. Existing methods adopt techniques, such as inference-time latent optimization or layout control, which fail to address both issues simultaneously. To tackle these problems, we propose \textbf{Comp-Attn}, a composition-aware cross-attention variant that follows a \textbf{``Present-and-Align” }paradigm: it decouples the two challenges by enforcing subject presence at the condition level and achieving relational alignment at the attention-distribution level. Specifically, 1) We introduce Subject-aware Condition Interpolation (SCI) to reinforce subject-specific conditions and ensure each subject's presence; 2) We propose Layout-forcing Attention Modulation (LAM), which dynamically enforces the attention distribution to align with the relational layout of multiple subjects. Comp-Attn can be seamlessly integrated into various T2V baselines in a training-free manner, boosting T2V-CompBench scores by 15.7\% and 11.7\% on Wan2.1-T2V-14B and Wan2.2-T2V-A14B with only a 5\% increase in inference time. Meanwhile, it also achieves strong performance on VBench and T2I-CompBench, demonstrating its scalability in general video generation and compositional text-to-image (T2I) tasks.

Applications · Chemistry, Physics, and Earth Sciences

Shrimon Mukherjee, KISHALAY DAS, Partha Basuchowdhuri, Pawan Goyal, Niloy Ganguly

Fast and accurate prediction of crystal properties is a central challenge in new materials design. Graph Neural Networks and transformer-based models have emerged as powerful tools for this task due to their ability to encode the local structural environment of atoms within a crystal. However, these models are data hungry and in practice labeled data for crystal properties are very scarce. Pretrain–finetuning strategies, particularly those based on diffusion models, have shown promise in addressing these limitations. In this work, we introduce a novel latent-diffusion based pretraining framework CrysLDNet designed to mitigate the data scarcity issue. Our approach integrates a Variational Autoencoder (VAE) with a diffusion model during the pretraining stage. The VAE encoder maps 3D crystal structures into a smooth latent space, within which the diffusion process is applied. This latent diffusion pretraining enables the graph encoder to effectively capture structural and chemical semantics from large-scale unlabeled data, which can then be finetuned for specific property prediction tasks. Comprehensive experiments on popular DFT datasets for property prediction reveal that CrysLDNet significantly outperforms both training-from-scratch and pretrained baselines, with improvements of 4.26% and 4.90% on JARVIS and MP datasets. Additionally, the learned representations remain robust under sparse data conditions and are expressive enough to correct DFT errors when finetuned with limited experimental data.

Deep Learning · Generative Models and Autoencoders

Seokwon Yoon, Youngbin Choi, Seunghyuk Cho, Seungbeom Lee, MoonJeong Park, Dongwoo Kim

Generative Flow Networks (GFlowNets) learn to sample diverse candidates in proportion to a reward function, making them well-suited for scientific discovery, where exploring multiple promising solutions is crucial. Further extending GFlowNets to multi-objective settings has attracted growing interest since real-world applications often involve multiple, conflicting objectives. However, existing approaches require additional training for each set of objectives, limiting their applicability and incurring substantial computational overhead. We propose a training-free mixing policy that composes pre-trained GFlowNets at inference time, enabling rapid adaptation without finetuning or retraining. Importantly, our framework is flexible, capable of handling diverse reward combinations ranging from linear scalarization to complex non-linear logical operators, which are often handled separately in previous literature. We prove that our method exactly recovers the target distribution for linear scalarization and quantify the approximation quality for nonlinear operators through a distortion factor. Experiments on a synthetic 2D grid and real-world molecule-generation tasks demonstrate that our approach achieves performance comparable to baselines that require additional training.

Deep Learning · Generative Models and Autoencoders

Sven Gutjahr, Riccardo De Santi, Luca Schaufelberger, Kjell Jorner, Andreas Krause

Adapting generative foundation models, in particular diffusion and flow models, to optimize given reward functions (e.g., binding affinity) while satisfying constraints (e.g., molecular synthesizability) is fundamental for their adoption in real-world scientific discovery applications such as molecular design or protein engineering. While recent works have introduced scalable methods for reward-guided fine-tuning of such models via reinforcement learning and control schemes, it remains an open problem how to algorithmically trade-off reward maximization and constraint satisfaction in a reliable and predictable manner. Motivated by this challenge, we first present a rigorous framework for *Constrained Generative Optimization*, which brings an optimization viewpoint to the introduced adaptation problem and retrieves the relevant task of constrained generation as a sub-case. Then, we introduce Constrained Flow Optimization (CFO), an algorithm that automatically and provably balances reward maximization and constraint satisfaction by reducing the original problem to progressive fine-tuning via established, scalable methods. We provide convergence guarantees for constrained generative optimization and constrained generation via CFO. Ultimately, we present an experimental evaluation of CFO on both synthetic, yet illustrative, settings, and a molecular design task.

Deep Learning · Generative Models and Autoencoders

Xiaochao Deng, Jie Chen, Xiaogang Deng

Accurately modeling the full distributions of possible states is crucial for understanding statistical properties and enabling reliable predictions in complex, unsteady physical systems. Recently, diffusion models and flow matching have shown promise in these tasks. However, they remain limited in uncovering the general principles of systems from multiple short trajectories across condition space. In addition, they exhibit inferior adaptability to large irregular geometries, particularly in regions with sharp gradients. In this paper, we propose a condition-aware graph flow matching (CGFM) method that combines condition-aware flow matching with a hierarchical graph structure to learn the full distributions of physical systems from incomplete training data. Specifically, CGFM constructs a flow enabling smooth interpolation across physical conditions and parameterizes the graph-conditioned vector field through HieraGraphNet. HieraGraphNet performs message passing across multilevel graphs to capture multi-scale dynamics and facilitate long-range information interactions in physical systems. Moreover, we introduce a topology- and geometry-aware graph coarsening scheme that incorporates topological connectivity and local geometric density to construct reliable coarse graphs. We validate the effectiveness of CGFM on three canonical scenarios across both 2D and 3D dynamics, which demonstrate its superior performance compared with that of state-of-the-art baselines.

Deep Learning · Generative Models and Autoencoders

Xiang Li, Dianbo Liu, Kenji Kawaguchi

Despite the remarkable fidelity of generative models, they frequently suffer from mode collapse. Existing strategies for enhancing diversity predominantly focus on intervening during the generation trajectory. We identify a critical oversight that the standard Gaussian initialization often causes trajectories to collapse into dominant modes because it is agnostic to the guidance potential landscape. In this work, we formulate selecting the initial noise from a *guidance potential posterior*, which effectively re-weights the prior towards diversity-rich regions. To sample from this distribution efficiently, we introduce *Diversity-inducing Initialization* (DivIn), which leverages Langevin dynamics to actively navigate the initialization landscape, steering initial noise away from collapsing regions while anchoring them to the valid data manifold. Our method serves as an inference-time diversity enhancement compatible with both diffusion and flow matching models. Extensive experiments show that DivIn exhibits a superior performance in both class-to-image and text-to-image scenarios. Furthermore, we highlight that as DivIn is orthogonal to trajectory-based methods, combining them significantly expands the diversity-quality Pareto frontier beyond what either achieves in isolation.

Deep Learning · Generative Models and Autoencoders

Shigui Li, Delu Zeng

A fundamental tension exists in the large-step inference of diffusion models via their deterministic probability flow ordinary differential equation (PF-ODE) paths, which we formally identify as the *contractivity trap*: efficient inference favors large step sizes, while stable convergence requires strong contractivity that limits expressiveness. To address this, we propose SteinDiff, an inference-time stabilization framework based on reference-free Stein corrections. Specifically, SteinDiff introduces a geometry-aware correction mechanism that stabilizes PF-ODE inference trajectories. To this end, we contribute closed-form correction estimators via Stein's identity in the continuous-time setting, enabling the method to adapt to local data geometry. We theoretically demonstrate that SteinDiff reduces integration error even when contractivity is violated and establishes its robustness against discretization-induced distributional shifts. Our analysis further reveals that these corrections act as persistent geometric anchors, providing new insights into the stability of SOTA EDM parameterizations. Extensive experiments demonstrate that SteinDiff significantly mitigates mode collapse and improves generative quality in large-step inference.