论文检索

Applications · Computer Vision

Yihang Liu, Ying Wen, Jiaxiong Yang, Longzhen Yang, Lianghua He, Heng Tao Shen

Medical foundation models (MFMs) aim to learn universal representations from multimodal medical images that can generalize effectively to diverse downstream clinical tasks. However, most existing MFMs suffer from information ambiguity that blend multimodal representations in a single embedding space, leading to the degradation of modality specificity and diversity. In this paper, we propose M-IDoL, a self-supervised ***M***FM that introduces ***I***nformation ***D***ecomposition for multim***o***dal representation ***L***earning via two objectives: i) maximize inter-modality entropy by dispersing multimodal representation into separable Mixture-of-Experts (MoE) subspaces to achieve representation specificity across modalities; and ii) minimize intra-modality uncertainty by performing fine-grained semantic discrimination within each MoE subspace to enrich representation diversity per modality. By pre-training on 1.15 million medical images, M-IDoL i) delivers superior generalization across 21 downstream clinical tasks, outperforming 20 foundation models on five imaging modalities (e.g., X-ray, fundus, OCT, dermoscopy and pathology), and ii) learns modality-specific and diverse representations, showing clearer separation of feature cluster across modalities and finer-grained feature discrimination within each modality.

Deep Learning · Self-Supervised Learning

Wenjie Yang, Hang Yu, Yuyu Guo, Peng Di

In this work, we address the critical yet underexplored challenge of symmetric multimodal-to-multimodal (MM2MM) retrieval, where queries and contexts are interchangeable. Existing universal multimodal retrieval works struggle with this task, as they are constrained by the labeled asymmetric datasets used. We produce SOLAR (Self-supervised jOint LeArning for symmetric multimodal Retrieval), a novel two-stage self-supervised framework that leverages readily available unlabeled web-scale image-text pairs. Based on the observation that both semantic alignment and discrepancies exist between two modalities, in the first stage, we learn the intersection mask of image-text pair, allowing us to align intersection while preserving semantic of difference. In the second stage, the learned mask is further utilized to construct positive and hardnegative samples via masking different parts of image/text, which enable us to conduct self-supervised multimodal embedding learning. Complementing this framework, we present a new benchmark featuring high-quality human-verified positive and hard-negative pairs to evaluate symmetric MM2MM retrieval under realistic conditions, as well as the corresponding pipeline. Extensive experiments against ten SOTA methods show SOLAR surpasses the strongest supervised VLM by 7.08 points on this benchmark, with over 50x fewer model parameters and a 5x smaller embedding dimension. Code will be available upon publication.

Deep Learning · Graph Neural Networks

Ling Ding, Zhizhi Yu, Cuiying Huo

Deep attribute graph clustering aims to learn discriminative node representations by leveraging both node attributes and graph topology to partition nodes into distinct clusters. Although substantial progress has been made in attribute-graph clustering in recent years, two key challenges remain: noisy edges in the original adjacency matrix degrade the quality of information propagation, and redundant feature information across different feature views hampers the learning of discriminative representations. To address these issues, we propose a self-supervised attribute graph clustering method based on topological reconstruction and correlation decorrelation. First, we reconstruct the graph topology by computing intersections between k-nearest neighbors and the original adjacency relationships, while simultaneously leveraging global semantic information from K-means clustering to filter out noisy nodes. This reconstructed topology effectively mitigates information redundancy during feature aggregation in Graph Neural Networks. Second, unlike existing augmentation-based contrastive methods, we treat the feature representations from an auto-encoder (AE) and a graph auto-encoder (GAE) as two complementary natural views. We then apply mutual information minimization and a decorrelation constraint to suppress redundant information between views, yielding more discriminative node representations. Extensive experiments on four widely-used graph datasets—ACM, DBLP, CITE, and AMAP—demonstrate that our method consistently outperforms six state-of-the-art baselines.

Applications · Chemistry, Physics, and Earth Sciences

Dexiong Chen, Andrei Manolache, Mathias Niepert, Karsten Borgwardt

Classifying protein topology is essential for deciphering biological function, but progress is held back by the lack of large-scale benchmarks that avoid duplicates and by models that do not scale well. We introduce TEDBench, a large-scale, non-redundant benchmark for protein fold classification constructed from the Encyclopedia of Domains (TED) and Foldseek-clustered AlphaFold structures. We show that on TEDBench, current protein representation learning methods either require very large models or fail to deliver strong performance. To address this challenge, we propose Masked Invariant Autoencoders (MiAE), a self-supervised framework for protein structure representation learning. MiAE uses an extremely high masking ratio of up to 90% with an $\mathrm{SE(3)}$-invariant encoder and a lightweight decoder that reconstructs backbone coordinates from the latent representation and mask tokens. MiAE scales well and outperforms supervised counterparts and state-of-the-art baselines on TEDBench, establishing a strong recipe for protein fold classification. To test transfer beyond AlphaFold structures, we further benchmark on a curated dataset from experimental structures of CATH 4.4. We will release TEDBench and model checkpoints.

Applications · Neuroscience, Cognitive Science

Wei Wang, Fang He, Yifan Li, Wanying Qu, Yawei Li, Quanying Liu, Yanwei Fu

Existing EEG models are limited by electrode heterogeneity and rigid "channel-first" architectures that treat sensors as independent features. We propose Brain Signal Rendering (BSR), which reinterprets EEG as a physical projection of neural activity and transforms raw signals into geometry-aware Spectrum Videos. By utilizing VideoMAE for self-supervised pre-training, BSR learns robust, layout-agnostic spatiotemporal representations that preserve neural topology. We further introduce subject-level few-shot learning and cross-montage fine-tuning to rigorously evaluate generalization across subjects and electrode configurations. Experiments show that VideoMAE model integrated with the BSR framework significantly outperforms state-of-the-art spectrum based methods, providing a scalable and data-efficient foundation for generalizable EEG modeling.

Deep Learning · Self-Supervised Learning

Jiwan Chung, Seon Joo Kim

A common assumption in representation learning is that globally well-distributed embeddings support robust and generalizable representations. This focus has shaped both training objectives and evaluation protocols, implicitly treating global geometry as a proxy for representational competence. While global geometry effectively encodes which elements are present, it is often insensitive to how they are composed. We investigate this limitation by testing the ability of geometric metrics to predict compositional binding across 21 vision encoders. We find that standard geometry-based statistics exhibit near-zero correlation with compositional binding. In contrast, functional sensitivity, as measured by the input-output Jacobian, reliably tracks this capability. We further provide an analytic account showing that this disparity arises from objective design, as existing losses explicitly constrain embedding geometry but leave the local input-output mapping unconstrained. These results suggest that global embedding geometry captures only a partial view of representational competence and establish functional sensitivity as a critical complementary axis for modeling composite structure.

Probabilistic Methods · Variational Inference

Pedro Dall’Antonia, Tiago Silva, Daniel Csillag, Salem Lahlou, Diego Mesquita

Generative Flow Networks (GFlowNets) are a flexible family of amortized samplers trained to generate discrete and compositional objects with probability proportional to a reward function. To this end, they learn a policy function over an intractably large state graph by minimizing a stochastic objective over sampled trajectories. However, learning efficiency is constrained by the model’s ability to rapidly explore diverse high-probability regions during training. To mitigate this issue, recent works have focused on incentivizing the exploration of unvisited and valuable states via curiosity-driven search and self-supervised random network distillation, which tend to waste samples on already well-approximated regions of the state space. In this context, we propose *Adaptive Complementary Exploration* (ACE), a principled algorithm for the effective exploration of novel and high-probability regions when learning GFlowNets. To achieve this, ACE introduces an *exploration* GFlowNet explicitly trained to search for high-reward states in regions underexplored by the *canonical* GFlowNet, which learns to sample from the target distribution. Through extensive experiments, we show that ACE consistently and significantly improves upon prior work in terms of approximation accuracy to the target distribution and discovery rate of diverse high-reward states.

Deep Learning · Generative Models and Autoencoders

Brett Levac, Jon Tamir, Marcelo Pereyra, Julián Tachella

Diffusion models (DMs) are a powerful framework for image generation and restoration. However, existing DMs are primarily trained in a supervised manner by using a large corpus of clean images. This poses fundamental challenges in many real-world scenarios, where acquiring noise-free data is hard or infeasible. While some methods are capable of training DMs using noisy data, they are effective only when the amount of noise is very mild or when additional noise-free data is available. In addition, existing methods for training DMs from incomplete measurements require access to multiple complementary acquisition processes, a significant practical limitation. Here we introduce the first approach for learning DMs for image restoration using only noisy measurement data from a single operator. First, we show that DMs, and more broadly minimum mean squared error denoisers, exhibit a weak form of scale equivariance linking rescaling in signal amplitude to changes in noise intensity. We then leverage this theoretical insight to develop a denoising score-matching strategy that generalizes robustly to noise levels below the training data, thereby enabling the learning of DMs from noisy measurements. For problems involving measurements both noisy and incomplete, we integrate our method with equivariant imaging, a complementary self-supervised learning framework that exploits the inherent invariants of imaging problems. This allows training DMs for image restoration from single-operator noisy measurements. We validate the effectiveness of our approach through extensive experiments on image denoising, demosaicing, inpainting, and MRI reconstruction along with comparisons with the state of the art.

Deep Learning · Self-Supervised Learning

Megi Dervishi, Mathurin VIDEAU, Yann LeCun

While decoders have rapidly scaled, encoders have remained largely unchanged since BERT. We examine this disparity by revisiting evaluation through the lens of finetuning under frozen backbone and linear probing. As models scale, their representations become increasingly unexploitable by frozen probes, despite improved perplexity. This suggests a misalignment between direct token prediction and the learning of rich, versatile, easily extractable representations. Hence, we propose CrossBERT, a two-part architecture that separates the learning of high-quality encoded representations from the rigid grounding of token reconstruction. This design further enables high masking ratios ($\ge 50\%$) and gradient collection over all token via a Complementary Masking Strategy, respectively increasing throughput by $1.5$-$2$× and sample efficiency by 2×. Overall, CrossBERT demonstrates monotonic scaling and superior performance on MTEB(eng, v2) and frozen GLUE benchmarks.

Deep Learning · Graph Neural Networks

Yuting Zhang, Zhitao Xiao, Zhitao Xiao, Lei Geng, Yanwei Pang, Xiao Wang

Self-supervised Continual Graph Learning (CGL) aims to successively learn from a graph sequence with different tasks without label supervision—a paradigm that has attracted widespread attention. Most existing self-supervised CGL methods rely on instance-level consistency objectives that enforce stability of individual node (or node-pair) embeddings. Due to optimizing nodes in isolation, these methods fail to maintain global relational structure, causing inter-node correspondences to progressively distort under continual learning. To this end, we propose a novel Structure-Aware Optimal Transport (SAOT) framework that explicitly captures and preserves relational structure within graph representations across sequential tasks. Specifically, SAOT leverages optimal transport theory to capture global inter-node correspondences, thereby facilitating and enhancing graph representation learning. Simultaneously, SAOT incorporates a cross-task knowledge distillation mechanism to preserve the previous structural knowledge. Extensive experiments on four CGL benchmark datasets demonstrate that SAOT outperforms existing self-supervised baselines. In particular, SAOT achieves significant performance gains, improving average accuracy by up to 6% on CoraFull-CL and over 15% on Products-CL compared with state-of-the-art methods in the Class-IL setting.

General Machine Learning · Transfer, Multitask and Meta-learning

Zimo Zhai, Manjie Xu, Wei Liang

Large language models have shown strong reasoning abilities and are increasingly explored as high-level coordinators for multi-agent systems. However, directly deploying LLMs for coordination remains challenging, as effective policies often fail to reliably emerge at the low-level control level, and inference costs limit scalability. We propose SynCoord (Synthetic Coordination Distillation), a self-supervised pipeline that distills task-level decision-making for cooperation from high-capacity reasoning models into lightweight agent policies. Our approach does not rely on explicit supervision or handcrafted coordination rules. Instead, we define a set of task-level tool interfaces that constrain LLM interaction and enable the collection of interaction trajectories, which are then used to train compact coordinated policies. This distillation process transfers coordination behaviors that are difficult to elicit through prompting alone, while substantially reducing inference overhead at execution time. We evaluate our method in the multi-agent cooperation benchmark Overcooked-AI with varying team sizes and environment layouts. Experimental results show that the distilled policies achieve success rates and efficiency comparable to reinforcement learning–based methods, while exhibiting fewer erroneous or redundant actions and generalizing across team sizes without retraining.

Applications · Robotics

Yan Zhang, Zheng WANG, Pengpeng Zeng, Xing Xu, Jingkuan Song, Heng Tao Shen

Visuo-tactile sensors have been widely adopted in robotic manipulation. However, inherent heterogeneity in sensor designs hinders the learning of unified tactile representations in cross-sensor scenarios. Existing methods that focus on reconstruction or task-specific supervision often fail to capture the common information between different tactile sensors, particularly in the presence of substantial sensor variations, resulting in limited generalization to unseen sensors. To address this, we propose Cross-Tactile Sensor Representation Learning (CTSRL), a unified framework for sensor-agnostic tactile representation learning. CTSRL introduces a Cross-Sensor Modulator (CSM) to eliminate sensor-specific biases and adopts a two-stage learning paradigm: (1) leveraging aligned synthetic data for cross-sensor self-supervised learning to extract shared latent representations across sensor domains; and (2) integrating real-world multimodal tactile data to bridge the sim-to-real semantic gap through cross-modal alignment, thereby enriching representations with fine-grained semantic attributes. Experimental results show that our method demonstrates strong multi-sensor generalization, significantly improving sensor-agnostic representation learning.

Deep Learning · Self-Supervised Learning

Heejeong Nam, Quentin Le Lidec, Lucas Maes, Yann LeCun, Randall Balestriero

World models require robust relational understanding to support prediction, reasoning, and control. While object-centric representations provide a useful abstraction, they are not sufficient to capture interaction-dependent dynamics. We therefore propose C-JEPA, a simple and flexible object-centric world model that extends masked joint embedding prediction from image patches to object-centric representations. By applying object-level masking that requires an object’s state to be inferred from other objects, C-JEPA induces latent interventions with counterfactual-like effects and prevents shortcut solutions, making interaction reasoning essential. Empirically, C-JEPA leads to consistent gains in visual question answering, with an absolute improvement of about 20\% in counterfactual reasoning over the same architecture without object-level masking. On agent control tasks, C-JEPA enables substantially more efficient planning by using only 1\% of the total latent input features required by patch-based world models, while achieving comparable performance. Finally, we provide a formal analysis demonstrating that object-level masking induces a causal inductive bias via latent interventions. Code will be available at *anonymous*.

Deep Learning · Other Representation Learning

Chen Huang, Xianhang Li, Vimal Thilak, Etai Littwin, Joshua M Susskind

Image-based Joint-Embedding Predictive Architecture (I-JEPA) offers a promising approach to visual self-supervised learning through masked feature prediction. However with the inherent visual uncertainty at masked positions, feature prediction remains challenging and may fail to learn semantic representations. In this work, we propose Text-Conditional JEPA (TC-JEPA) that uses image captions to reduce the prediction uncertainty. Specifically, we modulate the predicted patch features using a self-supervised, image-grounded text conditioner that computes sparse cross-attention over input text tokens. With such conditioning, patch features become predictable as a function of text, thus are more semantically meaningful. We show TC-JEPA improves downstream performance and training stability, with promising scaling properties. TC-JEPA also offers a new vision-language pretraining paradigm based on feature prediction only, outperforming contrastive methods on diverse tasks, especially those requiring fine-grained visual understanding and reasoning.

Deep Learning · Generative Models and Autoencoders

Gabriel Moreira, Manuel Marques, Joao Costeira, Chenyan Xiong

We introduce Spectral Guidance, a framework for controlling diffusion models by leveraging the intrinsic geometry of the generative process. As data is progressively corrupted by noise, only a small number of features remain informative for control. We characterize them as the singular functions of a conditional expectation operator and show that they can be learned via a self-supervised objective. Once recovered, this basis enables the projection of arbitrary guidance signals, such as labels, CLIP embeddings, or masks, directly onto the sampling trajectory. This approach allows for stable, high-fidelity control without retraining or denoiser backpropagation during sampling. Empirically, we improve conditional accuracy on CIFAR-10 by 37 percentage points over the strongest training-free baseline while offering $4\times$ faster sampling. Moreover, the same representations that support label and CLIP guidance also enable spatial control, such as mask-based guidance, without auxiliary models. Finally, our framework reveals a phase transition in the generative process, pinpointing the optimal time window for effective guidance.

Deep Learning · Self-Supervised Learning

Kawtar Zaher, Ilyass Moummad, Olivier Buisson, Alexis Joly

Most self-supervised learning (SSL) methods learn continuous visual representations by aligning different views of the same input, offering limited control over how information is structured across representation dimensions. In this work, we frame visual self-supervised learning as a discrete communication process between a teacher and a student network, where semantic information is transmitted through a fixed-capacity binary channel. Rather than aligning continuous features, the student predicts multi-label binary messages produced by the teacher. Discrete agreement is enforced through an element-wise binary cross-entropy objective, while a coding-rate regularization term encourages effective utilization of the constrained channel, promoting structured representations. We further show that periodically reinitializing the projection head strengthens this effect by encouraging embeddings that remain predictive across multiple discrete encodings. Extensive experiments demonstrate consistent improvements over continuous agreement baselines on image classification, retrieval, and dense visual prediction tasks, as well as under domain shift through self-supervised adaptation. Beyond backbone representations, we analyze the learned binary codes and show that they form a compact and informative discrete language, capturing semantic factors reusable across classes.

Applications · Time Series

Mengzhou Gao, Kaiwei Wang, Pengfei Jiao

Neural Flows efficiently model irregular multivariate time series by directly learning ODE solution trajectories with neural networks, bypassing step-by-step numerical solvers. Despite their efficiency, many existing approaches treat variables independently, leaving inter-variable interactions underexplored. Moreover, their one-step mapping makes interaction modeling inherently challenging, as it removes the iterative refinement of interactions during learning. To address this challenge, we propose one-step Graph-Structured Neural Flows (GSNF), which introduce two auxiliary-trajectory self-supervision strategies to strengthen interaction learning: (i) interaction-aware trajectory generation via re-initialization, which induces trajectory divergence to expose graph-induced interactions, with a theoretically derived lower bound on divergence; and (ii) reverse-time trajectory generation, which enforces forward–backward consistency to regularize graph learning, enabled by flow invertibility. Experiments on five real-world datasets show that GSNF achieves state-of-the-art classification performance with highly competitive training time and memory usage.

Deep Learning · Self-Supervised Learning

Dong Lao

This position paper argues that the absence of labels does not imply the absence of human supervision in visual learning, and therefore urges the research community to explicitly identify sources of supervision, rather than grouping all label-free approaches under the umbrella term "unsupervised". Many recent methods in computer vision build upon pre-trained representations learned from large-scale unlabeled data, and are therefore regarded as requiring no human supervision. We argue that this view conflates label-free learning with human-free learning, as data curation and filtering inevitably embed substantial human priors on which modern learning systems rely. This confusion risks gatekeeping fundamental unsupervised learning research, a trend reflected in the surprising decline of the term “unsupervised” in paper titles following the rise and widespread adoption of self-supervised pre-training, despite continued growth of the field. Rather than questioning the legitimacy of foundational pre-training within unsupervised learning, we advocate for greater conceptual clarity by encouraging authors to disclose data distribution priors and data-selective biases, and to specify which components of a learning pipeline depend on which assumptions. Standardized disclosure practices can improve academic communication, ensure fairer comparisons, and preserve methodological diversity in unsupervised learning.

Reinforcement Learning · Batch/Offline

Yuan Lu, Dongqi Han, Yansen Wang, Dongsheng Li

Diffusion planners are a strong approach for offline reinforcement learning, but they can fail when value-guided selection favours trajectories that score well yet are locally inconsistent with the environment dynamics, resulting in brittle execution. We propose Self-supervised Action Gating with Energies (SAGE), an inference-time re-ranking method that penalises dynamically inconsistent plans using a latent consistency signal. SAGE trains a Joint-Embedding Predictive Architecture (JEPA) encoder on offline state sequences and an action-conditioned latent predictor for short horizon transitions. At test time, SAGE assigns each sampled candidate an energy given by its latent prediction error and combines this feasibility score with value estimates to select actions. SAGE can integrates into existing diffusion planning pipelines that can sample trajectories and select actions via value scoring; it requires no environment rollouts and no policy re-training. Across locomotion, navigation, and manipulation benchmarks, SAGE improves the performance and robustness of diffusion planners.

Applications · Chemistry, Physics, and Earth Sciences

Zhixiang Cheng, Hongxin Xiang, Mingquan Liu, Tengfei Ma, Yingzhuo Tu, Wenjie Du, Bosheng Song, Yiping Liu, xiangxiang Zeng

Precise property prediction of organic materials is pivotal for next-generation electronic and energy devices. In density functional theory (DFT), the electron density (ED) serves as the fundamental determinant of material properties. Yet, establishing it as an input modality for material property prediction has been impeded by two practical barriers: scarce large-scale ED data and the enormous computational complexity of ED representation. To bridge these gaps, we introduce VisionED, an efficient physics-informed model pre-trained on electron-density images. We curate a dataset of 2 million molecules and represent ED as multi-shot images that efficiently encode both geometric and electronic structure. VisionED is then pre-trained on 12 million multi-shot ED images via cross-scale, physics-informed pretext tasks. Empirical evaluations on photovoltaic and organic chromophore datasets show that VisionED outperforms state-of-the-art baselines by up to 27.0\%, exhibiting superior robustness under distribution shifts and data scarcity. Notably, the model generalizes to unseen device-scale applications, successfully recovering experimental trends and mixing-ratio effects in ternary blends with an average accuracy of 93\%. Moreover, relative to the previous ED point cloud, the ED image improves performance by 26.2\% with 2.6$\times$ fewer memory and 4.6$\times$ lower time. The code and data are available at https://anonymous.4open.science/r/VisionED-AC1B.