The rise of cryptocurrencies like Bitcoin has not only increased trade volumes but also broadened the use of graph machine learning techniques, such as address embeddings, to analyze transactions and decipher user patterns. Traditional analysis methods rely on simple heuristics and extensive data gathering, while more advanced Graph Neural Networks encounter challenges such as scalability, poor interpretability, and label scarcity in massive blockchain transaction networks. To overcome existing techniques' computational and interpretability limitations, we introduce a topological approach, Chainlet Orbits, which embeds blockchain addresses by leveraging their topological characteristics in temporal transactions. We employ our innovative address embeddings to investigate financial behavior and e-crime in the Bitcoin and Ethereum networks, focusing on distinctive substructures that arise from user behavior. Our model demonstrates exceptional performance in node classification experiments compared to GNN-based approaches. Furthermore, our approach embeds all daily nodes of the largest blockchain transaction network, Bitcoin, and creates explainable machine learning models in less than 17 minutes which takes days for GNN-based approaches.
论文检索
输入标题、作者或关键词，从 7,876 篇学术成果中精准定位
The olfaction is hardly mentioned in the studies of multi-modal Large Language Models (LLMs). This demo presents a prototypical framework to embody prevalent LLMs with smelling ability using a plug-and-play olfactory signal processing service. To this end, we collect a dataset on Korean beers by self-developed electronic noses (e-noses) and an open-source dataset. An olfaction-related question-answering corpus is also generated to fine-tune LLMs. A gas classification model is applied to identify the smelling liquor upon the e-nose data. We then adopt and fine-tune LLMs on the generated datasets. The results show that LLMs under this framework can interact with the environment by its `nose' and provide olfaction-related answers augmented by our dataset. To the best of our knowledge, this is the first work on embodying LLMs with artificial olfaction. We additionally deployed the gas classification model and the trained LLM in a simple web-based system to show the feasibility of our prototype. Our demo video can be found at: https://bit.ly/4j8x6ZY.
TRIKOP: Exploring Visual Prompting Paradigms for Multi-Grade Knee Osteoarthritis Classification on MRI Images
PDF ↗Knee osteoarthritis (KOA) is a degenerative joint disease that significantly impacts quality of life. While transfer learning shows promise in medical imaging, its application to KOA diagnosis remains challenging due to medical data's unique characteristics. To address this, we propose TRIKOP, a framework leveraging Visual Prompting for KOA diagnosis on MRI. Our approach explores three prompt-generating strategies that extract clinically relevant information from input images. Each prompt type is encoded using a tailored method to integrate effectively into the Vision Transformer for optimal representation. Among them, the contrastive embedding prompting strategy achieves 63.04% accuracy on the OAI dataset, surpassing prior studies. Moreover, TRIKOP produces attention maps highlighting diagnostically significant regions, improving model interpretability. This work highlights TRIKOP’s potential to improve AI-driven KOA diagnosis and clinical support.
TimelyMed: AI-Driven Clinical Course Attribution and Temporal Mapping for Psychiatric Medical Records
PDF ↗Timely understanding of a patient’s clinical course is crucial for effective treatment. Extracting course-related information, such as temporal and medical events, from unstructured medical records is both challenging and time-consuming, especially when relying on manual identification by physicians. We introduce TimelyMed, a system powered by a locally deployed large language model (LLM) that ensures data security while efficiently organizing key psychiatric events and their corresponding temporal information. Additionally, our system is attributed, allowing clinicians to not only categorize events but also trace them back to their original textual descriptions, ensuring transparency and interpretability in clinical decision-making. By organizing temporal and medical event information into timelines, our system enables physicians to quickly grasp a patient’s medical history while effectively reducing their cognitive burden.
Large language models (LLMs) have demonstrated impressive progress in various natural language progress tasks. However, it has been observed that LLMs still struggle with complex causal and logical reasoning. To facilitate this research direction, we first proposed a training method to distinguish causal relationships from spurious correlations in sentiment classification tasks. Then we conducted a comprehensive survey categorizing existing approaches, firstly identifying the main challenges of complex logical question-answering tasks and logical inconsistency across different questions. Our ongoing projects mainly focus on two points: (1) incorporating modal and epistemic logic to evaluate and enhance LLMs’ reasoning ability to handle more complex and diverse reasoning tasks, and (2) phased training LLMs with curriculum learning to improve their logical reasoning performance.
With approximately 2 billion chest X-ray examinations conducted globally each year, the demand for radiological interpretation far surpasses the available expertise, particularly in resource-constrained regions. Recent advancements in artificial intelligence and computer vision present promising solutions for automated chest X-ray analysis. Nevertheless, integrating AI-driven diagnostics into clinical practice encounters several challenges, including data-centric issues, implementation barriers, deployment complexities, and the need for trustworthy AI. This dissertation focuses on the data-centric aspect, making significant contributions through enhanced data collection, the creation of novel datasets, algorithm development, privacy-preserving collaborative learning, and modelling for low-resolution data. It offers practical methodologies for embedding AI into chest radiology workflows, with a particular emphasis on addressing underserved conditions and healthcare settings with limited data availability. Furthermore, this work illustrates how tailored AI solutions can democratize access to high-quality radiological care while balancing privacy considerations and operational constraints across diverse environments.
Managing novelty in perception-based human activity recognition (HAR) is critical in realistic settings to improve task performance over time and ensure solution generalization outside of prior seen samples. Novelty manifests in HAR as unseen samples, activities, objects, environments, and sensor changes, among other ways. Novelty may be task-relevant, such as a new class or new features, or task-irrelevant resulting in nuisance novelty, such as never before seen noise, blur, or distorted video recordings. To perform HAR optimally, algorithmic solutions must be tolerant to nuisance novelty, and learn over time in the face of novelty. This paper 1) formalizes the definition of novelty in HAR building upon the prior definition of novelty in classification tasks, 2) proposes an incremental open world learning (OWL) protocol and applies it to the Kinetics datasets to generate a new benchmark KOWL-718, 3) analyzes the performance of current stateof-the-art HAR models when novelty is introduced over time, 4) provides a containerized and packaged pipeline for reproducing the OWL protocol and for modifying for any future updates to Kinetics. The experimental analysis includes an ablation study of how the different models perform under various conditions as annotated by Kinetics-AVA. The code may be used to analyze different annotations and subsets of the Kinetics datasets in an incremental open world fashion, as well as be extended as further updates to Kinetics are released.
CAM-based methods are widely-used post-hoc interpretability methods that produce a saliency map to explain the decision of an image classification model. The saliency map highlights the important areas of the image relevant to the prediction. In this paper, we show that most of these methods can incorrectly attribute an important score to parts of the image that the model cannot see. We show that this phenomenon occurs both theoretically and experimentally. On the theory side, we analyze the behavior of GradCAM on a simple masked CNN model at initialization. Experimentally, we train a VGG-like model constrained to not use the lower part of the image and nevertheless observe positive scores in the unseen part of the image. This behavior is evaluated quantitatively on two new datasets. We believe that this is problematic, potentially leading to mis-interpretation of the model's behavior.
We design a lightweight structural feature extraction technique for graph classification. It leverages node subsets and connection strength reflected by random-walk-based heuristics, presenting a scalable, unsupervised, and easily interpretable alternative. We provide theoretical insights into our technical design and establish a relation between the extracted structural features and the graph spectrum. We show our method achieves high levels of computational efficiency while maintaining robust classification accuracy.
Machine learning has rapidly progressed, resulting in a vast repository of both general and specialized models that address diverse practical needs. Reusing pre-trained models (PTMs) from public model zoos has emerged as an effective strategy, leveraging rich model resources and reshaping traditional machine learning workflows. These PTMs encapsulate valuable inductive biases beneficial for downstream tasks. Well-designed reuse strategies enable models to be adapted beyond their original scope, enhancing both performance and efficiency in target machine learning systems. This survey offers a unifying perspective on model reuse, establishing connections across various domains and presenting a novel taxonomy that encompasses the full lifecycle of PTM utilization---including selection from model zoos, adaptation techniques, and related areas such as model representation learning. We delve into the similarities and distinctions between reusing specialized and general PTMs, providing insights into their respective advantages and limitations. Furthermore, we discuss key challenges, emerging trends, and future directions in model reuse, aiming to guide research and practice in the era of large-scale pre-trained models. A comprehensive list of papers about model reuse is available at https://github.com/LAMDA-Model-Reuse/Awesome-Model-Reuse.
Fairness in Federated Learning (FL) is emerging as a critical factor driven by heterogeneous clients’ constraints and balanced model performance across various scenarios. In this survey, we delineate a comprehensive classification of the state-of-the-art fairness-aware approaches from a multifaceted perspective, i.e., model performance-oriented and capability-oriented. Moreover, we provide a framework to categorize and address various fairness concerns and associated technical aspects, examining their effectiveness in balancing equity and performance within FL frameworks. We further examine several significant evaluation metrics leveraged to measure fairness quantitatively. Finally, we explore exciting open research directions and propose prospective solutions that could drive future advancements in this important area, laying a solid foundation for researchers working toward fairness in FL.
A Survey on Temporal Interaction Graph Representation Learning: Progress, Challenges, and Opportunities
PDF ↗Temporal interaction graphs (TIGs), defined by sequences of timestamped interaction events, have become ubiquitous in real-world applications due to their capability to model complex dynamic system behaviors. As a result, temporal interaction graph representation learning (TIGRL) has garnered significant attention in recent years. TIGRL aims to embed nodes in TIGs into low-dimensional representations that effectively preserve both structural and temporal information, thereby enhancing the performance of downstream tasks such as classification, prediction, and clustering within constantly evolving data environments. In this paper, we begin by introducing the foundational concepts of TIGs and emphasizing the critical role of temporal dependencies. We then propose a comprehensive taxonomy of state-of-the-art TIGRL methods, systematically categorizing them based on the types of information utilized during the learning process to address the unique challenges inherent to TIGs. To facilitate further research and practical applications, we curate the source of datasets and benchmarks, providing valuable resources for empirical investigations. Finally, we examine key open challenges and explore promising research directions in TIGRL, laying the groundwork for future advancements that have the potential to shape the evolution of this field.
Differentiable rendering techniques like Gaussian Splatting and Neural Radiance Fields have become powerful tools for generating high-fidelity models of 3D objects and scenes. Their ability to produce both physically plausible and differentiable models of scenes are key ingredient needed to produce physically plausible adversarial attacks on DNNs. However, the adversarial machine learning community has yet to fully explore these capabilities, partly due to differing attack goals (e.g., misclassification, misdetection) and a wide range of possible scene manipulations used to achieve them (e.g., alter texture, mesh). This survey contributes a framework that unifies diverse goals and tasks, facilitating easy comparison of existing work, identifying research gaps, and highlighting future directions—ranging from expanding attack goals and tasks to account for new modalities, state-of-the-art models, tools, and pipelines, to underscoring the importance of studying real-world threats in complex scenes.
Journalistic manual fact-checking is the usual way to address fake news; however, this labor-intensive task regularly is not a match for the scale of the problem. The literature introduced automated fact-checking (AFC) as a potential solution; however, there is still missing functionality in the AFC pipeline, a lack of research benchmarking data, and a disconnect between their design and human factors crucial for adoption. We present a fully explainable AFC framework designed to augment professional journalists in the wild. A novel human annotation-free approach surpasses state-of-the-art multi-label classification by 12%. It is the first to demonstrate strong generalization across different claim subjects without retraining and to generate complete verdict explanation articles and their summaries. A focused user study of 103 professional journalists, with 93% having dedicated experience with fact-checking, validates the framework's level of explainability, transparency, and quality of generated fact-checking artifacts. The importance of establishing clear source selection and bias evaluation criteria reinforced the need for human augmentation, not replacement, by AFC systems.
Multimodal controversy detection, which involves determining whether a given video and its associated comments are controversial, plays a pivotal role in risk management on social video platforms. Existing methods typically provide only classification results, failing to identify what aspects are controversial and why, thereby lacking detailed explanations. To address this limitation, we propose a novel Agent-based Multimodal Controversy Detection architecture, termed AgentMCD. This architecture leverages Large Language Models (LLMs) as generative agents to simulate human behavior and improve explainability. AgentMCD employs a multi-aspect reasoning process, where multiple judges conduct evaluations from diverse perspectives to derive a final decision. Furthermore, a multi-agent simulation process is incorporated, wherein agents act as audiences, offering opinions and engaging in free discussions after watching videos. This hybrid framework enables comprehensive controversy evaluation and significantly enhances explainability. Experiments conducted on the MMCD dataset demonstrate that our proposed architecture outperforms existing LLM-based baselines in both high-resource and low-resource comment scenarios, while maintaining superior explainability.
International aid is a critical mechanism for promoting economic growth and well-being in developing nations, supporting progress toward the Sustainable Development Goals (SDGs). However, tracking aid contributions remains challenging due to labor-intensive data management, incomplete records, and the heterogeneous nature of aid data. Recognizing the urgency of this challenge, we partnered with government agencies to develop an AI model that complements manual classification and mitigates human bias in subjective interpretation. By integrating SDG-specific semantics and leveraging prior knowledge from language models, our approach enhances classification accuracy and accommodates the diversity of aid projects. When applied to a comprehensive dataset spanning multiple years, our model can reveal hidden trends in the temporal evolution of international development cooperation. Expert interviews further suggest how these insights can empower policymakers with data-driven decision-making tools, ultimately improving aid effectiveness and supporting progress toward SDGs.
In the realm of Natural Language Processing (NLP), common approaches for handling human disagreement consist of aggregating annotators' viewpoints to establish a single ground truth. However, prior studies show that disregarding individual opinions can lead to the side-effect of under-representing minority perspectives, especially in subjective tasks, where annotators may systematically disagree because of their preferences. Recognizing that labels reflect the diverse backgrounds, life experiences, and values of individuals, this study proposes a new multi-perspective approach using soft labels to encourage the development of the next generation of perspective-aware models—more inclusive and pluralistic. We conduct an extensive analysis across diverse subjective text classification tasks including hate speech, irony, abusive language, and stance detection, to highlight the importance of capturing human disagreements, often overlooked by traditional aggregation methods. Results show that the multi-perspective approach not only better approximates human label distributions, as measured by Jensen-Shannon Divergence (JSD), but also achieves superior classification performance (higher F1-scores), outperforming traditional approaches. However, our approach exhibits lower confidence in tasks like irony and stance detection, likely due to the inherent subjectivity present in the texts. Lastly, leveraging Explainable AI (XAI), we explore model uncertainty and uncover meaningful insights into model predictions. All implementation details are available at our github repo.
Memes are popular in the modern world and are distributed primarily for entertainment. However, harmful ideologies such as misogyny can be propagated through innocent-looking memes. The detection and understanding of why a meme is misogynous is a research challenge due to its multimodal nature (image and text) and its nuanced manifestations across different societal contexts. We introduce a novel multimodal approach, namely, MM-Misogyny to detect, categorize, and explain misogynistic content in memes. MM-Misogyny processes text and image modalities separately and unifies them into a multimodal context through a cross-attention mechanism. The resulting multimodal context is then easily processed for labeling, categorization, and explanation via a classifier and Large Language Model (LLM). The evaluation of the proposed model is performed on a newly curated dataset (What’s Beneath Misogynous Stereotyping (WBMS)) created by collecting misogynous memes from cyberspace and categorizing them into four categories, namely, Kitchen, Leadership, Working, and Shopping. The model not only detects and classifies misogyny, but also provides a granular understanding of how misogyny operates in operates in domains of life. The results demonstrate the superiority of our approach compared to existing methods. The code and dataset are available at https://github.com/Misogyny.
LogiDebrief: A Signal-Temporal Logic Based Automated Debriefing Approach with Large Language Models Integration
PDF ↗Emergency response services are critical to public safety, with 9-1-1 call-takers playing a key role in ensuring timely and effective emergency operations. To ensure call-taking performance consistency, quality assurance is implemented to evaluate and refine call-takers' skillsets. However, traditional human-led evaluations struggle with high call volumes, leading to low coverage and delayed assessments. We introduce LogiDebrief, an AI-driven framework that automates traditional 9-1-1 call debriefing by integrating Signal-Temporal Logic (STL) with Large Language Models (LLMs) for fully-covered rigorous performance evaluation. LogiDebrief formalizes call-taking requirements as logical specifications, enabling systematic assessment of 9-1-1 calls against procedural guidelines. It employs a three-step verification process: (1) contextual understanding to identify responder types, incident classifications, and critical conditions; (2) STL-based runtime checking with LLM integration to ensure compliance; and (3) automated aggregation of results into quality assurance reports. Beyond its technical contributions, LogiDebrief has demonstrated real-world impact. Successfully deployed at Metro Nashville Department of Emergency Communications, it has assisted in debriefing 1,701 real-world calls, saving 311.85 hours of active engagement. Empirical evaluation with real-world data confirms its accuracy, while a case study and extensive user study highlight its effectiveness in enhancing call-taking performance.
Maps are crucial for various smart city applications as a core component of city geographic information systems (GIS). Developing effective Map Entity Representation Learning methods can extract semantic information for downstream tasks like crime rate prediction and land use classification, with significant application potential. A map comprises three entity types: land parcels, road segments, and points of interest. Most existing methods focus on a single entity type, losing inter-entity relationships and weakening representation effectiveness for real-world applications. Thus, jointly modelling and representing multiple map entity types is essential. However, designing a unified framework is challenging due to map data's unstructured, complex, and heterogeneous nature. We propose a novel method, HygMap, to represent all map entity types. We model the map as a heterogeneous hypergraph, design an encoder for map entities, and introduce a hybrid self-supervised training scheme. This architecture comprehensively captures the heterogeneous relationships among map entities at different levels. Experiments on nine downstream tasks with two real-world datasets show that our framework outperforms all baselines, with good computational efficiency and scalability.