With the rapid advancement of video generation models such as Sora, video quality assessment (VQA) is becoming increasingly crucial for selecting high-quality videos from large-scale datasets used in pre-training. Traditional VQA methods, typically producing single numerical scores, often lack comprehensiveness and interpretability. To address these challenges, we introduce MVQA-68K, a novel multi-dimensional VQA dataset comprising over 68,000 carefully annotated videos, covering seven essential quality dimensions: overall aesthetics, camera movement, dynamic degree, texture detail, composition, visual quality, and factual consistency. Each annotation includes detailed chain-of-thought reasoning to facilitate interpretability and comprehensive understanding. Extensive experiments demonstrate that MVQA-68K significantly enhances the performance of various multimodal large language models (MLLMs) on the VQA task, achieving state-of-the-art results not only on our internal test set (Fig.1) but also on public benchmarks including LSVQ-test, LSVQ-1080p, and LIVE-VQC. Meantime, incorporating explicit reasoning process during VQA training substantially boosts the zero-shot generalization. Code and dataset will be available at github: https://github.com/Controller01-ai/MVQA-68K.
论文检索
输入标题、作者或关键词，从 1,386 篇学术成果中精准定位
Multimodal Large Language Models (MLLM) have significantly advanced AI-assisted medical diagnosis, but often generate factually inconsistent responses that deviate from established medical knowledge. Retrieval-Augmented Generation (RAG) enhances factual accuracy by integrating external sources, but it presents two key challenges. First, insufficient retrieval can miss critical information, whereas excessive retrieval can introduce irrelevant or misleading content, disrupting model output. Second, even when the model initially provides correct answers, over-reliance on retrieved data can lead to factual errors. To address these issues, we introduce Multimodal Intelligent Retrieval and Augmentation (MIRA) framework, designed to optimize factual accuracy in MLLM. MIRA consists of two key components: (1) a calibrated Rethinking and Rearrangement module that dynamically adjusts the number of retrieved contexts to manage factual risk, and (2) A medical RAG framework integrating image embeddings and a medical knowledge base with a query-rewrite module for efficient multimodal reasoning. This enables the model to effectively integrate both its inherent knowledge and external references. Our evaluation of publicly available medical VQA and report generation benchmarks demonstrates that MIRA substantially enhances factual accuracy and overall performance, achieving new state-of-the-art results. Code is released at https://github.com/mbzuai-oryx/MIRA.
Visual impairment affects over 200 million individuals globally, creating significant challenges in daily visual tasks. While vision-language models offer transformative assistive potential, existing systems based on Multimodal Large Language Models (MLLMs) face a serious cross-contamination problem when processing real-world images captured by blind and low-vision (BLV) users: when jointly processing imperfect images and specific questions, current models are often misled by question assumptions rather than adhering to visual facts, generating hallucinations about objects not present in the image. We introduce DR-VQA (Decompose-then-Reconstruct Visual Question Answering), a novel framework that balances user intent with visual facts. Our approach prevents cross-contamination through structured reasoning. Our approach deliberately separates image processing from question analysis, ensuring model-generated descriptions are strictly based on image facts without being influenced by questions. Subsequently, through a structured decomposition mechanism, the system generates targeted sub-questions relevant to user intent, gradually aligning visual descriptions with user needs while minimizing question bias. During final synthesis, a memory-reset LLM reconstructs the reasoning chain with detailed information to generate responses that either provide evidence-supported conclusions or transparently acknowledge information limitations. Experimental evaluations demonstrate our framework's effectiveness in reducing hallucination risks while improving answer accuracy. By systematically balancing user intent with factual visual evidence, this work advances BLV-assistive technologies from probabilistic outputs to reliable visual assistance services.
Emotional Video Captioning (EVC) is an emerging task that aims to describe factual content with the intrinsic emotions expressed in videos. Existing EVC methods perceive global emotional cues through visual features at first, and then combine them with the video features to guide the emotional caption generation, which ignores the critical characteristic of the EVC task that emotional cues have intrinsic motivational causes reflected in the video content. Such video causes have a facilitative effect on both emotion perception and emotion-attributed caption generation. To this end, a multi-round mutual emotion-cause pair extraction network (MM-ECPE) is proposed in this paper for the joint extraction of emotional cues and visual causes through iterative mutual refinement. Specifically, in the 1st-round mutual learning, we propose a spatio-temporal disentangled visual adaptive refinement (ST-DVAR) and a multi-level video-guided emotion affine transformation (MV-EAT) to achieve preliminary refinement on video features and emotion lexicon to eliminate the noise caused by emotion-irrelevant visual information and video-irrelevant emotional information. Then, in the 2nd-round mutual learning, we exploit the cross-attention of the preliminary refined features and the original features to obtain the ultimate emotional cues and visual causes, and couple them in pair-wise extraction through contrastive loss. Overall, our approach optimizes complex semantic understanding and emotion perception of videos, leading to a promising performance in emotional captioning. Extensive experiments on three challenging datasets demonstrate the superiority of our approach and each proposed module, e.g., improving the latest records by +97.5% and +76.2% w.r.t. CIDEr and CFS, respectively, on the EVC-MSVD dataset.
With the rapid development of short video platforms (such as Kuaishou and TikTok), these platforms have increasingly become important channels for the spread of fake news.Therefore, multi-modal fake news detection has attracted extensive attention.Existing studies mainly focus on directly integrating multi-modal information or discovering implicit clues in posts to improve detection performance.However, due to the abuse of video editing techniques, event-irrelevant segments (e.g., advertisements) are frequently mixed into videos, introducing noise information, thereby weakening models' ability to learn crucial information.Moreover, video creators often inject personal tampered information into original news content through audio modality manipulation, potentially distorting the factual. To address these challenges, we propose a novel Event Consistency-aware Robust Fake News Detection (ECR-FND) framework, comprising two key components: an Event-aware Video Denoising Learning (EVDL) and an Audio Tampering-information Capturing Module (ATCM).Specifically, the EVDL filters out the event-irrelevant segments within video modality to focus on core news events. The ATCM adaptively amplifies tampering information in audio modality, enhancing the model's capacity to detect manipulation attempts.Extensive experiments on two benchmark datasets (FakeSV and FakeTT) demonstrate ECR-FND's effectiveness.Our source code is available at https://github.com/immc-lab/ECR-FND.
LargePiG for Hallucination-Free Query Generation: Your Large Language Model is Secretly a Pointer Generator
PDF ↗Recent research on query generation has focused on using Large Language Models (LLMs), which, despite achieving state-of-the-art performance, also introduce hallucination issues in generated queries. In this work, we categorize these issues into relevance hallucination and factuality hallucination, proposing a new typology for hallucinations arising from LLM-based query generation. We present an effective approach to decouple content from form in LLM-generated queries, preserving the factual knowledge extracted and integrated from inputs while leveraging the LLM's linguistic capabilities to construct syntactic structures, including function words. Specifically, we introduce a model-agnostic and training-free method that transforms the Large Language Model into a Pointer-Generator (LargePiG), where the pointer attention distribution utilizes the LLM's inherent attention weights, and the copy probability is derived from the difference between the vocabulary distribution in the model's high layers and the last layer. To validate the effectiveness of LargePiG, we constructed two datasets for assessing hallucination issues in query generation, covering both document and video scenarios. Empirical studies on various LLMs demonstrated LargePiG's superiority across both datasets. Additional experiments further verified that LargePiG reduces hallucination in large vision-language models and enhances the accuracy of document-based question-answering and factuality evaluation tasks. The source code and dataset are available at https://github.com/Jeryi-Sun/LargePiG.
Social media platforms employ various content moderation techniques to remove harmful, offensive, and toxic content, with moderation levels varying across platforms and evolving over time. Parler, a fringe platform popular among conservative users, initially had minimal moderation, promoting itself as a space for open discussion. However, in 2021, it was removed from the Apple and Google App Stores and suspended from Amazon Web Services due to inadequate moderation of harmful content. After a month-long suspension, Parler returned with stricter guidelines, offering a unique opportunity to study the impact of platform-wide policy changes on user behavior and content outcomes. In this paper, we analyzed Parler data to assess the causal associations of these moderation changes on content toxicity and factuality. Using a longitudinal dataset of 17M posts from 432K users, who were active both before and after replatforming, we employed quasi-experimental analysis, controlling for confounding factors. We introduced a novel approach by using data from another social media platform, Twitter, to account for a critical confounding factor: offline events. This allowed us to isolate the effects of Parler's replatforming policies from external real-world influences. Our findings demonstrate that Parler's moderation changes are causally associated with a significant reduction in all forms of toxicity (p < 0.001). Additionally, we observed an increase in the factuality of the news sites shared and a reduction in the number of conspiracy/ pseudoscience sources.
Large Language Models (LLMs) have achieved impressive results in various tasks but struggle with hallucination problems and lack of relevant knowledge, especially in deep complex reasoning and knowledge-intensive tasks. Knowledge Graphs (KGs), which capture vast amounts of facts in a structured format, offer a reliable source of knowledge for reasoning. However, existing KG-based LLM reasoning methods face challenges like handling multi-hop reasoning, multi-entity questions, and effectively utilizing graph structures. To address these issues, we propose Paths-over-Graph (PoG), a novel method that enhances LLM reasoning by integrating knowledge reasoning paths from KGs, improving the interpretability and faithfulness of LLM outputs. PoG tackles multi-hop and multi-entity questions through a three-phase dynamic multi-hop path exploration, which combines the inherent knowledge of LLMs with factual knowledge from KGs. In order to improve the efficiency, PoG prunes irrelevant information from the graph exploration first and introduces efficient three-step pruning techniques that incorporate graph structures, LLM prompting, and a pre-trained language model (e.g., SBERT) to effectively narrow down the explored candidate paths. This ensures all reasoning paths contain highly relevant information captured from KGs, making the reasoning faithful and interpretable in problem-solving. PoG innovatively utilizes graph structure to prune the irrelevant noise and represents the first method to implement multi-entity deep path detection on KGs for LLM reasoning tasks. Comprehensive experiments on five benchmark KGQA datasets demonstrate PoG outperforms the state-of-the-art method ToG across GPT-3.5-Turbo and GPT-4, achieving an average accuracy improvement of 18.9%. Notably, PoG with GPT-3.5-Turbo surpasses ToG with GPT-4 by up to 23.9%.
Online hatred has become an increasingly pervasive issue, affecting individuals and communities across various digital platforms. To combat hate speech in such platforms, counter narratives (CNs) are regarded as an effective method. In recent years, there has been growing interest in using generative AI tools to construct CNs. However, most of the generative models produce generic responses to hate speech and can hallucinate, reducing their effectiveness. To address the above limitations, we propose a counter narrative generation method that enhances CNs by providing non-aggressive, fact-based narratives with relevant background knowledge from two distinct sources, including a web search module. Furthermore, we conduct a comprehensive evaluation using multiple metrics, including LLM-based measures for persuasion, factuality, and informativeness, along with human and traditional NLP evaluations. Our method significantly outperforms baselines, achieving an average factuality score of 0.915, compared to 0.741, 0.701, and 0.69 for competitive baselines, and performs well in human evaluations.
Large Language Models (LLMs) exhibit remarkable capabilities in storing and retrieving vast amounts of factual knowledge. However, they retain outdated or incorrect information from Web corpora. Since full retraining is costly, locate-and-edit model editing methods offer a feasible alternative. Current methods typically follow a two-stage paradigm: (1) identifying critical layers that store knowledge and (2) updating their parameters to store new knowledge. However, both phases have their inherent limitations. Firstly, layer identification is independent of the knowledge being updated, ignoring the differences in knowledge storage patterns. Secondly, parameter updating suffers from high computational overhead due to gradient descent. To solve these, we propose an Explainable and effiCient model Editing method, termed ECE. Specifically, we integrate LLM explainability into the editing process, enabling the adaptive identification of the crucial neurons. Through clustering similar knowledge, we enable batch optimization in a single gradient step, significantly reducing computational time without compromising effectiveness. Extensive experiments demonstrate that ECE can achieve superior performance, showcasing the potential of explainability-driven editing methods for LLMs. Code is available at https://github.com/tianyuzhangterry/ECE.
MSTI-Plus: Introducing Non-Sarcasm Reference Materials to Enhance Multimodal Sarcasm Target Identification
PDF ↗Sarcasm is a subtle expression that indicates the incongruity between literal meanings and factual opinions. For multimodal posts in social medias which consist of both images and texts, sarcasm expressions are even more widespread. Recent works have paid attentions to Multimodal Sarcasm Target Identification (MSTI), which focuses on detecting aspect terms of mockery or ridicule as sarcasm targets. However, the current MSTI benchmark only contains annotations on fine-grained sarcasm targets within sarcastic samples. In practice, it will be featured by two major limitations. First, there lack annotations on non-sarcasm aspects to inform deep models to perceive the semantic difference between sarcasm targets and non-sarcasm aspects. As a result, deep models will tend to incorrectly recognize non-sarcasm aspects as sarcasm targets. Second, there lack non-sarcasm samples to inform deep models to perceive the inherent semantics of sarcasm intentions. Due to the subtle characteristic of sarcasm expressions, models trained with only fine-grained supervision signals cannot thoroughly understand the sarcasm semantics, making the fine-grained task of sarcasm target identification restricted. Motivated by these limitations, this work reconstructs a more comprehensive MSTI benchmark by introducing both fine-grained non-sarcasm aspect annotations for existing sarcasm samples and non-sarcastic samples as non-sarcasm references to enable deep models to clearly perceive the mentioned information during training. Based on the multi-granularity (i.e., both aspect-level and sample-level) non-sarcasm information introduced into this new benchmark, this work further proposes a pluggable Semantics-aware Sarcasm Target Identification mechanism to enhance sarcasm target identification by modeling the overall semantics of sarcasm intentions via an auxiliary sample-level sarcasm recognition task. By modeling the overall semantics of sarcasm intention, deep models can obtain a more comprehensive understanding on sarcasm semantics, leading to improved performance on fine-grained sarcasm target identification. Extensive experiments are conducted to validate our contribution. Both the dataset and code are available at https://github.com/tiggers23/MSTI-Plus.
Graphs have emerged as a foundation for a variety of applications, including capturing factual knowledge, semantic data integration, social networks, and informing machine learning algorithms. Formalising properties of the data and ensuring data quality requires describing schemas of such graphs. Driven by diverse applications, the Semantic Web and database communities developed not only different graph data models-RDF and property graphs-but also different graph schema languages-SHACL, ShEx, and PG-Schema. Each language has its unique approach to defining constraints and validating graph data, leaving potential users in the dark about their commonalities and differences. In this paper, we provide concise formal definitions of the core components of these languages, employ a uniform framework to facilitate a comprehensive comparison between them, and identify a common set of functionalities, shedding light on both overlapping and distinctive features.
A Survey on Unifying Large Language Models and Knowledge Graphs for Biomedicine and Healthcare
PDF ↗In recent years, the landscape of digital biomedicine and healthcare has been reshaped due to the disruptive breakthroughs in AI-facilitated by tremendous data and high-performance computers, large language models (LLMs) have transformed information technology from accessing data to performing analytical tasks. While demonstrating unprecedented capabilities, LLMs have been found unreliable in tasks requiring factual knowledge and rigorous reasoning. Biomedicine and healthcare, as an important vertical domain rapidly benefitting from progress in AI, necessitates strict requirements on the accuracy, controllability, and interpretability of analytical models, posing critical challenges for LLMs. Despite recent studies addressing the hallucination problem of LLMs, research on empowering LLMs with the ability to plan, reason, and ground with explicit knowledge has also started to prosper, especially in the biomedicine and healthcare domain. On the other hand, biomedical data are enormous and notoriously complex, coming from various sources (e.g., biomedical knowledge bases, online literature, and hospitals) and bearing various modalities (e.g., tables, texts, images and time-series). Healthcare professionals have spent decades collecting, cleaning, and curating various types of data. The processes are extremely costly, producing various datasets with different data schemas, coding systems, and quality standards, many privately owned by the creators, making their integrative analysis and utilization through unified AI techniques still rather challenging. The generalizability of LLMs across different types of data endow them strong promises in automating the processing of large-scale complex healthcare data such as into unified knowledge graphs (KGs). Our goal in this survey is to systematically investigate and summarize recent studies on the unification of LLMs and KGs, towards fully utilizing the value of complex data, unleashing the power of generative AI, and expediting next-generation AI for biomedicine and healthcare applications.
With the increasing use of Large Language Models (LLMs) in fields such as e-commerce, domain-specific concept evaluation benchmarks are crucial for assessing their domain capabilities. Existing LLMs may generate factually incorrect information within the complex e-commerce applications. Therefore, it is necessary to build an e-commerce concept benchmark. Existing benchmarks encounter two primary challenges: (1) handle the heterogeneous and diverse nature of tasks(2) distinguish between generality and specificity within the e-commerce field. To address these problems, we propose ChineseEcomQA, a scalable question-answering benchmark focused on fundamental e-commerce concepts. ChineseEcomQA is built on three core characteristics: Focus on Fundamental Concept, E-commerce Generality and E-commerce Expertise. Fundamental concepts are designed to be applicable across a diverse array of e-commerce tasks, thus addressing the challenge of heterogeneity and diversity. Additionally, by carefully balancing generality and specificity, ChineseEcomQA effectively differentiates between broad e-commerce concepts, allowing for precise validation of domain capabilities. We achieve this through a scalable benchmark construction process that combines LLM validation, Retrieval-Augmented Generation (RAG) validation, and rigorous manual annotation. Based on ChineseEcomQA, we conduct extensive evaluations on mainstream LLMs and provide some valuable insights. We hope that ChineseEcomQA could guide future domain-specific evaluations, and facilitate broader LLM adoption in e-commerce applications.
The emergence of foundation models with high generalization performance increases expectations for their application in AI Contact Centers (AICC) services. However, research on AICC services based on foundation models remains insufficient, particularly in the context of non-English languages. To bridge this gap, we construct a Korean phone call-based customer service dataset designed to evaluate the feasibility and fairness of foundation model-based AICC systems. Our utterance-level audio dataset incorporates diverse speaker attributes such as accent, gender, and age, ensuring comprehensive assessments across various demographic factors. Furthermore, we provide dialect-aware scripts to enhance linguistic diversity and realism in AICC agent evaluation. Building upon this dataset, we propose a customized prompt engineering approach to optimize foundation models for AICC services. Our system prompt improves the natural flow of conversation while ensuring accurate information generation by LLMs. In addition, we address the limitations of conventional human-based evaluations, which are costly and susceptible to bias, by introducing an automatic hierarchical evaluation metric. This metric leverages LLMs to systematically assess AI agent responses, considering both overall conversation quality and factual accuracy. Finally, using our fine-grained Korean speech dataset, we conduct an experimental fairness evaluation of AI agents in AICC services. We also demonstrate that our proposed evaluation metric exhibits a higher correlation with human judgments compared to previous predominant metrics. Our work contributes to the development of more reliable, fair, and effective AICC services by providing a high-quality dataset and robust evaluation framework. The code and materials are available at https://anonymous.4open.science/r/fomo4aicc-EC45.
In e-commerce, product descriptions and other forms of copywriting play a critical role in shaping consumer purchasing decisions. However, manually crafting such content is both time-consuming and costly, particularly given the vast and diverse item catalogs. Recent advances in large language models (LLMs) have transformed automated text generation, offering immense potential to streamline this process. Despite their capabilities, LLMs continue to face obstacles in e-commerce applications, including a lack of diversity and an inability to fully grasp the nuanced details of specific items. To address these limitations, we propose a novel framework that integrates graph-based knowledge into Retrieval-Augmented Generation (RAG) to enhance content generation. Our approach leverages user reviews to construct an item-feature graph, capturing both explicit and implicit connections between items and features. This structured representation enables the retrieval of diverse, contextually relevant, and factually grounded information, effectively addressing key deficiencies of existing methods. With the constructed graph, we design a graph traversal mechanism that explores a broader range of item-related features, augmenting the generation process with more varied and informative inputs. Extensive experiments demonstrate that our method significantly improves diversity while preserving fidelity, marking a major advancement in automated e-commerce content generation.
Counterfactual learning is emerging as an important paradigm, rooted in causality, which promises to alleviate common issues of graph neural networks (GNNs), such as fairness and interpretability. However, as in many real-world application domains where conducting randomized controlled trials is impractical, one has to rely on available observational (factual) data to detect counterfactuals. In this paper, we introduce and tackle the problem of searching for counterfactual evidences for the GNN-based node classification task. A counterfactual evidence is a pair of nodes such that, regardless they exhibit great similarity both in the features and in their neighborhood subgraph structures, they are classified differently by the GNN. We develop effective and efficient search algorithms and a novel indexing solution that leverages both node features and structural information to identify counterfactual evidences, and generalizes beyond any specific GNN. Through various downstream applications, we demonstrate the potential of counterfactual evidences to enhance fairness and accuracy of GNNs.
MOTTO: A Mixture-of-Experts Framework for Multi-Treatment, Multi-Outcome Treatment Effect Estimation
PDF ↗Multi-treatment multi-outcome treatment effect estimation plays a vital role in today's industry-level applications. For example, in social media ads, practitioners simultaneously deploy multiple interventions to users' experience and track multi-faceted metrics (e.g., ad performance, engagement, churn). However, existing methods for estimating treatment effects struggle to simultaneously address the complex interplays and ensure robust counterfactual balancing across treatment-outcome pairs. In our paper, we propose MOTTO (Multi-Outcome, Multi-Treat-ment Transfer Model for Treatment Effect Estimation), a Mixture-of-Experts-based framework designed to jointly handle these multi-faceted challenges. MOTTO explicitly learns the relationships across outcomes and distribution overlaps across treatment groups by isolating shared experts within its partitioned architecture. It then employs selective distribution alignment to these treatment-shared experts, optimizing the balance between factual and counterfactual predictions. Importantly, MOTTO can scale to a large number of treatments and outcomes without significantly increasing the number of parameters, as its shared experts can be primarily used for efficiency with comparable performance. We validate MOTTO on synthetic data, demonstrating MOTTO's strong adaptability to varying levels of outcome correlations and confounding. We further highlight its performance improvements across real-world benchmarks and a large-scale advertising ecosystem on one of the world's largest social media platforms.
Warning: This paper contains examples of toxic text. Knowledge editing has emerged as an efficient technique for updating the knowledge of large language models (LLMs), attracting increasing attention in recent years. However, there is a lack of effective measures to prevent the malicious misuse of this technique, which could lead to harmful edits in LLMs. These malicious modifications could cause LLMs to generate toxic content, misleading users into inappropriate actions. In front of this risk, we introduce a new task, Knowledge Editing Type Identification (KETI), aimed at identifying different types of edits in LLMs, thereby providing timely alerts to users when encountering illicit edits. As part of this task, we propose KETIBench, which includes five types of harmful edits covering the most popular toxic types, as well as one benign factual edit. We develop five classical classification models and three BERT-based models as baseline identifiers for both open-source and closed-source LLMs. Our experimental results, across 92 trials involving four models and three knowledge editing methods, demonstrate that all eight baseline identifiers achieve decent identification performance, highlighting the feasibility of identifying malicious edits in LLMs. Additional analyses reveal that the performance of the identifiers is independent of the reliability of the knowledge editing methods and exhibits cross-domain generalization, enabling the identification of edits from unknown sources. All data and code are available in https://github.com/xpq-tech/KETI.
CADS: A Systematic Literature Review on the Challenges of Abstractive Dialogue Summarization (Abstract Reprint)
PDF ↗Abstractive dialogue summarization is the task of distilling conversations into informative and concise summaries. Although focused reviews have been conducted on this topic, there is a lack of comprehensive work that details the core challenges of dialogue summarization, unifies the differing understanding of the task, and aligns proposed techniques, datasets, and evaluation metrics with the challenges. This article summarizes the research on Transformer-based abstractive summarization for English dialogues by systematically reviewing 1262 unique research papers published between 2019 and 2024, relying on the Semantic Scholar and DBLP databases. We cover the main challenges present in dialog summarization (i.e., language, structure, comprehension, speaker, salience, and factuality) and link them to corresponding techniques such as graph-based approaches, additional training tasks, and planning strategies, which typically overly rely on BART-based encoder-decoder models. Recent advances in training methods have led to substantial improvements in language-related challenges. However, challenges such as comprehension, factuality, and salience remain difficult and present significant research opportunities. We further investigate how these approaches are typically analyzed, covering the datasets for the subdomains of dialogue (e.g., meeting, customer service, and medical), the established automatic metrics (e.g., ROUGE), and common human evaluation approaches for assigning scores and evaluating annotator agreement. We observe that only a few datasets (i.e., SAMSum, AMI, DialogSum) are widely used. Despite its limitations, the ROUGE metric is the most commonly used, while human evaluation, considered the gold standard, is frequently reported without sufficient detail on the inter-annotator agreement and annotation guidelines. Additionally, we discuss the possible implications of the recently explored large language models and conclude that our described challenge taxonomy remains relevant despite a potential shift in relevance and difficulty.