Measuring and Enhancing Human Value Alignment in Zero-Shot Document-Level Claim Extraction

摘要

Identifying central claims from long documents is a fundamental yet challenging task in automated fact-checking pipelines. Manual extraction at the document level is costly and requires domain expertise, while existing automatic methods for claim extraction and evaluation tend to overlook critical dimensions that human fact-checkers consider essential for determining claim quality. In this paper, we propose a novel Human Value-Aligned framework that enables zero-shot document-level claim extraction and evaluation by aligning with expert preferences. We first elicit a structured set of Human Value Alignment (HVA) dimensions from expert annotations and incorporate them into prompt design, instructing large language models (LLMs) to extract high-quality claims that align with expert values. To assess the quality of extracted claims, we further introduce an LLM-based automatic evaluator that scores claims across HVA dimensions and quantifies alignment with expert-written claims. Furthermore, we propose a multi-level agreement metric to evaluate the reliability of automatic HVA evaluator. Experiment results show that our method significantly improves central claim extraction performance, achieving state-of-the-art chrF and P@N scores. Moreover, the proposed HVA evaluator achieves high agreement with human judgments and offers interpretable dimension-level assessments of extracted claims. The HVA framework establishes a reliable and scalable way for human-aligned document-level claim extraction and evaluation in real-world scenarios.