LJPCheck: Functional Tests for Legal Judgment Prediction
Nanjing University · nanjing university · Fudan University, Harbin Institute of Technology, Dalian University of Technology, Shanghai Jiaotong University, Shandong University, Peking University, Zhejiang University, University of Science and Technology of China, Hunan University, Beijing Institute of Technology, University of the Chinese Academy of Sciences, Southeast University, Sichuan University, Monash University, Malaysia Campus, Tianjin University, Beijing University of Aeronautics and Astronautics, Wuhan University of Technology, Yale University, Technische Universität München, Wuhan University, nanjing university, Tsinghua University and Wuhan University · University of Texas at Dallas
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。DOI 10.18653/v1/2024.findings-acl.350 ↗
摘要
Legal Judgment Prediction (LJP) refers to the task of automatically predicting judgment results (e.g., charges, law articles and term of penalty) given the fact description of cases. While SOTA models have achieved high accuracy and F1 scores on public datasets, existing datasets fail to evaluate specific aspects of these models (e.g., legal fairness, which significantly impact their applications in real scenarios). Inspired by functional testing in software engineering, we introduce LJPCHECK, a suite of functional tests for LJP models, to comprehend LJP models’ behaviors and offer diagnostic insights. We illustrate the utility of LJPCHECK on five SOTA LJP models. Extensive experiments reveal vulnerabilities in these models, prompting an in-depth discussion into the underlying reasons of their shortcomings.