Semantic Similarity Is Not Legal Correctness: Evaluating RAG Systems in Brazilian Civil Procedure

摘要

Evaluating retrieval-augmented generation (RAG) systems in legal domains is challenging due to the nuanced nature of legal reasoning and scarcity of domain-specific benchmarks. We investigate semantic similarity and legal correctness in Brazilian legal question answering, developing a synthetic dataset of 3,012 evaluation instances from the Brazilian Civil Procedure Code spanning seven query types, validated through human expert assessment. Our evaluation framework combines BERTScore with domain-adapted LLM-as-Judge (GPT-4o-mini), validated against expert legal assessment. Analysis reveals 50.8% disagreement between semantic similarity and legal correctness, with correlation varying substantially by query type (ρ = 0.464 to ρ = 0.732). Explicit article citation emerges as the strongest quality predictor (Cohen’s d = 1.099), with cited responses achieving 145% higher legal correctness scores. Concrete examples demonstrate bidirectional divergence: responses may achieve high semantic similarity yet cite incorrect articles, or provide legally perfect answers with minimal lexical overlap. Findings demonstrate that semantic metrics alone are insufficient for legal RAG evaluation, with disagreement patterns structured by query type.