Polishing-Only Policies in Peer Reviews are Currently Not Enforceable

摘要

With growing concerns about reviewers using Large Language Models (LLMs) for writing peer reviews, several conferences and journals have enacted policies thatprohibit LLM usage except for polishing, paraphrasing, and grammar correction of otherwise human-written reviews. But, are these policies enforceable? To answer this question, we assemble a dataset of peer reviews simulating multiple levels of human-AI collaboration, and evaluate $5$ state-of-the-art detectors, including two commercial systems. Our analysis shows that all detectors misclassify a substantial fraction of LLM-polished reviews as AI-generated, thereby risking false accusations of academic misconduct. We further investigate whether peer-review-specific signals, including access to the paper manuscript and the constrained domain of scientific writing, can be leveraged to improve detection. While incorporating such signals yields measurable gains in some settings, we identify limitations in each approach and find that none meets the accuracy standards required for identifying AI use in peer reviews. If enforcement of policies is a priority, we recommend completely prohibiting AI use for writing peer reviews.