P PaperCompassAI 论文工作台
← 返回论文检索
NeurIPS 2024PosterAccept (Poster)

Consent in Crisis: The Rapid Decline of the AI Data Commons

Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, Kevin Klyman, Christopher Klamm, Hailey Schoelkopf, Nikhil Singh, Manuel Cherep, Ahmad Anis, An Dinh, Caroline Shamiso Chitongo, Da Yin, Damien Sileo, Deividas Mataciunas, Diganta Misra, Emad Alghamdi, Enrico Shippole, Jianguo Zhang, Joanna Materzynska, Kun Qian, Kushagra Tiwary, Lester James V. Miranda, Manan Dey, Minnie Liang, Mohammed Hamdy, Niklas Muennighoff, Seonghyeon Ye, Seungone Kim, Shrestha Mohanty, Vipul Gupta, Vivek Sharma, Minh Chien Vu, Xuhui Zhou, Yizhi Li, Caiming Xiong, Luis Villa, Stella Biderman, Hanlin Li, Daphne Ippolito, Sara Hooker, Jad Kabbara, Alex Pentland

Massachusetts Institute of Technology · Harvard University · Boston University · The University of Edinburgh · McGill University, Topos · MIT · MLC · Minion AI · Stanford University · Universität Mannheim · EleutherAI · Dartmouth College · University of London · UCLA · INRIA · AQ22 · Max-Planck-Institute for Intelligent Systems, Max-Planck Institute · King Abdulaziz University · Teraflop AI · Senior Research Scientist @ Salesforce AI Research · Columbia University · Allen Institute for Artificial Intelligence · SAP · Independent · Korea Advanced Institute of Science and Technology · Carnegie Mellon University · Microsoft · Pennsylvania State University · Sony AI, Sony Research · DETOMO Inc. · CMU, Carnegie Mellon University · The University of Manchester · Salesforce Research · Luis Villa Law · Booz Allen Hamilton EleutherAI · University of Texas at Austin · School of Engineering and Applied Science, University of Pennsylvania · Cohere For AI

论文页面 ↗ 登录后记笔记

PDF 由论文原始站点提供，PaperCompass 不保存论文文件。

摘要

General-purpose artificial intelligence (AI) systems are built on massive swathes of public web data, assembled into corpora such as C4, RefinedWeb, and Dolma. To our knowledge, we conduct the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora. Our audit of 14,000 web domains provides an expansive view of crawlable web data and how codified data use preferences are changing over time. We observe a proliferation of AI-specific clauses to limit use, acute differences in restrictions on AI developers, as well as general inconsistencies between websites' expressed intentions in their Terms of Service and their robots.txt. We diagnose these as symptoms of ineffective web protocols, not designed to cope with the widespread re-purposing of the internet for AI. Our longitudinal analyses show that in a single year (2023-2024) there has been a rapid crescendo of data restrictions from web sources, rendering ~5\%+ of all tokens in C4, or 28%+ of the most actively maintained, critical sources in C4, fully restricted from use. For Terms of Service crawling restrictions, a full 45% of C4 is now restricted. If respected or enforced, these restrictions are rapidly biasing the diversity, freshness, and scaling laws for general-purpose AI systems. We hope to illustrate the emerging crises in data consent, for both developers and creators. The foreclosure of much of the open web will impact not only commercial AI, but also non-commercial AI and academic research.