Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations
Stanford University · Hugging Face · Queen Mary University of London · Temple University · StickFlux Labs & Queer in AI · EleutherAI · Iowa State University · SFU · Independent / Eleuther AI · Northwestern University · University College London · ETHZ - ETH Zurich · Meta AI · Universität Leipzig · Simula · Weizenbaum Institut · Hebrew University, Hebrew University of Jerusalem · Harvey · Trustible · Oxford University · Alinia AI · Independent · Harvard University · CMU, Carnegie Mellon University · Singapore Management University · Lawrence Berkeley National Lab · Cornell University · University of Edinburgh, University of Edinburgh · Stanford University // Virtue AI
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Foundation models are increasingly central to high-stakes AI systems, and governance frameworks now depend on evaluations to assess their risks and capabilities. Although general capability evaluations are widespread, social impact assessments covering bias, fairness, privacy, environmental costs, and labor remain uneven. To characterize this landscape, we conduct the first comprehensive analysis of social impact evaluation reporting, examining 186 first-party release reports and 248 third-party evaluation sources, supplemented by developer interviews. We find a stark division of labor: first-party reporting is sparse, often superficial, and declining in areas like environmental impact and bias, while third-party evaluators provide broader, more rigorous coverage of bias, harmful content, and performance disparities. However, only developers can authoritatively report on data provenance, content moderation labor, costs, and infrastructure, yet interviews reveal these disclosures are deprioritized unless tied to product adoption or compliance. Current practices leave major gaps in assessing societal impacts, underscoring the need for policies that mandate developer transparency, strengthen independent evaluation ecosystems, and create shared infrastructure for aggregating third-party evaluations.