Uncovering and Quantifying Social Biases in Code Generation
Zhejiang University · Peking University · Microsoft · City University of Hong Kong · University of the Chinese Academy of Sciences · IBM Research · Department of Computer Science and Engineering, The Chinese University of Hong Kong
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
With the popularity of automatic code generation tools, such as Copilot, the study of the potential hazards of these tools is gaining importance. In this work, we explore the social bias problem in pre-trained code generation models. We propose a new paradigm to construct code prompts and successfully uncover social biases in code generation models. To quantify the severity of social biases in generated code, we develop a dataset along with three metrics to evaluate the overall social bias and fine-grained unfairness across different demographics. Experimental results on three pre-trained code generation models (Codex, InCoder, and CodeGen) with varying sizes, reveal severe social biases. Moreover, we conduct analysis to provide useful insights for further choice of code generation models with low social bias.