Estimated Judge Reliabilities for Weighted Bradley-Terry-Luce Are Not Reliable

摘要

There are many applications for which we want to learn a latent scale for subjective properties, such as the excitement of a photo or the legibility of a font; however, obtaining human-labeled data is costly and time-consuming. One oft-used method for acquiring these labels, despite the cost being quadratic in the number of items, is the method of pairwise comparisons since this method minimizes the effect of biases and generally can be used effectively outside of a controlled environment. Crowdsourcing appears to be a panacea since online platforms provide affordable access to numerous people, but these participants, judges, vary in diligence and expertise. Several methods have been proposed to assign weights to judges based on their responses relative to everyone else, the goal being to reduce exposure to poor performers, hopefully upgrading the quality of the data. Our research focuses on two natural extensions to the Bradley-Terry-Luce formulation of scaling that jointly optimize for both scale value and judge weights. While both methods appear to perform at least as well as the unweighted formulation on average with well-behaved judges, we report a previously unknown flaw, revealing that the resultant judge weights should not be interpreted as reliabilities. Consequently, these values should not be leveraged for decisions about the judges, such as for active sampling or to validate the participant pool.