Small Mechanisms and Algorithms for Improving Peer Selection

NSF-BSF: RI: Small: Mechanisms and Algorithms for Improving Peer Selection

The project will expand our knowledge and building mechanisms for peer evaluation and selection through four specific aims. The first aim is to develop novel metrics for the evaluation of peer selection mechanisms by defining both normative and quantitative properties that allow to precisely describe features of the peer evaluation and selection process. The second aim is to develop distributed peer selection mechanisms that are able to be used without requiring a centralized controller. This project will develop tools to understand how these mechanisms behave in this distributed setting as well as opportunities to create novel mechanisms for the unique challenges this setting poses. The third aim is to develop our understanding of multi-stage peer evaluation for peer selection. Motivated by the rolling review cycle of many academic conferences, journals, and even some NSF programs, there is a need to investigate the properties of peer evaluation and selection mechanisms when reviews (evaluations) may propagate between specific selection settings. The final aim is to incentivize effort in peer selection: There is a fundamental tension between the classic social choice properties of impartiality, i.e., an agent may not affect their own probability of getting accepted, and provide incentives for reviewers to invest effort in the peer evaluation process. This project will develop a tool kit of mechanisms that allow system designers to rationally choose tradeoffs between the amount of information an agent knows, incentives for effort, and potential for malicious behavior.