Insights and updates from The Referee Project — exploring the future of academic research evaluation.
Academia claims to value collegiality, yet the peer review system reveals a cutthroat reality marked by retaliation fears, competitive scooping, and reluctance to provide thorough reviews. Referee proposes removing the 'peer' from peer review, democratizing the process beyond credentialed gatekeepers to enable direct, unfiltered criticism that strengthens research evaluation.
At Referee, we embrace the hacker's mindset: anticipating and encouraging attempts to game the system makes it stronger. This living document explores potential attack vectors—from collusive reviewing to reputation laundering—and outlines robust countermeasures that build trust through adversarial resilience.
With academics spending 100 million hours annually on peer review yet still unable to keep pace with millions of new papers, AI agents offer the only scalable solution. By focusing AI on granular, specific research weaknesses rather than holistic reviews, Referee enables specialized detection that overcomes concerns about context, hallucinations, and subjective judgments.