Adding quality and nuance to the scholarly record
The Referee Project is a non-profit initiative that develops reliability scores for research papers, ranging from 0 to 100. We calculate these scores using a detailed taxonomy of research weaknesses. Moreover, a bug bounty program motivates individuals to identify flaws in these papers. Each identified weakness becomes part of the metadata, clarifying why a paper received its specific score. Users can easily access this metadata through APIs. This transparency helps researchers and others understand the strengths and limitations of various studies.
Isn’t it crucial to know the reliability of the research you rely on?
The Referee Project addresses critical flaws in research evaluation and paper reliability communication. Academia’s emphasis on publishing has skewed incentives, distorting the scholarly record. Meanwhile, the existing system offers only vague indicators of paper reliability—papers are labeled as published (trustworthy), retracted (untrustworthy), or unpublished (questionable). We aim to revolutionize this system by implementing a universal reliability score, underpinned by a standardized research weakness taxonomy and a dynamic bug bounty system.
The current academic peer review system faces several significant issues that undermine its effectiveness and integrity:
There are numerous initiatives aimed at addressing the problems highlighted previously, primarily through two approaches:
There’s just one problem with these efforts: they’re all echoes of the current system that doesn’t work. And why doesn’t the current system work? Because all the evidence suggests that most academics don’t want to do the hard work of peer review.
Even among those who take reviews seriously, few can be expected to master all aspects of a research paper, from statistical nuances to sampling procedures. This is precisely why peer reviews exist—to have another set of eyes catch potential flaws. Despite this, even the most diligent scrutiny can allow some errors to slip through, leading to the publication of papers with overlooked defects.
A final flawed assumption of these initiatives is that only academics can conduct such reviews. The field of software security demonstrates that many non-academic individuals possess the motivation and capability to master complex systems, sometimes even surpassing academics in their expertise in specific cases.
Let’s stop relying solely on academics to solve this problem!
Referee’s overall goal is to create a reliability score for academic papers using a standard taxonomy for research assessment and a targeted bug bounty approach to incentivize engagement. This model offers several superior benefits compared to traditional models, including the following:
Market Theory of Value: Unlike traditional peer review that often operates on a labor theory of value (pay by the hour or by the paper), Referee’s bug bounties are based on the market theory of value. This ensures that compensations are made only for results that add real value, rather than just effort.
Common Research Weakness Enumeration (CRWE): Referee uses a tiered framework similar to the cybersecurity’s Common Weakness Enumeration (CWE), which brings multiple advantages:
Democratization of Knowledge Curation: Bug bounties democratize the process of knowledge curation, reducing the influence of status and traditional gatekeepers like prestigious institutions and journals. This opens up opportunities for a broader range of participants to contribute to the vetting process.
Reliability Scoring System: Introduces a reliability score for papers that can be tracked and influenced by subsequent research. This feature ensures that ongoing research builds on a foundation that is scrutinized for accuracy and reliability.
Rectifies Past Research: Unlike many DeSci projects that focus primarily on future research infrastructure, Referee prioritizes addressing existing problems in published research. This approach is crucial because much of the current academic challenges stem from past research inaccuracies and biases.
Supplementary to Existing Metrics: Aims to supplement traditional academic metrics such as the h-index with a quality score that reflects the reliability and integrity of research, providing a more holistic view of a researcher’s output.