Peer review is the most influential instrument of quality control in science. But peer reviewers often disagree. All too often, inter-rater reliability, i. e. the correlation between two or more reviews of the same research paper or grant proposal, is low. From a sociological perspective, disagreement among reviewers is to be expected and does not pose a serious threat to the rationale of the review process. In the sociology of science, the normative question of which reviewer (if any) is right, or closer to the mark, is less important than empirical questions about the factors that enhance or diminish agreement, the way dissent is dealt with in review procedures, and how those procedures are organized in order to reach a decision in the absence of unanimity.
In philosophy, disagreement among peer reviewers is a special case of the phenomenon of “peer disagreement”, which has received much attention in recent epistemology. The philosophical debate is concerned with the question of how epistemic peers, i. e. people with the same intellectual capacities, evidence and thoughtfulness in considering that evidence, should react when encountering disagreements. Is it rationally required for them to remain steadfast, or should they rather reconcile their views by according the judgment of the other some weight? Or should both parties even suspend judgment?
Assessing the quality of scholarly research may not be a matter of determining or overlooking the truth. But the mere fact that discordant members of review panels and editorial boards reason with one another indicates that quality assessment is also not simply a matter of subjective preference. The conference aims to explore the phenomenon of disagreement in peer review from various perspectives. It brings together sociologists, philosophers, legal scholars, journal editors and practitioners from funding bodies.
Conference Venue: Humboldt-Universität Berlin, Topoi Building, Hannoversche Str. 6, Room: 1.03