BetterPR: A Dataset for Estimating the Constructiveness of Peer Review Comments

Prabhat Kumar Bharti, Tirthankar Ghosal, Mayank Agarwal, Asif Ekbal

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

Review comments play an important role in the improvement of scientific articles. There are typically many rounds of review-revision before the different reviewers with varying backgrounds arrive at a consensus on a submission. However, not always the reviews are helpful. Sometimes the reviewers are unnecessarily critical of the work without justifying their comments. Peer reviews are always meant to be critical yet constructive feedback on the scientific merit of a submitted article. However, with the rising number of paper submissions leading to the involvement of novice or less experienced reviewers in the reviewing process, the reviewers tend to spend less expert time on their voluntary reviewing job. This results in lackluster reviews where the authors do not have many takeaways from their reviews. The entire scientific enterprise is heavily dependent on this very human peer-review process. In this paper, we make an attempt to automatically distinguish between constructive and non-constructive peer reviews. We deem constructive comment to be the one that, despite being critical, is polite and provides feedback to the authors to improve their submissions. To this end, we present BetterPR, a manually annotated dataset to estimate the constructiveness of peer review comments. Further, we benchmark BetterPR with standard baselines and analyze their performance. We collect the peer reviews from open access forums and design an annotation scheme to label whether a review comment is constructive or non-constructive. We provide our dataset and codes (https://github.com/PrabhatkrBharti/BetterPR.git ) for further exploration by the community.

Original languageEnglish
Title of host publicationLinking Theory and Practice of Digital Libraries - 26th International Conference on Theory and Practice of Digital Libraries, TPDL 2022, Proceedings
EditorsGianmaria Silvello, Oscar Corcho, Paolo Manghi, Giorgio Maria Di Nunzio, Koraljka Golub, Nicola Ferro, Antonella Poggi
PublisherSpringer Science and Business Media Deutschland GmbH
Pages500-505
Number of pages6
ISBN (Print)9783031168017
DOIs
StatePublished - 1 Jan 2022
Externally publishedYes
Event26th International Conference on Theory and Practice of Digital Libraries, TPDL 2022 - Padua, Italy
Duration: 20 Sep 202223 Sep 2022

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13541 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference26th International Conference on Theory and Practice of Digital Libraries, TPDL 2022
Country/TerritoryItaly
CityPadua
Period20/09/2223/09/22

Keywords

  • Peer review quality
  • Peer reviews
  • Review constructiveness

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science (all)

Fingerprint

Dive into the research topics of 'BetterPR: A Dataset for Estimating the Constructiveness of Peer Review Comments'. Together they form a unique fingerprint.

Cite this