TY - JOUR
T1 - Over-optimization of academic publishing metrics
T2 - Observing Goodhart's Law in action
AU - Fire, Michael
AU - Guestrin, Carlos
N1 - Publisher Copyright:
© 2019 The Author(s).
PY - 2019/6/10
Y1 - 2019/6/10
N2 - Background The academic publishing world is changing significantly, with ever-growing numbers of publications each year and shifting publishing patterns. However, the metrics used to measure academic success, such as the number of publications, citation number, and impact factor, have not changed for decades. Moreover, recent studies indicate that these metrics have become targets and follow Goodhart's Law, according to which, "when a measure becomes a target, it ceases to be a good measure." Results In this study, we analyzed >120 million papers to examine how the academic publishing world has evolved over the last century, with a deeper look into the specific field of biology. Our study shows that the validity of citation-based measures is being compromised and their usefulness is lessening. In particular, the number of publications has ceased to be a good metric as a result of longer author lists, shorter papers, and surging publication numbers. Citation-based metrics, such citation number and h-index, are likewise affected by the flood of papers, self-citations, and lengthy reference lists. Measures such as a journal's impact factor have also ceased to be good metrics due to the soaring numbers of papers that are published in top journals, particularly from the same pool of authors. Moreover, by analyzing properties of >2,600 research fields, we observed that citation-based metrics are not beneficial for comparing researchers in different fields, or even in the same department. Conclusions Academic publishing has changed considerably; now we need to reconsider how we measure success.
AB - Background The academic publishing world is changing significantly, with ever-growing numbers of publications each year and shifting publishing patterns. However, the metrics used to measure academic success, such as the number of publications, citation number, and impact factor, have not changed for decades. Moreover, recent studies indicate that these metrics have become targets and follow Goodhart's Law, according to which, "when a measure becomes a target, it ceases to be a good measure." Results In this study, we analyzed >120 million papers to examine how the academic publishing world has evolved over the last century, with a deeper look into the specific field of biology. Our study shows that the validity of citation-based measures is being compromised and their usefulness is lessening. In particular, the number of publications has ceased to be a good metric as a result of longer author lists, shorter papers, and surging publication numbers. Citation-based metrics, such citation number and h-index, are likewise affected by the flood of papers, self-citations, and lengthy reference lists. Measures such as a journal's impact factor have also ceased to be good metrics due to the soaring numbers of papers that are published in top journals, particularly from the same pool of authors. Moreover, by analyzing properties of >2,600 research fields, we observed that citation-based metrics are not beneficial for comparing researchers in different fields, or even in the same department. Conclusions Academic publishing has changed considerably; now we need to reconsider how we measure success.
KW - Academic publishing metrics
KW - Big data
KW - Data science
KW - Goodhart's Law
KW - Science of science
KW - Scientometrics
UR - http://www.scopus.com/inward/record.url?scp=85067287872&partnerID=8YFLogxK
U2 - 10.1093/gigascience/giz053
DO - 10.1093/gigascience/giz053
M3 - Article
AN - SCOPUS:85067287872
SN - 2047-217X
VL - 8
JO - GigaScience
JF - GigaScience
IS - 6
M1 - giz053
ER -