From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442).

Nicola Ferro; Norbert Fuhr; Gregory Grefenstette; Joseph A. Konstan; Pablo Castells; Elizabeth M. Daly; Thierry Declerck; Michael D. Ekstrand; Werner Geyer; Julio Gonzalo; Tsvi Kuflik; Krister Lindén; Bernardo Magnini; Jian-Yun Nie; Raffaele Perego; Bracha Shapira; Ian Soboroff; Nava Tintarev; Karin Verspoor; Martijn C. Willemsen; Justin Zobel

doi:10.4230/DagMan.7.1.96

From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442).

Nicola Ferro, Norbert Fuhr, Gregory Grefenstette, Joseph A. Konstan, Pablo Castells, Elizabeth M. Daly, Thierry Declerck, Michael D. Ekstrand, Werner Geyer, Julio Gonzalo, Tsvi Kuflik, Krister Lindén, Bernardo Magnini, Jian-Yun Nie, Raffaele Perego, Bracha Shapira, Ian Soboroff, Nava Tintarev, Karin Verspoor, Martijn C. WillemsenJustin Zobel

Department of Software and Information Systems Engineering

Research output: Contribution to journal › Article › peer-review

Abstract

We describe the state-of-the-art in performance modeling and prediction for Information Retrieval (IR), Natural Language Processing (NLP) and Recommender Systems (RecSys) along with its shortcomings and strengths. We present a framework for further research, identifying five major problem areas: understanding measures, performance analysis, making underlying assumptions explicit, identifying application features determining performance, and the development of prediction models describing the relationship between assumptions, features and resulting performance.

Original language	English
Article number	1
Pages (from-to)	96-139
Number of pages	44
Journal	Dagstuhl Manifestos
Volume	7
Issue number	1
DOIs	https://doi.org/10.4230/DagMan.7.1.96
State	Published - 11 Dec 2018

Access to Document

10.4230/DagMan.7.1.96

https://dblp.org/db/journals/dagstuhl-manifestos/dagstuhl-manifestos7.html#FerroFGKCDDEGGK18

Cite this

Ferro, N., Fuhr, N., Grefenstette, G., Konstan, J. A., Castells, P., Daly, E. M., Declerck, T., Ekstrand, M. D., Geyer, W., Gonzalo, J., Kuflik, T., Lindén, K., Magnini, B., Nie, J.-Y., Perego, R., Shapira, B., Soboroff, I., Tintarev, N., Verspoor, K., ... Zobel, J. (2018). From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442). Dagstuhl Manifestos, 7(1), 96-139. Article 1. https://doi.org/10.4230/DagMan.7.1.96

@article{fd3b284d29ad49b695a1c3db481f31ad,

title = "From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442).",

abstract = "We describe the state-of-the-art in performance modeling and prediction for Information Retrieval (IR), Natural Language Processing (NLP) and Recommender Systems (RecSys) along with its shortcomings and strengths. We present a framework for further research, identifying five major problem areas: understanding measures, performance analysis, making underlying assumptions explicit, identifying application features determining performance, and the development of prediction models describing the relationship between assumptions, features and resulting performance.",

author = "Nicola Ferro and Norbert Fuhr and Gregory Grefenstette and Konstan, {Joseph A.} and Pablo Castells and Daly, {Elizabeth M.} and Thierry Declerck and Ekstrand, {Michael D.} and Werner Geyer and Julio Gonzalo and Tsvi Kuflik and Krister Lind{\'e}n and Bernardo Magnini and Jian-Yun Nie and Raffaele Perego and Bracha Shapira and Ian Soboroff and Nava Tintarev and Karin Verspoor and Willemsen, {Martijn C.} and Justin Zobel",

note = "DBLP License: DBLP's bibliographic metadata records provided through http://dblp.org/ are distributed under a Creative Commons CC0 1.0 Universal Public Domain Dedication. Although the bibliographic metadata records are provided consistent with CC0 1.0 Dedication, the content described by the metadata records is not. Content may be subject to copyright, rights of privacy, rights of publicity and other restrictions.",

year = "2018",

month = dec,

day = "11",

doi = "10.4230/DagMan.7.1.96",

language = "English",

volume = "7",

pages = "96--139",

journal = "Dagstuhl Manifestos",

issn = "2193-2433",

publisher = "Schloss Dagstuhl--Leibniz-Zentrum fuer Informatik",

number = "1",

}

Ferro, N, Fuhr, N, Grefenstette, G, Konstan, JA, Castells, P, Daly, EM, Declerck, T, Ekstrand, MD, Geyer, W, Gonzalo, J, Kuflik, T, Lindén, K, Magnini, B, Nie, J-Y, Perego, R, Shapira, B, Soboroff, I, Tintarev, N, Verspoor, K, Willemsen, MC & Zobel, J 2018, 'From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442).', Dagstuhl Manifestos, vol. 7, no. 1, 1, pp. 96-139. https://doi.org/10.4230/DagMan.7.1.96

From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442). / Ferro, Nicola; Fuhr, Norbert; Grefenstette, Gregory et al.
In: Dagstuhl Manifestos, Vol. 7, No. 1, 1, 11.12.2018, p. 96-139.

Research output: Contribution to journal › Article › peer-review

TY - JOUR

T1 - From Evaluating to Forecasting Performance

T2 - How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442).

AU - Ferro, Nicola

AU - Fuhr, Norbert

AU - Grefenstette, Gregory

AU - Konstan, Joseph A.

AU - Castells, Pablo

AU - Daly, Elizabeth M.

AU - Declerck, Thierry

AU - Ekstrand, Michael D.

AU - Geyer, Werner

AU - Gonzalo, Julio

AU - Kuflik, Tsvi

AU - Lindén, Krister

AU - Magnini, Bernardo

AU - Nie, Jian-Yun

AU - Perego, Raffaele

AU - Shapira, Bracha

AU - Soboroff, Ian

AU - Tintarev, Nava

AU - Verspoor, Karin

AU - Willemsen, Martijn C.

AU - Zobel, Justin

N1 - DBLP License: DBLP's bibliographic metadata records provided through http://dblp.org/ are distributed under a Creative Commons CC0 1.0 Universal Public Domain Dedication. Although the bibliographic metadata records are provided consistent with CC0 1.0 Dedication, the content described by the metadata records is not. Content may be subject to copyright, rights of privacy, rights of publicity and other restrictions.

PY - 2018/12/11

Y1 - 2018/12/11

N2 - We describe the state-of-the-art in performance modeling and prediction for Information Retrieval (IR), Natural Language Processing (NLP) and Recommender Systems (RecSys) along with its shortcomings and strengths. We present a framework for further research, identifying five major problem areas: understanding measures, performance analysis, making underlying assumptions explicit, identifying application features determining performance, and the development of prediction models describing the relationship between assumptions, features and resulting performance.

AB - We describe the state-of-the-art in performance modeling and prediction for Information Retrieval (IR), Natural Language Processing (NLP) and Recommender Systems (RecSys) along with its shortcomings and strengths. We present a framework for further research, identifying five major problem areas: understanding measures, performance analysis, making underlying assumptions explicit, identifying application features determining performance, and the development of prediction models describing the relationship between assumptions, features and resulting performance.

U2 - 10.4230/DagMan.7.1.96

DO - 10.4230/DagMan.7.1.96

M3 - Article

SN - 2193-2433

VL - 7

SP - 96

EP - 139

JO - Dagstuhl Manifestos

JF - Dagstuhl Manifestos

IS - 1

M1 - 1

ER -

From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442).

Abstract

Access to Document

Fingerprint

Cite this