HEvaS: Headline Evaluation System

Marina Litvak; Natalia Vanetik; Itzhak Eretz Kdosha

doi:10.1109/WI.2018.00-16

HEvaS: Headline Evaluation System

Marina Litvak, Natalia Vanetik, Itzhak Eretz Kdosha

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Automatic headline generation is a sub-task of oneline summarization with many reported applications. Evaluation of systems generating headlines is a very challenging and undeveloped area. In this paper, we introduce a system that performs automatic evaluation of systems in terms of a quality of the generated headlines. The evaluation is performed using multiple metrics for comparing evaluated headlines with the gold standard ones or measuring their coverage of main document topics. Both types of metrics evaluate headline's content and informativeness, but not grammatical structure. The only input required by our system is a set of documents with gold standard and automatically generated headlines. The Headline Evaluation System (HEvaS) provides a user with a choice from multiple (10) metrics, then calculates the chosen metrics, performs statistical analysis of the evaluated systems and visualizes the results. Multiple headline generation systems can be evaluated at the same run. This paper describes all evaluation metrics and architecture, utilized by our system. As an evaluation of the HEvaS, we perform a case study with a couple of baseline systems and report the results. Although we tested the system on English content only, the multilingual content can also be supported.

Original language	English
Title of host publication	Proceedings - 2018 IEEE/WIC/ACM International Conference on Web Intelligence, WI 2018
Publisher	Institute of Electrical and Electronics Engineers
Pages	664-667
Number of pages	4
ISBN (Electronic)	9781538673256
DOIs	https://doi.org/10.1109/WI.2018.00-16
State	Published - 10 Jan 2019
Externally published	Yes
Event	18th IEEE/WIC/ACM International Conference on Web Intelligence, WI 2018 - Santiago, Chile Duration: 3 Dec 2018 → 6 Dec 2018

Publication series

Name	Proceedings - 2018 IEEE/WIC/ACM International Conference on Web Intelligence, WI 2018

Conference

Conference	18th IEEE/WIC/ACM International Conference on Web Intelligence, WI 2018
Country/Territory	Chile
City	Santiago
Period	3/12/18 → 6/12/18

Keywords

Headline evaluation
LDA
LSI
ROUGE
Word embedding

ASJC Scopus subject areas

Artificial Intelligence
Computer Networks and Communications

Access to Document

10.1109/WI.2018.00-16

Cite this

Litvak, M., Vanetik, N., & Eretz Kdosha, I. (2019). HEvaS: Headline Evaluation System. In Proceedings - 2018 IEEE/WIC/ACM International Conference on Web Intelligence, WI 2018 (pp. 664-667). Article 8609667 (Proceedings - 2018 IEEE/WIC/ACM International Conference on Web Intelligence, WI 2018). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/WI.2018.00-16

@inproceedings{8512e66082c046b89bcdf1975511c9fb,

title = "HEvaS: Headline Evaluation System",

abstract = "Automatic headline generation is a sub-task of oneline summarization with many reported applications. Evaluation of systems generating headlines is a very challenging and undeveloped area. In this paper, we introduce a system that performs automatic evaluation of systems in terms of a quality of the generated headlines. The evaluation is performed using multiple metrics for comparing evaluated headlines with the gold standard ones or measuring their coverage of main document topics. Both types of metrics evaluate headline's content and informativeness, but not grammatical structure. The only input required by our system is a set of documents with gold standard and automatically generated headlines. The Headline Evaluation System (HEvaS) provides a user with a choice from multiple (10) metrics, then calculates the chosen metrics, performs statistical analysis of the evaluated systems and visualizes the results. Multiple headline generation systems can be evaluated at the same run. This paper describes all evaluation metrics and architecture, utilized by our system. As an evaluation of the HEvaS, we perform a case study with a couple of baseline systems and report the results. Although we tested the system on English content only, the multilingual content can also be supported.",

keywords = "Headline evaluation, LDA, LSI, ROUGE, Word embedding",

author = "Marina Litvak and Natalia Vanetik and {Eretz Kdosha}, Itzhak",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 18th IEEE/WIC/ACM International Conference on Web Intelligence, WI 2018 ; Conference date: 03-12-2018 Through 06-12-2018",

year = "2019",

month = jan,

day = "10",

doi = "10.1109/WI.2018.00-16",

language = "English",

series = "Proceedings - 2018 IEEE/WIC/ACM International Conference on Web Intelligence, WI 2018",

publisher = "Institute of Electrical and Electronics Engineers",

pages = "664--667",

booktitle = "Proceedings - 2018 IEEE/WIC/ACM International Conference on Web Intelligence, WI 2018",

address = "United States",

}

Litvak, M, Vanetik, N & Eretz Kdosha, I 2019, HEvaS: Headline Evaluation System. in Proceedings - 2018 IEEE/WIC/ACM International Conference on Web Intelligence, WI 2018., 8609667, Proceedings - 2018 IEEE/WIC/ACM International Conference on Web Intelligence, WI 2018, Institute of Electrical and Electronics Engineers, pp. 664-667, 18th IEEE/WIC/ACM International Conference on Web Intelligence, WI 2018, Santiago, Chile, 3/12/18. https://doi.org/10.1109/WI.2018.00-16

HEvaS: Headline Evaluation System. / Litvak, Marina; Vanetik, Natalia; Eretz Kdosha, Itzhak.
Proceedings - 2018 IEEE/WIC/ACM International Conference on Web Intelligence, WI 2018. Institute of Electrical and Electronics Engineers, 2019. p. 664-667 8609667 (Proceedings - 2018 IEEE/WIC/ACM International Conference on Web Intelligence, WI 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - HEvaS

T2 - 18th IEEE/WIC/ACM International Conference on Web Intelligence, WI 2018

AU - Litvak, Marina

AU - Vanetik, Natalia

AU - Eretz Kdosha, Itzhak

PY - 2019/1/10

Y1 - 2019/1/10

N2 - Automatic headline generation is a sub-task of oneline summarization with many reported applications. Evaluation of systems generating headlines is a very challenging and undeveloped area. In this paper, we introduce a system that performs automatic evaluation of systems in terms of a quality of the generated headlines. The evaluation is performed using multiple metrics for comparing evaluated headlines with the gold standard ones or measuring their coverage of main document topics. Both types of metrics evaluate headline's content and informativeness, but not grammatical structure. The only input required by our system is a set of documents with gold standard and automatically generated headlines. The Headline Evaluation System (HEvaS) provides a user with a choice from multiple (10) metrics, then calculates the chosen metrics, performs statistical analysis of the evaluated systems and visualizes the results. Multiple headline generation systems can be evaluated at the same run. This paper describes all evaluation metrics and architecture, utilized by our system. As an evaluation of the HEvaS, we perform a case study with a couple of baseline systems and report the results. Although we tested the system on English content only, the multilingual content can also be supported.

AB - Automatic headline generation is a sub-task of oneline summarization with many reported applications. Evaluation of systems generating headlines is a very challenging and undeveloped area. In this paper, we introduce a system that performs automatic evaluation of systems in terms of a quality of the generated headlines. The evaluation is performed using multiple metrics for comparing evaluated headlines with the gold standard ones or measuring their coverage of main document topics. Both types of metrics evaluate headline's content and informativeness, but not grammatical structure. The only input required by our system is a set of documents with gold standard and automatically generated headlines. The Headline Evaluation System (HEvaS) provides a user with a choice from multiple (10) metrics, then calculates the chosen metrics, performs statistical analysis of the evaluated systems and visualizes the results. Multiple headline generation systems can be evaluated at the same run. This paper describes all evaluation metrics and architecture, utilized by our system. As an evaluation of the HEvaS, we perform a case study with a couple of baseline systems and report the results. Although we tested the system on English content only, the multilingual content can also be supported.

KW - Headline evaluation

KW - LDA

KW - LSI

KW - ROUGE

KW - Word embedding

UR - http://www.scopus.com/inward/record.url?scp=85061927439&partnerID=8YFLogxK

U2 - 10.1109/WI.2018.00-16

DO - 10.1109/WI.2018.00-16

M3 - Conference contribution

AN - SCOPUS:85061927439

T3 - Proceedings - 2018 IEEE/WIC/ACM International Conference on Web Intelligence, WI 2018

SP - 664

EP - 667

BT - Proceedings - 2018 IEEE/WIC/ACM International Conference on Web Intelligence, WI 2018

PB - Institute of Electrical and Electronics Engineers

Y2 - 3 December 2018 through 6 December 2018

ER -

HEvaS: Headline Evaluation System

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this