Summarization evaluation methods: Experiments and analysis

Two methods are used for evaluation of summarization systems: an evaluation of generated summaries against an" ideal" summary and evaluation of how well summaries help a person perform in a task such as information retrieval. We carried out two large experiments to study the two evaluation methods. Our results show that different parameters of an experiment can (hamatically affect how well a system scores. For example, summary length was found to affect both types of evaluations. For the "ideal" summary based types of evaluations. accuracy decreases as summary length and accuracy on an information retrieval task appear to correlate randomly. In this paper, we show how this parameter and others can effect evaluation results and describe how parameters can be controlled to produce a sound evaluation.