Sentence Embedding Evaluation Using Pyramid Annotation

Tal Baumel, Raphael Cohen, Michael Elhadad

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


Word embedding vectors are used as input for a variety of tasks. Choosing the right model and features for producing such vectors is not a trivial task and different embedding methods can greatly affect results. In this paper we repurpose the "Pyramid Method" annotations used for evaluating automatic summarization to create a benchmark for comparing embedding models when identifying paraphrases of text snippets containing a single clause. We present a method of converting pyramid annotation files into two distinct sentence embedding tests. We show that our method can produce a good amount of testing data, analyze the quality of the testing data, perform test on
several leading embedding methods, and finally explain the downstream usages of our task and its significance.
Original languageEnglish GB
Title of host publicationProceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval@ACL 2016, Berlin, Germany, August 2016
PublisherAssociation for Computational Linguistics
Number of pages5
StatePublished - 2016


Dive into the research topics of 'Sentence Embedding Evaluation Using Pyramid Annotation'. Together they form a unique fingerprint.

Cite this