Using wikipedia links to construct word segmentation corpora

David Gabay; Ben Eliahu Ziv; Michael Elhadad

Using wikipedia links to construct word segmentation corpora

David Gabay
, Ben Eliahu Ziv
, Michael Elhadad

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

Tagged corpora are essential for evaluating and training natural language processing tools. The cost of constructing large enough manually tagged corpora is high, even when the annotation level is shallow. This article describes a simple method to automatically create a partially tagged corpus, using Wikipedia hyperlinks. The resulting corpus contains information about the correct segmentation of 523,599 non-consecutive words in 363,090 sentences. We used our method to construct a corpus of Modern Hebrew (which we have made available at http://www.cs.bgu.ac.il/-nlpproj). The method can also be applied to other languages where word segmentation is difficult to determine, such as East and South-East Asian languages.

Original language	English
Title of host publication	Wikipedia and Artificial Intelligence
Subtitle of host publication	An Evolving Synergy - Papers from the 2008 AAAI Workshop
Pages	61-63
Number of pages	3
State	Published - 1 Dec 2008
Event	2008 AAAI Workshop - Chicago, IL, United States Duration: 13 Jul 2008 → 13 Jul 2008

Publication series

Name	AAAI Workshop - Technical Report
Volume	WS-08-15

Conference

Conference	2008 AAAI Workshop
Country/Territory	United States
City	Chicago, IL
Period	13/07/08 → 13/07/08

ASJC Scopus subject areas

General Engineering

Cite this

@inproceedings{ae35003424224ca8961eee0be5413c2c,

title = "Using wikipedia links to construct word segmentation corpora",

abstract = "Tagged corpora are essential for evaluating and training natural language processing tools. The cost of constructing large enough manually tagged corpora is high, even when the annotation level is shallow. This article describes a simple method to automatically create a partially tagged corpus, using Wikipedia hyperlinks. The resulting corpus contains information about the correct segmentation of 523,599 non-consecutive words in 363,090 sentences. We used our method to construct a corpus of Modern Hebrew (which we have made available at http://www.cs.bgu.ac.il/-nlpproj). The method can also be applied to other languages where word segmentation is difficult to determine, such as East and South-East Asian languages.",

author = "David Gabay and Ziv, \{Ben Eliahu\} and Michael Elhadad",

year = "2008",

month = dec,

day = "1",

language = "English",

isbn = "9781577353836",

series = "AAAI Workshop - Technical Report",

pages = "61--63",

booktitle = "Wikipedia and Artificial Intelligence",

note = "2008 AAAI Workshop ; Conference date: 13-07-2008 Through 13-07-2008",

}

Using wikipedia links to construct word segmentation corpora. / Gabay, David; Ziv, Ben Eliahu; Elhadad, Michael.
Wikipedia and Artificial Intelligence: An Evolving Synergy - Papers from the 2008 AAAI Workshop. 2008. p. 61-63 (AAAI Workshop - Technical Report; Vol. WS-08-15).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Using wikipedia links to construct word segmentation corpora

AU - Gabay, David

AU - Ziv, Ben Eliahu

AU - Elhadad, Michael

PY - 2008/12/1

Y1 - 2008/12/1

N2 - Tagged corpora are essential for evaluating and training natural language processing tools. The cost of constructing large enough manually tagged corpora is high, even when the annotation level is shallow. This article describes a simple method to automatically create a partially tagged corpus, using Wikipedia hyperlinks. The resulting corpus contains information about the correct segmentation of 523,599 non-consecutive words in 363,090 sentences. We used our method to construct a corpus of Modern Hebrew (which we have made available at http://www.cs.bgu.ac.il/-nlpproj). The method can also be applied to other languages where word segmentation is difficult to determine, such as East and South-East Asian languages.

AB - Tagged corpora are essential for evaluating and training natural language processing tools. The cost of constructing large enough manually tagged corpora is high, even when the annotation level is shallow. This article describes a simple method to automatically create a partially tagged corpus, using Wikipedia hyperlinks. The resulting corpus contains information about the correct segmentation of 523,599 non-consecutive words in 363,090 sentences. We used our method to construct a corpus of Modern Hebrew (which we have made available at http://www.cs.bgu.ac.il/-nlpproj). The method can also be applied to other languages where word segmentation is difficult to determine, such as East and South-East Asian languages.

UR - https://www.scopus.com/pages/publications/70449775446

M3 - Conference contribution

AN - SCOPUS:70449775446

SN - 9781577353836

T3 - AAAI Workshop - Technical Report

SP - 61

EP - 63

BT - Wikipedia and Artificial Intelligence

T2 - 2008 AAAI Workshop

Y2 - 13 July 2008 through 13 July 2008

ER -

Using wikipedia links to construct word segmentation corpora

Abstract

Publication series

Conference

ASJC Scopus subject areas

Fingerprint

Cite this