TY - GEN
T1 - Language classification and segmentation of noisy documents in Hebrew scripts
AU - Dershowitz, Nachum
AU - Zhicharevich, Alex
N1 - Publisher Copyright:
© 2012 Association for Computational Linguistics.
PY - 2012
Y1 - 2012
N2 - Language classification is a preliminary step for most natural-language related processes. The significant quantity of multilingual documents poses a problem for traditional language-classification schemes and requires segmentation of the document to monolingual sections. This phenomenon is characteristic of classical and medieval Jewish literature, which frequently mixes Hebrew, Aramaic, Judeo-Arabic and other Hebrew-script languages. We propose a method for classification and segmentation of multi-lingual texts in the Hebrew character set, using bigram statistics. For texts, such as the manuscripts found in the Cairo Genizah, we are also forced to deal with a significant level of noise in OCR-processed text.
AB - Language classification is a preliminary step for most natural-language related processes. The significant quantity of multilingual documents poses a problem for traditional language-classification schemes and requires segmentation of the document to monolingual sections. This phenomenon is characteristic of classical and medieval Jewish literature, which frequently mixes Hebrew, Aramaic, Judeo-Arabic and other Hebrew-script languages. We propose a method for classification and segmentation of multi-lingual texts in the Hebrew character set, using bigram statistics. For texts, such as the manuscripts found in the Cairo Genizah, we are also forced to deal with a significant level of noise in OCR-processed text.
UR - http://www.scopus.com/inward/record.url?scp=85052016810&partnerID=8YFLogxK
M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???
AN - SCOPUS:85052016810
T3 - Proceedings of the 6th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, LaTeCH 2012 at the 13th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2012
SP - 112
EP - 117
BT - Proceedings of the 6th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, LaTeCH 2012 at the 13th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2012
A2 - Zervanou, Kalliopi
A2 - van den Bosch, Antal
PB - Association for Computational Linguistics (ACL)
T2 - 6th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, LaTeCH 2012 at the 13th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2012
Y2 - 24 April 2012
ER -