文章基本信息

标题：Evaluation of Language Models over Croatian Newspaper Texts
其他标题：Evaluation of Language Models over Croatian Newspaper Texts
本地全文：下载
作者：Beliga, Slobodan ; Ipšić, Ivo ; Martinčić-Ipšić, Sanda 等
期刊名称：Engineering Economics
印刷版ISSN：2029-5839
出版年度：2017
卷号：46
期号：4
页码：425-444
DOI：10.5755/j01.itc.46.4.18367
语种：English
出版社：Kaunas University of Technology
摘要：Statistical language modeling involves techniques and procedures that assign probabilities to word sequences or, said in other words, estimate the regularity of the language. This paper presents basic characteristics of statistical language models, reviews their use in the large set of speech and language applications, explains their formal definition and shows different types of language models. Detailed overview of n-gram and class-based models (as well as their combinations) is given chronologically, by type and complexity of models, and in aspect of their use in different NLP applications for different natural languages. The proposed experimental procedure compares three different types of statistical language models: n-gram models based on words, categorical models based on automatically determined categories and categorical models based on POS tags. In the paper, we propose a language model for contemporary Croatian texts, a procedure how to determine the best n-gram and the optimal number of categories, which leads to significant decrease of language model perplexity, estimated from the Croatian News Agency articles (HINA) corpus. Using different language models estimated from the HINA corpus, we show experimentally that models based on categories contribute to a better description of the natural language than those based on words. These findings of the proposed experiment are applicable, except for Croatian, for similar highly inflectional languages with rich morphology and non-mandatory sentence word order.DOI: http://dx.doi.org/10.5755/j01.itc.46.4.18367
关键词：Statistical language model; Natural language regularity; Word-based language model; Category-based language model; Brown algorithm; POS class; N-gram; Perplexity; Croatian corpora