文章基本信息

标题：A Study of Analogical Density in Various Corpora at Various Granularity
本地全文：下载
作者：Rashel Fam ; Yves Lepage
期刊名称：Information
电子版ISSN：2078-2489
出版年度：2021
卷号：12
期号：8
页码：314
DOI：10.3390/info12080314
语种：English
出版社：MDPI Publishing
摘要：In this paper, we inspect the theoretical problem of counting the number of analogies between sentences contained in a text. Based on this, we measure the analogical density of the text. We focus on analogy at the sentence level, based on the level of form rather than on the level of semantics. Experiments are carried on two different corpora in six European languages known to have various levels of morphological richness. Corpora are tokenised using several tokenisation schemes: character, sub-word and word. For the sub-word tokenisation scheme, we employ two popular sub-word models: unigram language model and byte-pair-encoding. The results show that the corpus with a higher Type-Token Ratio tends to have higher analogical density. We also observe that masking the tokens based on their frequency helps to increase the analogical density. As for the tokenisation scheme, the results show that analogical density decreases from the character to word. However, this is not true when tokens are masked based on their frequencies. We find that tokenising the sentences using sub-word models and masking the least frequent tokens increase analogical density.