Statistical language models of Lithuanian based on word clustering and morphological decomposition
Date |
---|
2004 |
This paper describes our research on statistical language modeling of Lithuanian. The idea of improving sparse n-gram models of highly inflected Lithuanian language by interpolating them with complex n-gram models based on word clustering and morphological word decomposition was investigated. Words, word base forms and part-of-speech tags were clustered into 50 to 5000 automatically generated classes. Multiple 3-gram and 4-gram class-based language models were built and evaluated on Lithuanian text corpus, which contained 85 million words. Class-based models linearly interpolated with the 3-gram model led up to a 13% reduction in the perplexity compared with the baseline 3-gram model. Morphological models decreased out-of-vocabulary word rate from 1.5% to 1.02%.
Journal | IF | AIF | AIF (min) | AIF (max) | Cat | AV | Year | Quartile |
---|---|---|---|---|---|---|---|---|
INFORMATICA | 0.26 | 0.847 | 0.665 | 1.029 | 2 | 0.346 | 2004 | Q4 |
Journal | IF | AIF | AIF (min) | AIF (max) | Cat | AV | Year | Quartile |
---|---|---|---|---|---|---|---|---|
INFORMATICA | 0.26 | 0.847 | 0.665 | 1.029 | 2 | 0.346 | 2004 | Q4 |