informИнформатикаInformatics1816-03012617-6963UIIP NASBinform-358Research ArticleИНТЕЛЛЕКТУАЛЬНЫЕ СИСТЕМЫINTELLIGENT SYSTEMSАВТОМАТИЧЕСКОЕ ОПРЕДЕЛЕНИЕ ЯЗЫКА ТЕКСТОВОГО ДОКУМЕНТА ДЛЯ ОСНОВНЫХ ЕВРОПЕЙСКИХ ЯЗЫКОВКрапивинЮ. Б.Брестский государственный технический университетBelarus20111904201803(31)112117Copyright © Крапивин Ю.Б., 20182018Крапивин Ю.Б.Крапивин Ю.Б.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://inf.grid.by/jour/article/view/358

Проводится анализ основных методов решения задачи автоматического определения языка текстового документа и предлагается алгоритм, основанный на комбинировании алфавитного метода, метода грамматических слов и алфавитно-триграммного метода, сочетающий в себе возможности минимального статистического и лингвистического анализа языковых данных и обеспечивающий эффективное решение указанной задачи.

ReferencesКрапивин, Ю.Б. К задаче автоматического распознавания воспроизведенных фрагментов текстовых документов / Ю.Б. Крапивин // Вестник БрГТУ : Физика, математика, информатика. – 2009. – № 5 (59). – С. 120–123.Крапивин, Ю.Б. К задаче автоматического распознавания воспроизведенных фрагментов текстовых документов / Ю.Б. Крапивин // Вестник БрГТУ : Физика, математика, информатика. – 2009. – № 5 (59). – С. 120–123.Grefenstette, G. Comparing two language identification schemes / G. Grefenstette // TheGrefenstette, G. Comparing two language identification schemes / G. Grefenstette // TheThird Intern. Conf. on Statistical Analysis of Textual Data. – Rome, 1995.Third Intern. Conf. on Statistical Analysis of Textual Data. – Rome, 1995.Sibun, P. Language Determination: Natural Language Processing from Scanned DocumentSibun, P. Language Determination: Natural Language Processing from Scanned DocumentImages / P. Sibun, A.L. Spitz // Proc. of the 4th ACL Conf. on Applied Natural Language Proceeding (ANLP). – Stuttgart, Germany, 1994.Images / P. Sibun, A.L. Spitz // Proc. of the 4th ACL Conf. on Applied Natural Language Proceeding (ANLP). – Stuttgart, Germany, 1994.Cowie, J. Language recognition for mono- and multilingual documents / J. Cowie,Cowie, J. Language recognition for mono- and multilingual documents / J. Cowie,Y. Ludovic, R. Zacharski // Proc. of the Vextal Conference. – Venice, 1999.Y. Ludovic, R. Zacharski // Proc. of the Vextal Conference. – Venice, 1999.Natural Language Identification using Corpus-based Models / C. Souter [et al.] // HermesNatural Language Identification using Corpus-based Models / C. Souter [et al.] // HermesJournal of Linguistics. – 1994. – № 13. – P. 183–203.Journal of Linguistics. – 1994. – № 13. – P. 183–203.Cavnar, W.B. N-Gram-Based Text Categorization / W.B. Cavnar, J.M. Trenkle // Proc. of theCavnar, W.B. N-Gram-Based Text Categorization / W.B. Cavnar, J.M. Trenkle // Proc. of therd Annual Symposium on Document Analysis and Information Retrieval (SDAIR). – Las Vegas, 1994. – P. 161–175.rd Annual Symposium on Document Analysis and Information Retrieval (SDAIR). – Las Vegas, 1994. – P. 161–175.Prager, J.M. Linguini: Language identification for multilingual documents / J.M. Prager //Prager, J.M. Linguini: Language identification for multilingual documents / J.M. Prager //Proc. of the 32nd Hawaii Intern. Conf. on System Sciences. – Maui, Hawaii, USA, 1999.Proc. of the 32nd Hawaii Intern. Conf. on System Sciences. – Maui, Hawaii, USA, 1999.Dunning, T. Statistical Identification of Language / T. Dunning // Computing Research Laboratory.Dunning, T. Statistical Identification of Language / T. Dunning // Computing Research Laboratory.Technical report MCCS. – New Mexico State University, 1994. – P. 94–273.Technical report MCCS. – New Mexico State University, 1994. – P. 94–273.Sibun, P. Language identification: Examing the issues / P. Sibun, J.C. Reynar // Proc. of theSibun, P. Language identification: Examing the issues / P. Sibun, J.C. Reynar // Proc. of theth Symposium on Document Analysis and Information Retrieval (SDAIR). – Las Vegas, 1996. – P. 125–135.th Symposium on Document Analysis and Information Retrieval (SDAIR). – Las Vegas, 1996. – P. 125–135.Poutsma, A. Applying MonteCarlo Techniques to Language Identification / A. Poutsma //Poutsma, A. Applying MonteCarlo Techniques to Language Identification / A. Poutsma //Proc. of Computational Linguistics in the Netherlands. – Amsterdam, Netherlands, 2001.Proc. of Computational Linguistics in the Netherlands. – Amsterdam, Netherlands, 2001.Biemann, C. Disentangling from Babylonian Confusion – Unsupervised Language Identification / C. Biemann, S. Teresniak // Proc. of the CICLing-2005. – Mexico City, 2005.Biemann, C. Disentangling from Babylonian Confusion – Unsupervised Language Identification / C. Biemann, S. Teresniak // Proc. of the CICLing-2005. – Mexico City, 2005.Kruengkrai, C. Language Identification Based on String Kernels / C. Kruengkrai // Proc. ofKruengkrai, C. Language Identification Based on String Kernels / C. Kruengkrai // Proc. ofthe 5th Intern. Symposium on Communications and Information Technologies (ISCIT-2005). – Beijing, China, 2005.the 5th Intern. Symposium on Communications and Information Technologies (ISCIT-2005). – Beijing, China, 2005.Giguet, E. Categorization according to Language: A step toward combining LinguisticGiguet, E. Categorization according to Language: A step toward combining LinguisticKnowledge and Statistic Learning / E. Giguet // 4th Intern. Workshop of Parsing Technologies. – Prague, Karlovy Vary, Czech Republic, 1995.Knowledge and Statistic Learning / E. Giguet // 4th Intern. Workshop of Parsing Technologies. – Prague, Karlovy Vary, Czech Republic, 1995.Newman, P. Foreign language identification: First step in the translation process / P. Newman // Proc. of the 28th Annual Conf. of the American Translators Accociation. – Albuquerque NM, USA, 1987. – P. 509–516.Newman, P. Foreign language identification: First step in the translation process / P. Newman // Proc. of the 28th Annual Conf. of the American Translators Accociation. – Albuquerque NM, USA, 1987. – P. 509–516.Kullback – Leibler_divergence [Electronic resource] // Wikipedia. – Mode of access :Kullback – Leibler_divergence [Electronic resource] // Wikipedia. – Mode of access :http://en.wikipedia.org/wiki/ Kullback-Leibler_divergence. – Date of access : 15.12.2010.http://en.wikipedia.org/wiki/ Kullback-Leibler_divergence. – Date of access : 15.12.2010.Ukkonen, E. On-line construction of suffix trees / E. Ukkonen // Algorithmica. – 1995. –Ukkonen, E. On-line construction of suffix trees / E. Ukkonen // Algorithmica. – 1995. –№ 14 (3). – P. 249–260.№ 14 (3). – P. 249–260.Function word [Electronic resource] // Wikipedia. – Mode of access : http://en.wikipedia.org/wiki/Function_word. – Date of access : 14.12.2010.Function word [Electronic resource] // Wikipedia. – Mode of access : http://en.wikipedia.org/wiki/Function_word. – Date of access : 14.12.2010.Quasthoff, U. Corpus Portal for Search in Monolingual Corpora / U. Quasthoff, M. Richter,Quasthoff, U. Corpus Portal for Search in Monolingual Corpora / U. Quasthoff, M. Richter,C. Biemann // Proc. of the Fifth Intern. Conf. on Language Resources and Evaluation, LREC 2006. – Genoa, 2006. – P. 1799–1802.C. Biemann // Proc. of the Fifth Intern. Conf. on Language Resources and Evaluation, LREC 2006. – Genoa, 2006. – P. 1799–1802.Lancaster-Oslo-Bergen Corpus [Electronic resource] // Wikipedia. – Mode of access :Lancaster-Oslo-Bergen Corpus [Electronic resource] // Wikipedia. – Mode of access :http://en.wikipedia.org/wiki/LOB_Corpus. – Date of access : 15.12.2010.http://en.wikipedia.org/wiki/LOB_Corpus. – Date of access : 15.12.2010.

The authors declare that there are no conflicts of interest present.