informИнформатикаInformatics1816-03012617-6963UIIP NASB10.37661/1816-0301-2022-19-1-96-110inform-1167Research ArticleИНТЕЛЛЕКТУАЛЬНЫЕ СИСТЕМЫINTELLIGENT SYSTEMSПрименение модели освоения языка к решению задачи обработки малых языковApplying the language acquisition model to the solution small language processing tasksКачковД. И.KachkouDz. I.

Качков Дмитрий Ильич - аспирант кафедры многопроцессорных систем и  сетей факультета прикладной математики и информатики.

пр. Независимости, 4, Минск, 220030.

Dzmitry I. Kachkou - Postgraduate Student of Department of Multiprocessor Systems and Networks of the Faculty of Applied Mathematics and Informatics, Belarusian State University.

Nezavisimosti av., 4, Minsk, 220030.

dmitriydikanskiy@gmail.comБелорусский государственный университетBelarusian State University20220501202219196110Copyright © Качков Д.И., 20222022Качков Д.И.Kachkou D.I.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://inf.grid.by/jour/article/view/1167

Решается задача построения компьютерной модели малого языка. Ее актуальность обусловлена необходимостью устранения информационного неравенства между носителями различных языков, востребованностью новых инструментов для исследования малоизученных языков и инновационных подходов к моделированию языка в условиях дефицита ресурсов, необходимостью поддержки и развития языков малых народов.

При решении задачи обработки малых языков на этапе описания проблемной ситуации преследуются три основные цели: обоснование проблемы моделирования языка в условиях дефицита ресурсов как особой задачи в сфере моделирования естественных языков, обзор литературы по соответствующей теме и разработка концепции модели усвоения языка с относительно малым числом доступных ресурсов. Используются методы компьютерного моделирования с применением нейронных сетей, обучение с частичным привлечением учителя и обучение с подкреплением.

В  работе  приведен обзор  литературы, посвященной моделированию  изучения  лексики,  морфологии и грамматики родного языка ребенком. На основании современных представлений о ходе изучения языка предложена архитектура системы обработки малого языка, которая при обучении опирается на компьютерное моделирование онтогенеза. Выделены основные компоненты системы и принципы их взаимодействия. В основе системы лежит модуль, построенный на базе современных диалоговых языковых моделей  и  обученный на  каком-либо крупном языке,  например английском. При обучении используется промежуточный слой, который представляет высказывания в некотором абстрактном виде, например, в символах формальной семантики. Соотношение между формальной записью высказываний и их переводом на целевой малый язык изучается методом моделирования процесса усвоения лексики и грамматики языка ребенком. Отдельный компонент имитирует неязыковой контекст, в котором происходит изучение языка.

В статье исследуется задача моделирования малых языков. Дано подробное обоснование актуальности моделирования малых языков: показана социальная значимость этой проблемы, польза ее решения для лингвистики, этнографии, этнологии и культурной антропологии. Отмечена неэффективность подходов, применяемых к крупным языкам, в условиях дефицита ресурсов. Предложена модель изучения языка с помощью имитации онтогенеза, которая опирается как на полученные результаты в области компьютерного моделирования, так и на данные психолингвистики.

The problem of building a computer model of a small language was under solution. The relevance of this task is due to the following considerations: the need to eliminate the information inequality between speakers of different languages; the need for new tools for the study of poorly understood languages, as well as innovative approaches to language modeling in the low-resource context; the problem of supporting and developing small languages.

There are three main objectives in solving the problem of small natural language processing at the stage of describing the problem situation: to justify the problem of modeling language in the context of resource scarcity as a special task in the field of natural languages processing, to review the literature on the relevant topic, to develop the concept of language acquisition model with a relatively small number of available resources. Computer modeling techniques using neural networks, semi-supervised learning and reinforcement learning were involved.

The paper provides a review of the literature on modeling the learning of vocabulary, morphology, and grammar of a child's native language. Based on the current understanding of the language acquisition and existing computer models of this process, the architecture of the system of small language processing, which is taught through modeling of ontogenesis, is proposed. The main components of the system and the principles of their interaction are highlighted. The system is based on a module built on the basis of modern dialogical language models and taught in some rich-resources language (e.g., English). During training, an intermediate layer is used which represents statements in some abstract form, for example, in the symbols of formal semantics. The relationship between the formal recording of utterances and their translation into the target low-resource language is learned by modeling the child's acquisition of vocabulary and grammar of the language. One of components stands for the non-linguistic context in which language learning takes place.

This article explores the problem of modeling small languages. A detailed substantiation of the relevance of modeling small languages is given: the social significance of the problem is noted, the benefits for linguistics, ethnography, ethnology and cultural anthropology are shown. The ineffectiveness of approaches applied to large languages in conditions of a lack of resources is noted. A model of language learning by means of ontogenesis simulation is proposed, which is based both on the results obtained in the field of computer modeling and on the data of psycholinguistics.

информационные технологииязыковые моделиобработка малого языкаусвоение языкаобучение с подкреплениемнейронные сетиархитектура Transformerinformation technologylanguage modelslow-resource language processinglanguage acquisitionreinforcement learningneural networksTransformer architectureReferencesA Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios [Electronic resource] / M. A. Hedderich [et al.]. – 2020. – Mode of access: https://arxiv.org/abs/2010.12309. – Date of access: 12.10.2021.Hedderich M. A., Lange L., Adel H., Strötgen J., Klakow D. A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios, 2020. Available at: https://arxiv.org/abs/2010.12309 (accessed 12.10.2021).Dai, A. M. Semi-supervised sequence learning [Electronic resource] / A. M. Dai, Q. V. Le // Proc. of the 28th Intern. Conf. on Neural Information Processing Systems. – 2015. – Vol. 2. – P. 3079–3087. https://doi.org/10.18653/v1/P17-1161Dai A. M., Le Q. V. Semi-supervised sequence learning. Proceedings of the 28th International Conference on Neural Information Processing Systems, 2015, vol. 2, pp. 3079–3087. https://doi.org/10.18653/v1/P17-1161TICO-19: the translation initiative for Covid-19 [Electronic resource] / A. Anastasopoulos [et al.] // Proc. of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020. – Dec. 2020. – Mode of access: https://aclanthology.org/2020.nlpcovid19-2.5/. – Date of access: 12.10.2021. https://doi.org/10.18653/v1/2020.nlpcovid19-2.5Anastasopoulos A., Cattelan A., Dou Z.-Y., Federico M., Federman C., …, Tur S. TICO-19: the translation initiative for Covid-19. Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020. December 2020. Available at: https://aclanthology.org/2020.nlpcovid19-2.5/ (accessed 12.10.2021). https://doi.org/10.18653/v1/2020.nlpcovid19-2.5Enabling low-resource transfer learning across Covid-19 corpora by combining event-extraction and co-training / A. Spangher [et al.] // Proc. of the 1st Workshop on NLP for COVID-19 at ACL 2020. – July 2020. – Mode of access: https://aclanthology.org/2020.nlpcovid19-acl.4/. – Date of access: 12.10.2021.Spangher A., Peng N., May J., Ferrara E. Enabling low-resource transfer learning across Covid-19 corpora by combining event-extraction and co-training. Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020. July 2020. Available at: https://aclanthology.org/2020.nlpcovid19-acl.4/ (accessed 12.10.2021).Attention is all you need / A. Vaswani [et al.] // Proc. of the 31st Intern. Conf. on Neural Information Processing Systems, Long Beach, California, USA, 4–9 Dec. 2017. – Long Beach, 2017. – P. 6000–6010.Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., …, Polosukhin I. Attention is all you need. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, California, USA, 4–9 December 2017. Long Beach, 2017, pp. 6000–6010.Качков, Д. И. Моделирование языка и двунаправленные представления кодировщиков: обзор ключевых технологий / Д. И. Качков // Информатика. – 2020. – Т. 17, № 4. – С. 61–72. https://doi.org/10.37661/1816-0301-2020-17-4-61-72Kachkou D. I. Language modeling and bidirectional coders representations: an overview of key technologies. Informatika [Informatics], 2020, vol. 17, no. 4, pp. 61−72 (In Russ.). https://doi.org/10.37661/1816-0301-2020-17-4-61-72Cloze-driven pretraining of self-attention networks / A. Baevski [et al.] // Proc. of the 2019 Conf. on Empirical Methods in Natural Language Processing and the 9th Intern. Joint Conf. on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 3–7 Nov. 2019. – Hong Kong, 2019. – P. 5360–5369. https://doi.org/10.18653/v1/D19-1539Baevski A., Edunov S., Liu Y., Zettlemoyer L., Auli M. Cloze-driven pretraining of self-attention networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 3–7 November 2019. Hong Kong, 2019, pp. 5360–5369. https://doi.org/10.18653/v1/D19-1539RoBERTa: A Robustly Optimized BERT Pretraining Approach [Electronic resource] / Y. Liu [et al.]. – 2019. – Mode of access: https://arxiv.org/abs/1907.11692. – Date of access: 12.10.2021.Liu Y., Ott M., Goyal N., Du J., Joshi M., …, Stoyanov V. RoBERTa: A Robustly Optimized BERT Pretraining Approach, 2019. Available at: https://arxiv.org/abs/1907.11692 (accessed 12.10.2021).Замятин, К. Как и зачем сохранять языки народов России / К. Замятин, А. Пасанен, Я. Саарикиви. – Хельсинки, 2012. – 181 с.Zamyatin K., Pasanen A., Saarikivi Ya. Kak i zachem sohranyat’ yazyki narodov Rossii. How and Why to Save Languages of Ethnic Groups in Russia. Helsinkim, 2012, 181 p. (In Russ.).Meisel, J. M. First and Second Language Acquisition (Cambridge Textbooks in Linguistics) / J. M. Meisel. – Cambridge University Press, 2011. – 318 р.Meisel J. M. First and Second Language Acquisition (Cambridge Textbooks in Linguistics). Cambridge University Press, 2011, 318 р.Clark, E. V. First Language Acquisition / E. V. Clark. – Cambridge University Press, 2009. – 2nd ed. – 490 р.Clark E. V. First Language Acquisition. Cambridge University Press, 2nd ed., 2009, 490 р.Лурия, А. Р. Язык и сознание / А. Р. Лурия ; под ред. Е. Д. Хомской. – М. : Изд-во Моск. ун-та, 1979. – 320 с.Luriya A. R. Yazyk i soznanie. Language and Conscience. In Homskaya E. D. (ed.). Moscow, Izdatel'stvo Moskovskogo universtiteta, 1979, 320 p. (In Russ.).Бурлак, С. А. Происхождение языка. Факты, исследования, гипотезы / С. А. Бурлак. – М. : Альпина Диджитал, 2019. – 609 с.Burlak S. A. Proishozhdenie yazyka. Fakty, issledovaniya, gipotezy. Origin of the language. Facts, Researches, Hypothesis. Moscow, Alpina digital, 2019, 609 р. (In Russ.).Немов, Р. С. Общая психология в 3 т. Том II в 4 кн. Книга 4. Речь. Психические состояния : учебник и практикум для академического бакалавриата / Р. С. Немов. – 6-е изд., перераб. и доп. – М. : Юрайт, 2017. – 243 с.Nemov R. S. Obshchaya psihologiya. General Psychology. Vol. 2, kniga 4. Rech’. Psihicheskie sostoyaniya: uchebnik i praktikum dlya akademicheskogo bakalavriata. Speech. Psychological States: Textbook and Workshop for Bachelors. 6th ed., Moscow, Yurait, 2017, 243 p. (In Russ.).Evans, V. The Language Myth Why Language Is Not an Instinct / V. Evans. – Cambridge University Press, 2014. – 314 р.Evans V. The Language Myth Why Language Is Not an Instinct. Cambridge University Press, 2014, 314 р.Пирс, Ч. С. Принципы философии : в 2 т. / Ч. С. Пирс ; пер. с англ. В. В. Кирющенко, М. В. Колопотина. – СПб. : Санкт-Петербургское философское общество, 2001. – Т. 2. – 313 с.Peirce C. S. Collected Papers of Charles Sanders Peirce, Volumes I and II: Principles of Philosophy and Elements of Logic. Belknap Press, 1932, vol. II, 535 p.Виноград, Т. Программа, понимающая естественный язык / Т. Виноград. – М. : Мир, 1976. – 296 с.Winograd T. Programma, ponimaushchaya estestvennyj yazyk. Understanding Natural Language. Moscow, Mir, 1976, 296 p. (In Russ.).VQA: visual question answering / S. Antol [et al.] // IEEE Intern. Conf. on Computer Vision (ICCV). – Santiago, Chile, 2015. – P. 2425–2433. https://doi.org/10.1109/ICCV.2015.279Antol S., Agrawal A., Lu J., Mitchell M., Batra D., …, Parikh D. VQA: visual question answering. IEEE International Conference on Computer Vision (ICCV). Santiago, Chile, 2015, pp. 2425–2433. https://doi.org/10.1109/ICCV.2015.279Embodied question answering / A. Das [et al.] // Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–23 June 2018. – Salt Lake City, 2018. – P. 1–10.Das A., Datta S., Gkioxari G., Lee S., Parikh D., Batra D. Embodied question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–23 June 2018. Salt Lake City, 2018, pp. 1–10.A survey of reinforcement learning informed by natural language / J. Luketina [et al.] // Proc. of the Twenty-Eighth Intern. Joint Conf. on Artificial Intelligence, Macao, China, 10–16 Aug. 2019. – Macao, 2019. – P. 6309–6317. https://doi.org/10.24963/ijcai.2019/880Luketina J., Nardelli N., Farquhar G., Foerster J., Andreas J., …, Rocktäschel T. A survey of reinforcement learning informed by natural language. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, Macao, China, 10–16 August 2019. Macao, 2019, pp. 6309–6317. https://doi.org/10.24963/ijcai.2019/880Janner, M. Representation learning for grounded spatial reasoning / M. Janner, K. Narasimhan, R. Barzilay // Transactions of the Association for Computational Linguistics. – 2018. – Vol. 6. – P. 49–61. https://doi.org/10.1162/tacl_a_00004Janner M., Narasimhan K., Barzilay R. Representation learning for grounded spatial reasoning. Transactions of the Association for Computational Linguistics, 2018, vol. 6, pp. 49–61. https://doi.org/10.1162/tacl_a_00004Côté, M.-A. TextWorld: A learning environment for text-based games / M.-A. Côté ; T. Cazenave, A. Saffidine, N. Sturtevant (eds.) // Computer Games. CGW 2018. Communications in Computer and Information Science. – Cham : Springer, 2018. – Vol. 1017. – Р. 41–75. https://doi.org/10.1007/978-3-030-24337-1_3Côté M.-A., Kádár Á., Yuan X., Kybartas B., Barnes T., …, Trischler A. TextWorld: A learning environment for text-based games. Computer Games. CGW 2018. Communications in Computer and Information Science, Cham, Springer, 2018, vol. 1017, рр. 41–75. https://doi.org/10.1007/978-3-030-24337-1_3Arora, S. A survey of inverse reinforcement learning: Challenges, methods and progress [Electronic resource] / S. Arora, P. Doshi // Artificial Intelligence. – 2021. – Vol. 297. – Mode of access: https://arxiv.org/abs/1806.06877. – Date of access: 12.10.2021. https://doi.org/10.1016/j.artint.2021.103500Arora S., Doshi P. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 2021, vol. 297. Available at: https://arxiv.org/abs/1806.06877 (accessed 12.10.2021). https://doi.org/10.1016/j.artint.2021.103500A general reinforcement learning algorithm that masters chess, shogi, and go through self-play / D. Silver [et al.] // Science. – 2018. – Vol. 362, no. 6419. – P. 1140–1144. https://dx.doi.org/10.1126%2Fscience.aar6404Silver D., Hubert T., Schrittwieser J., Antonoglou I., Lai M., …, Hassabis D. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 2018, vol. 362, no. 6419, pp. 1140–1144. https://dx.doi.org/10.1126%2Fscience.aar6404Freudenthal, D. Computational models of language development / D. Freudenthal, A. Alishahi ; P. J. Brooks, V. Kempe (eds.) // Encyclopedia of Language Development. – 1st ed. – SAGE Publications Inc., 2014. – P. 92–96.Freudenthal D., Alishahi A. Computational models of language development. Encyclopedia of Language Development. In Brooks P. J., Kempe V. (eds.). 1st ed., SAGE Publications Inc., 2014, pp. 92–96.Fazly, A. A probabilistic computational model of cross‐situational word learning / A. Fazly, A. Alishahi, S. Stevenson // Cognitive Science. – 2010. – Vol. 34, iss. 6. – P. 1017–1063. https://doi.org/10.1111/j.1551-6709.2010.01104.xFazly A., Alishahi A., Stevenson S. A probabilistic computational model of cross‐situational word learning. Cognitive Science, 2010, vol. 34, iss. 6, pp. 1017–1063. https://doi.org/10.1111/j.1551-6709.2010.01104.xChristiansen, M. H. Connectionist natural language processing: the state of the art / M. H. Christiansen, N. Chater // Cognitive Science. – 1999. – Vol. 23, iss. 4. – P. 417–437. https://doi.org/10.1207/s15516709cog2304_2Christiansen M. H., Chater N. Connectionist natural language processing: the state of the art. Cognitive Science, 1999, vol. 23, iss. 4, pp. 417–437. https://doi.org/10.1207/s15516709cog2304_2Buttery, P. J. Computational models for first language acquisition / P. J. Buttery // Technical Report UCAM-CL-TR-675. – University of Cambridge, 2006. – Mode of access: https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-675.pdf. – Date of access: 21.03.2021.Buttery P. J. Computational models for first language acquisition. Technical Report UCAM-CL-TR-675, University of Cambridge, 2006. Available at: https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-675.pdf (accessed 21.03.2021).MacWhinney, B. The CHILDES Project: Tools for Analyzing Talk: Transcription Format and Programs (3rd ed.) / B. MacWhinney. – Lawrence Erlbaum Associates Publishers, 2000.MacWhinney, B. The CHILDES Project: Tools for Analyzing Talk: Transcription Format and Programs. 3rd ed., Lawrence Erlbaum Associates Publishers, 2000.Jones, G. A process model of children’s early verb use / G. Jones, F. Gobet, J. M. Pine // Proc. of the 22th Annual Conf. of the Cognitive Science Society, Philadelphia, PA, 13–15 Aug. 2000. – Philadelphia, 2000. – P. 723–728.Jones G., Gobet F., Pine J. M. A process model of children’s early verb use. Proceedings of the 22th Annual Conference of the Cognitive Science Society, Philadelphia, PA, 13–15 August 2000. Philadelphia, 2000, pp. 723–728.Alishahi, A. Computational Modeling of Human Language Acquisition / A. Alishahi. – Morgan & Claypool, 2010. – 107 p.Alishahi A. Computational Modeling of Human Language Acquisition. Morgan & Claypool, 2010, 107 р.Andersen, E. S. The impact of input: language acquisition in the visually impaired / E. S. Andersen, A. Dunlea, L. Kekelis // First Language. – 1993. – Vol. 13, no. 37. – P. 23–49. https://doi.org/10.1177/014272379301303703Andersen E. S, Dunlea A., Kekelis L. The impact of input: language acquisition in the visually impaired. First Language, 1993, vol. 13, no. 37, pp. 23–49. https://doi.org/10.1177/014272379301303703Vlasov, V. Dialogue Transformers [Electronic resource] / V. Vlasov, J. E. M. Mosig, A. Nicho. – 2019. – Mode of access: https://arxiv.org/abs/1910.00486. – Date of access: 12.10.2021.Vlasov V., Mosig J. E. M., Nicho A. Dialogue Transformers, 2019. Available at: https://arxiv.org/abs/1910.00486 (accessed 12.10.2021).Андреев, А. В. Введение в формальную семантику : учеб. пособие / А. В. Андреев, О. А. Митрофанова, К. В. Соколов. – СПб. : СПбГУ, 2014. – 88 с.Andreev A. V., Mitrofanova O. A., Sokolov K. V. Vvedenie v formal’nuyu semantiky: uchebnoe posobie. Introduction Into Formal Semantics: Handbook. Saint-Petersburg, Saint-Petersburg State University, 2014, 88 p. (In Russ.).Goddard, C. The search for the shared semantic core of all languages / C. Goddard ; C. Goddard, A. Wierzbicka (eds.) // Meaning and Universal Grammar – Theory and Empirical Findings. – Amsterdam : John Benjamins, 2002. – Vol. I. – P. 5–40.Goddard C. The search for the shared semantic core of all languages. In Goddard C., Wierzbicka A. (eds.). Meaning and Universal Grammar – Theory and Empirical Findings. Amsterdam, John Benjamins, 2002, vol. I, pp. 5–40.Barnes, J. Evidentials in the Tuyuca Verb / J. Barnes // Intern. J. of American Linguistics. – 1984. – Vol. 50, no. 3. – P. 255–271.Barnes J. Evidentials in the tuyuca verb. International Journal of American Linguistics, 1984, vol. 50, no. 3, pp. 255–271.

The authors declare that there are no conflicts of interest present.