Understanding Syntax in Large Language Models: Successes and Limitations
DOI:
https://doi.org/10.59992/IJESA.2025.v4n1p1Keywords:
Syntax, Large Language Models, Natural Language Processing, Syntactic Competence, Neural Language Models, Computational LinguisticsAbstract
This study investigates the ability of Large Language Models (LLMs) to process complex syntactic phenomena, including relative clauses, wh-movement, and center- embedding. By analyzing examples derived from linguistic literature, the study highlights both the strengths and limitations of LLMs in handling syntax. The results reveal that while LLMs exhibit competence in simpler syntactic constructions, they struggle with deeper hierarchical dependencies and abstract syntactic constraints. The study underscores the need for integrating explicit syntactic principles into LLM architectures to bridge the gap between surface-level fluency and generative linguistic competence.
References
- Chomsky, N. (1965). Aspects of the Theory of Syntax. MIT Press.
- Chomsky, N. (1977). On wh-movement. In P. W. Culicover, T. Wasow, & A. Akmajian (Eds.), Formal Syntax (pp. 71-132). Academic Press.
- Chomsky, N. (1981). Lectures on Government and Binding: The Pisa Lectures. Foris Publications.
- Comrie, B. (1989). Language Universals and Linguistic Typology: Syntax and Morphology. Blackwell.
- Frazier, L., & Fodor, J. D. (1978). The sausage machine: A new two-stage parsing model. Cognition, 6(4), 291-325. https://doi.org/10.1016/0010-0277(78)90002-1
- Gibson, E. (1998). Linguistic complexity: Locality of syntactic dependencies. Cognition, 68(1), 1-76. https://doi.org/10.1016/S0010-0277 (98)00034-1.
- Goldberg, Y. (2019). Neural Network Methods for Natural Language Processing. Morgan & Claypool Publishers.
- Hawkins, J. A. (2004). Efficiency and Complexity in Grammars. Oxford University Press.
- Kulmizev, A., & Nivre, J. (2022). Schrödinger's treeu2014On syntax and neural language models. Frontiers in Artificial Intelligence, 5, 796788. https://doi.org/10.3389/frai.2022.796788.
- Manning, C. D., Clark, K., Hewitt, J., Khandelwal, U., & Levy, O. (2020). Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117(48), 30046-30054. https://doi.org/10.1073/pnas.1907367117.
- Marvin, R., & Linzen, T. (2018). Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 1192-1202). Association for Computational Linguistics. https://aclanthology.org/D18-1151/.
- Newman, B., Ang, K.-S., Gong, J., & Hewitt, J. (2021). Refining targeted syntactic evaluation of language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 3710-3723). Association for Computational Linguistics. https://aclanthology.org/2021.naacl-main.290/.
- Shi, H., & Knight, K. (2017). Neural language modeling by jointly learning syntax and lexicon. arXiv preprint arXiv:1711.02013. Retrieved from https://arxiv.org/abs/1711.02013.
- Van Schijndel, M., Mueller, A., & Linzen, T. (2019). Quantity doesnu2019t buy quality syntax with neural language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (pp. 5831-5837). Association for Computational Linguistics. https://aclanthology.org/D19-1592/.
- Wilcox, E., Levy, R., Morita, T., & Futrell, R. (2023). Neural networks as cognitive models of the processing of syntactic dependencies: An evaluation using controlled psycholinguistic stimuli. Open Mind: Discoveries in Cognitive Science. Advance online publication. https://doi.org/10.1162/opmi_a_00137.
- Zhou, H., Hou, Y., Li, Z., Wang, X., Wang, Z., Duan, X., & Zhang, M. (2023). How well do large language models understand syntax? An evaluation by asking natural language questions. arXiv preprint arXiv:2311.08287. Retrieved from https://arxiv.org/abs/2311.08287
