Exploring Multimodal Discourse in the AI Age: Opportunities and Prospects

doi:10.3969/j.issn.1674-8921.2025.03.011

Abstract

Abstract:

With the rise of generative AI, social semiotic multimodality studies face new opportunities and challenges. This paper reviews recent developments in multimodal researches to explore the application and impacts of AI technologies in social semiotics. Our findings show that AI helps overcome methodological limitations in multimodal discourse research, stimulates innovation in multimodal genres, and contributes to a paradigm shift in social semiotics theory—one that considers all communicative elements in the dynamic meaning exchange process. To address AI’s limitations in empathy and multimodal comprehension, future efforts should focus on building harmonious human-AI collaboration, encouraging interdisciplinary work and enriching the interpretation of multimodal discourse theory.

Key words: social semiotics, artificial intelligence, GAI, multimodal discourse, digital genre

CLC Number:

H319

LI Zhanzi, HAN Zeting. Exploring Multimodal Discourse in the AI Age: Opportunities and Prospects[J]. Contemporary Foreign Languages Studies, 2025, 25(3): 119-128.

References 27

[1]	Aiello G. & T. van Leeuwen. 2023. Michel Pastoureau and the history of visual communication[J]. Visual Communication 22(1):27-45.
[2]	Bateman J. A. 2022. Multimodality, where next? Some meta-methodological considerations[J]. Multimodality & Society 2(1): 41-63.
[3]	Bewersdorff A., C. Hartmann M. Hornberger, et al. 2025. Taking the next step with generative artificial intelligence: The transformative role of multimodal large language models in science education[J]. Learning and Individual Differences 118: 102601-102601.
[4]	Halliday M. A. K. 1978. Language as Social Semiotic: The Social Interpretation of Language and Meaning[M]. London: Edward Arnold.
[5]	Han Y. & B. E. Smith. 2023. An ecological perspective on the use of memes for language learning[J]. Language Learning and Technology 27(2):155-175.
[6]	Kress G. & T. van Leeuwen. 1996. Reading Images: The Grammar of Visual Design[M]. London: Routledge.
[7]	Kress G. & T. van Leeuwen. 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication[M]. London: Hodder Arnold.
[8]	Lee Y.-J. 2023. Language learning affordances of Instagram and TikTok[J]. Innovation in Language Learning and Teaching 17(2): 408-423.
[9]	Pratschke B. M. 2024. Generative AI and Education: Digital Pedagogies, Teaching Innovation and Learning Design[M]. Berlin & London: Springer Nature.
[10]	Polli C. & M. G. Sindoni. 2024. Multimodal computation or interpretation? Automatic vs. critical understanding of text-image relations in racist memes in English[J]. Discourse Context & Media 57: 100755-100755.
[11]	Shirahata M., M. Lahti & M. Siitonen. 2023. Language ideologies in a Finnish university student union’s Facebook communication practices[J]. Social Semiotics 35(1): 114-132.
[12]	Stöckl H. 2020. Linguistic multimodality-multimodal linguistics:A state-of-the-art sketch[A]. In J. Wildfeuer, J. Pflaeging, J. A. Bateman. et al. (eds.). Multimodality: Disciplinary Thoughts and the Challenge of Diversity[C]. Berlin, Boston: De Gruyter. 41-68.
[13]	Valeiras-Jurado J. & E. Bernad-Mechó. 2022. Modal density and coherence in science dissemination: Orchestrating multimodal ensembles in online ted talks and YouTube science videos[J]. Journal of English for Academic Purposes 58(1): 101118-101118.
[14]	Westberg G. & G. Kvåle. 2024. The generic uniqueness of ai imagery: A critical approach to DALLE as semiotic technology[J]. Discourse & Society: 1-24.
[15]	陈凯泉、张春雪、吴玥玥、等. 2019. 教育人工智能(EAI)中的多模态学习分析、适应性反馈及人机协同[J]. 远程教育杂志 37(5): 24-34.
[16]	崔中良. 2024. 通感、具身与意识:人机交互实现的感知基础[J]. 湖北大学学报(哲学社会科学版) 51(2): 143-152.
[17]	胡易容、马翾昂、尹婧雯. 2024. 意义理论再出发:2023年传播符号学研究前沿综述[J]. 新闻界(1): 59-68.
[18]	胡钰. 2020. 正确面对人工智能新闻业的崛起[N]. 中国社会科学报(8-06):A30.
[19]	刘海涛. 2024. 从语言数据到语言智能——数智时代对语言研究者的挑战[J]. 中国外语 21(5): 60-66.
[20]	梅冰. 2024. 师范生对人工智能赋能外语教学接受度探析[J]. 当代外语研究(2): 89-99. doi: 10.3969/j.issn.1674-8921.2024.02.008
[21]	欧志刚、刘玉屏、覃可、等. 2024. 人工智能多模态教学资源的生成与评价——基于AIGC在国际中文教育的应用[J]. 现代教育技术 34(9): 37-47.
[22]	钱锋、董毅敏、张铁明、等. 2023. 元宇宙多模态、跨模态内容时代的科技出版理论探讨与实践研究[J]. 中国科技期刊研究 34(7): 902-908. doi: 10.11946/cjstp.202306180449
[23]	王建民、曹静. 2020. 人类学的多模态转向及其意义[J]. 民族研究(4): 61-73, 139-140.
[24]	杨凝. 2022. 国际视野下教育机器人的研究现状与发展趋势——基于对25项系统化文献综述的再综述[J]. 现代教育技术 32(6): 122-126.
[25]	袁毓林. 2024. ChatGPT语境下语言学的挑战和出路[J]. 现代外语(4): 1-6.
[26]	张敏、张可、张东鑫. 2024. 政务社交媒体中公众情感治理:理论框架与实施策略[J]. 情报科学(5):1-21.
[27]	赵星植. 2021. 社会符号学的兴起与前景:一个学术史的回顾[J]. 重庆广播电视大学学报 33(5): 3-9.