Current approaches to the investigation of multimodality
https://doi.org/10.17021/2712-9519-2025-2-74-89
Abstract
The article is dedicated to the study of the semiotics of multimodal phenomena. Multimodality is an essential feature of an interdisciplinary approach that aims to describe a range of modes (visual, acoustic, etc.). Multimodality research focuses on the systemic analysis of communication conditions, including digital media and technologies. Since the pragmatics of language is only realised in a multimodal context, a multimodal approach implies a wide range of methods for analysing digital media and discourse.
This article is a critical overview of the approaches of multimodal language research. It aims to systematise them for the analysis of semiotically complex texts. Some of the approaches presented for investigating the relationship between language and image, such as verbal-visual semiotics, analyse the interaction between language and image in the creation of meaning. Others focus on pragmatic communication, such as visual-verbal rhetoric.
About the Author
K. Köpke
Perm state national research university
Russian Federation
Russian Federation
Katrin Köpke - associate professor, department of linvodidactics, Perm state national research university.
Perm
References
1. Sachs-Hombach K. Bildtheorien. Anthropologische und kulturelle Grundlagen des Visualistic-Turns. – Frankfurt a.M.: Suhrkamp, 2009. – 437 s.
2. Kress G. Literacy in the New Media Age. – London: Routledge, 2003. – 208 p.
3. Bucher H.-J. „Man sieht, was man hört“ oder Multimodales Verstehen als interaktionale Aneignung. Eine Blickaufzeichnungsstudie zur audiovisuellen Rezeption. In: Jan Georg Schneider/Hartmut Stöckl (Hrsg.): Medientheorien und Multimodalität. Ein TV-Werbespot – Sieben methodische Beschreibungsansätze. – Köln, 2011. – Ss. 109-150.
4. Jewitt C. The Routledge Handbook of Multimodal Analysis. 2nd edition. – London: Routledge, 2014. – 562 p.
5. Kress G., van Leeuwen T. Reading Images: The Grammar of Visual Design. Second Edition. – London: Routledge, 2006. – 288 с.
6. Bateman J., Wildfeuer J., Hiippalle T. Multimodality. Foundations, Research and Analysis. A Problem Oriented Introduction. – Berlin, Boston: De Gruyter, 2017. – 424 p.
7. van Leeuwen T. Introducing Social Semiotics. – London: Routledge, New York, 2005. – 301 p.
8. Stöckl H. Multimodalität – Semiotische und textlinguistische Grundlagen. In: Stöckl, Hartmut, Klug, Nina-Maria (HG) Handbuch Sprache im multimodalen Kontext. – Berlin und Boston: de Gruyter, 2016. – Ss. 3-35.
9. van Leeuwen T. Semiotic and Iconography. In: Leeuwen, van Theo; Jewitt, Carey (ed.): Handbook of Visual Analysis, London: Routledge, 2001. – Ss. 92‐118.
10. Kress G. Multimodality. A social semiotic approach to contemporary communication. – London, New York: Routledge, 2010. – 212 p.
11. Halliday M.A.K. Language as social semiotic. The social interpretation of language and meaning. – London: Edward Arnold, 1978. – 256 p.
12. MODE (2012). Glossary of multimodal terms [National Centre for Research Methods]. Available at: https://multimodalityglossary.wordpress.com/genre/ (accessed 12 February 2025).
13. Bateman J. Multimodality and Genre. A Foundation for the Systematic Analysis of Multimodal Documents. – London: Palgrave Macmillan, 2008. – 331 p.
14. Kress Gunther/van Leeuwen, Theo Multimodal Discourse. The modes and media of contemporary communication. – London: Bloomsbury Academic, 2017. – 142 p.
15. Mayr Andrea Multimodal Critical Discourse Analysis (MCDA). In: Stöckl, Hartmut, Klug, Nina-Maria (HG) Handbuch Sprache im multimodalen Kontext. – Berlin und Boston: de Gruyter, 2016. – Pp. 261-276.
16. Tereick J. Klimawandel im Diskurs: Multimodale Diskursanalyse Crossmedialer Corpora. – Berlin und Boston: de Gruyter, 2016. – 443 p.
17. Sorokina Iu. V. Poniatie mul'timodalnosti i voprosy analiza mul'timodal'nogo lektsionnogo diskursa. – Tabov: Gramota, 2017. – Pp. 168-170.
18. Egbert S. Diskurs und Materialität. Eine Dispositivanalyse des Drogentests. – Wiesbaden: Springer, 2022. – 348 p.
19. Klug N.-M., Stöckl H. Sprache im multimodalen Kontext. In: Felder, Ekkehard, Gardt, Andreas (HG): Handbuch Sprache und Wissen. – Berlin und Boston: de Gruyter, 2015. – Pp. 242-264.
20. Forceville C. Pictorial and multimodal Metaphor. In: Stöckl, Hartmut, Klug, Nina-Maria (HG) Handbuch Sprache im multimodalen Kontext. – Berlin und Boston: de Gruyter, 2016. – Pp. 241-260.
21. Knape J. Multimodalität aus rhetoriktheoretischer Sicht // IMAGE. Zeitschrift für interdisziplinäre Bildwissenschaft. – 2018. – Heft 28, Jg. 14. Nr. 2. – Pp. 47-60. DOI: https://doi.org/10.25969/mediarep/16403.
22. Koch N., Schirren T. Verbal-visuelle Rhetorik. In: Stöckl, Hartmut, Klug, Nina-Maria (HG) Handbuch Sprache im multimodalen Kontext. – Berlin und Boston: de Gruyter, 2016. – Pp. 217-20.
23. Spitzmüller J. Typographie – Sprache als Schriftbild. In: Stöckl, Hartmut, Klug, Nina-Maria (HG) Handbuch Sprache im multimodalen Kontext. – Berlin und Boston: de Gruyter, 2016. – Pp. 99-120.
24. Crystal D. Toward a typographical linguistics // Type. A Journal of the Association Typographique Internationale. – 1998. – 2 (1). – Pp. 7-23.
25. Fix U. Nichtsprachliches als Textfaktor. Medialität, Materialität, Lokalität // Zeitschrift für Germanistische Linguistik. – 2008. – 36 (3). – Pp. 343-354.
26. Spitzmüller J. Graphische Variation als soziale Praxis. Eine soziolinguistische Theorie skripturaler ‚Sichtbarkeit‘. – Berlin/Boston: de Gruyter, 2013. – 521 p.
Review
For citations:
Köpke K. Current approaches to the investigation of multimodality. Linguistics & education. 2025;5(2):74-89. (In Russ.) https://doi.org/10.17021/2712-9519-2025-2-74-89
Views:
186
JATS XML
Email this article 