Multimodality

The term ‘multimodality’ dates from the 1920s. It was a technical term in the then relatively new field of the psychology of perception, denoting the effect different sensory perceptions have on each other. An example of this is the so-called McGurk effect: if people are shown a video of someone articulating a particular syllable, e.g. /ga/, while hearing another syllable, e.g. /ba/, they perceive neither /ga/, nor /ba/, but /da/ (Stork 1997: 239). In other words, perception is multimodal. It integrates information received by different senses. More recently, linguists and discourse analysts have taken up the term, broadening it to

denote the integrated use of different communicative resources, such as language, image, sound and music in multimodal texts and communicative events. As soon as they had begun to study texts and communicative events rather than isolated sentences, they realized what they should have known all along: that communication is multimodal; that spoken language cannot be adequately understood without taking non-verbal communication into account; and that many forms of contemporary written language cannot be adequately understood unless we look, not just at language, but also at images, layout, typography and colour. In the past twenty or so years this led to the development of multimodality as a field of study investigating the common as well as the distinct properties of the different modes in the multimodal mix and the way they integrate in multimodal texts and communicative events. It is not difficult to see why such a field of study should have developed. From the 1920s

onwards, public communication had become increasingly multimodal. Film had changed acting, enlarging subtle aspects of non-verbal communication, and so influencing how people talk and move and smile the world over. Later, television had made non-verbal communication a decisive factor in politics, most famously in the televised debate between Nixon and Kennedy. Writing, too, had become multimodal, as illustrations and layout elements such as boxes and sidebars broke up and reshaped the pages of books and magazines. Like scholars in other fields of study, linguists took notice. In the course of the twentieth century, several schools of linguistics engaged with communicative modes other than language. The first was the Prague School, which, in the 1930s and 1940s, extended linguistics into the visual arts and the non-verbal aspects of theatre, and which included studies of folklore, and collaborations

with avant-garde artists (see, for example, Matejka and Titunik 1976). Paris School structuralist semiotics of the 1960s also used concepts and methods from linguistics to understand communicative modes other than language. Largely inspired by the work of Roland Barthes, it mostly focused on analyses of popular culture and the mass media, rather than on folklore or avant-garde art (e.g. Barthes 1967, 1977, 1983). In roughly the same period, American linguists began to take an interest in the multimodal analysis of spoken language and non-verbal communication. Birdwhistell (e.g. 1973) developed an intricate set of tools for analyzing body motion, and Pittenger et al. (1960) published a highly detailed and groundbreaking multimodal analysis of the first five minutes of a psychiatric interview. In the late 1960s, conversation analysis replaced the 16mm film sound camera with the cassette recorder as the research tool of choice, which diminished attention to non-verbal communication, although some scholars in this tradition have, more recently, re-introduced it (Ochs 1979; Goodwin 2001). A fourth school emerged in the 1990s. Inspired by the linguistics of M. A. K. Halliday (1978, 1985), it was this school which adopted and broadened the term ‘multimodality’, and introduced it into applied linguistics, and especially into the study of language and literacy in education. More recently a further approach, mediated discourse analysis, inspired by the work of Ron and Suzie Scollon (2003, 2004), returned to American micro-analysis of social interaction, but in a new way, linking it to the wider social and political context, and adding a new emphasis on technological mediation (see, for example, Jones 2009). By now, multimodality has its own bi-annual conference and a range of edited books

(e.g. O’Halloran 2004; Ventola et al. 2004; Levine and Scollon 2004; Norris and Jones 2005; Unsworth 2008; Jewitt 2009), and it is regularly included in handbooks and encyclopedias of linguistics, discourse analysis, visual communication and so on. Although it encompasses a number of distinct theoretical and methodological approaches, it has nevertheless remained a united field of study, with productive dialogue and mutual influence between the different ‘schools’ (cf. Jewitt 2009, for a more extensive overview of the different approaches in the field).