ABSTRACT
Multimodality is fast becoming the main conceptual framework for the study of audiovisual texts, i.e. texts that create meaning through the use of multiple semiotic modes, such as films. Individual semiotic modes include the aural-verbal mode (dialogues and lyrics); the aural, non-verbal mode (music and sound effects); the visual-verbal mode (various types of text on screen); and the visual, non-verbal mode (images) (Delabastita 1989, Remael 2001, Zabalbeascoa 2008). Multimodality examines how these individual modes function and how they can be combined into a unified whole. All the modes have a role to play in the creation of meaning in a text, but their importance will vary: in some films or film scenes music may be dominant; in others, the images may carry the story forward. What is more, combining different modes creates supplementary meaning, on top of the meanings conveyed by the individual modes. This is what Baldry and Thibault (2006) refer to as the ‘resource integration principle’. What matters, however, is ‘how they [the semiotic modes] all add up and combine with each other so that viewers can interpret them in certain ways’ (Zabalbeascoa 2008: 25). To guide users’ interpretations, filmmakers insert different types of implicit and explicit links between modes, which serve as cues for the users to reconstruct a coherent end product. In this chapter, multimodal texts are seen to work through multimodal cohesion, which is accomplished through different cross-modal ties between the different semiotic modes. We pragmatically define multimodal cohesion as any instance of implicit or explicit ‘sense-relation’ between two or more signs, from the same or different modes, within a given text that helps the viewer create a coherent textual semantic unit (for an overview of key concepts in multimodal theory, see the publications listed in the Further Reading section, at the end of this chapter).
