Interfacing comics and games A socio-affective multimodal approach

• Published in: Comics and Videogames Vol. 1; pp. 29 - 44
• Main Author: Ng, Carman
• Format: Book Chapter
• Language: English
• Published: Routledge 2021
• Edition: 1
• Subjects: Amazing Spider Man; Military Entertainment Complex; Audiovisual Aesthetics; Metal Gear; Graphic Novel; Young Man; Leap Motion Controller; Dynamic Panels; Multimodal Artifacts; Gameplay Experience; Metal Gear Solid; Multimodal Analyses; Semiotic Modes; Gamified Comics; Media User Experiments; Vice Versa; Digital Games; Metal Gear Solid 2; Human Technology Interactions; Transmedial Storytelling; Breathing Room; Virtual Reality Headsets; Semiotic Resources
• ISBN: 0367474190; 9780367474195; 9780367539016; 0367539012
• DOI: 10.4324/9781003035466-4

The nexus of comics and games evidences an ecosystem of interplay and multilateral experimentation of porous medial boundaries. As part of an ongoing project driven by sociosemiotic multimodality and affective sciences, this chapter applies digital annotation and gameplay analysis to examine how digital games transfect aesthetics of various audiovisual narratives. I examine such illustrative examples as the second digital graphic novel of the genre-defining Japanese stealth game series Metal Gear Solid (1998-2015), the comic-game Gorogoa (2017), and the gesture-based literary art Breathing Room (2014). My focus is on analyzing how movement of in-text semiotic resources (e.g., graphics, linguistic texts, panels) and kinetics intersect imaginaries of human-media interfaces, indicating the rising influence of game design on digital interfaces and users' temporal-spatial perceptions. Specifically, the chapter problematizes hybridity, a common denominator of digitalized media that has arguably attracted insufficient empirical scrutiny. I contend that a multifaceted approach anchored by fine-grained multimodal analyses appropriately engages with the dynamics among intermediality/transmedial storytelling, theory/empiricism, experience/design to build a foundation for further affect-focused investigations. Combining empirical work and theories of digital media studies and multimodality, this chapter emphasizes the need for interdisciplinary research to deliberate the mutual malleability of comics and games, exemplary of evolving contemporary semiotic practices.