Language processing in the visual world: Effects of preview, visual complexity, and prediction

• Published in: Journal of memory and language Vol. 69; no. 3; pp. 165 - 182
• Main Authors: Ferreira, Fernanda, Foucart, Alice, Engelhardt, Paul E.
• Format: Journal Article
• Language: English
• Published: Amsterdam Elsevier Inc 01.10.2013; Elsevier; Elsevier BV
• Subjects: Ambiguity; Audition; Biological and medical sciences; Comprehension; Eye Movements; Fundamental and applied biological sciences. Psychology; Garden Path Effect; Information processing; Language; Language Processing; Language-vision interactions; Learning; Listening comprehension; Perception; Processing strategies; Production and perception of spoken language; Psychology. Psychoanalysis. Psychiatry; Psychology. Psychophysiology; Sentences; Syntactic ambiguity resolution; Syntactic Processing; Vision; Visual Processing; Voice communication; Learning; Processing strategies; Syntactic ambiguity resolution; Eye movements; Language-vision interactions; Human; Verbal comprehension; Prediction; Preview; Cognition; Verbal perception; Experimental study; Eye movement; Hearing; Acquisition process; Language; Vision; Information processing; Perception; Stimulus complexity; Speech; Strategy; Syntax; Ambiguity
• ISSN: 0749-596X; 1096-0821
• DOI: 10.1016/j.jml.2013.06.001

•Processing of utterances containing syntactic ambiguities was investigated.•Utterances were interpreted against the context of a relevant visual world.•Garden-path effects occur only when subjects preview displays with few objects.•Production data suggest listeners may use preview to predict upcoming utterances.•Results support a flexible and adaptive language processing system. This study investigates how people interpret spoken sentences in the context of a relevant visual world by focusing on garden-path sentences, such as Put the book on the chair in the bucket, in which the prepositional phrase on the chair is temporarily ambiguous between a goal and modifier interpretation. In three comprehension experiments, listeners heard these types of sentences (along with disambiguated controls) while viewing arrays of objects. These experiments demonstrate that a classic garden-path effect is obtained only when listeners have a preview of the display and when the visual context contains relatively few objects. Results from a production experiment suggest that listeners accrue knowledge that may allow them to have certain expectations of the upcoming utterance based on visual information. Taken together, these findings have theoretical implications for both the role of prediction as an adaptive comprehension strategy, and for how comprehension tendencies change under variable visual and temporal processing demands.