Paper presented at the 7th annual meeting of the International Society for Gesture Studies, Paris, July 18-22.
We report on a novel approach for archiving repertoires of multimodal pedagogical techniques that enlists animated Graphic Interchange Format (GIF) files. This methodology was developed by an interdisciplinary team of learning scientists (LS-team) and computers scientists (CS-team) building a virtual, animated, mathematics tutor capable of multimodal communication.
Using an extensive video corpus of mathematics instruction, the LS-team identified a repertoire of pedagogical gestures for the CS-team to simulate virtually in the animated character. However, early on the LS-team struggled to communicate this repertoire. Video clips of pedagogical gestures occurring “in the wild” were over-situated in the idiosyncratic spatial configurations of their environments (Goodwin, 2007) and could not delineate the generic, core specifications of the gestures to be reproduced.At the same time, the complex trajectories and morphologies of these gestures did not reduce well to verbal description with static images. Spontaneous, situated motion is notoriously difficult to inscribe (Guest, 1998). Of the various gesture classification schemes currently available (Kendon, 2004), none offer a level of specificity necessary for accurate three-dimensional, dynamic reproduction.
The solution we developed bypassed the need to represent gestures with either static inscription or in situ video clips. We created a digital library of animated-GIFs of re-enacted gestures and succeeded in using this library to support an actor’s motion capture performance. By re-enacting gestures from the video corpus, we were able to create idealized, contextually generic forms of the spontaneous gesturing techniques we observed in learning settings. Animated-GIFs unequivocally convey the three-dimensional, dynamic details of each gesture, allowing a viewer to quickly and accurately learn the form. An unexpected benefit is that animated-GIFs also capture other critical semiotic resources of the multimodal Gestalts (Mondada, 2014) that accompany gesturing such as patterns of gaze and facial expressions (Streeck, 2009).
There is strong consensus that teacher gesture during instruction is essential for students’ learning (Nathan & Alibali, 2011) and therefore there is a growing need to develop materials to support teachers in effective use of multimodality in lessons. Currently, common gesture annotation systems—verbal narratives and static images with multiple elaborate arrows—leave too much spatio-dynamic information (e.g., trajectory) ambiguous. Teachers working from such illustrated scripts cannot faithfully reenact the original movements. Animated-GIF libraries of re-enacted pedagogical gestures are clearly depictive, circumvent privacy issues, and can be stored in broadly accessible formats (e.g., web). Therefore, we believe our animated-GIF banks present exciting possibilities for productively disseminating pedagogical gesturing techniques directly to in-service and pre-service teachers as part of professional development.
Guest, A. H. (1998). Choreo-graphics: A comparison of dance notations systems from the fifteenth century to the present. Amsterdam: Gordon and Breach.
Goodwin, C. (2007). Environmentally coupled gestures. In S. Duncan, J. Cassell, & E. Levy (Eds.), Gesture and the dynamic dimension of language (pp. 195–212). Amsterdam, The Netherlands: John Benjamins.
Kendon, A. (2004). Gesture: Visible Action as Uttterance. Cambridge University Press.
Mondada, L. (2014). The local constitution of multimodal resources for social interaction. Journal of Pragmatics, 65, 137-156.
Nathan, M. J., & Alibali, M. W. (2011). How gesture use enables intersubjectivity in the classroom. In G. Stam & M. Ishino (Eds.), Integrating gestures: The interdisciplinary nature of gesture (pp. 257–266). Amsterdam, The Netherlands: John Benjamins.
Streeck, J. (2009). Gesturecraft: The manu-facture of meaning. Amsterdam: Benjamins.