Gesture Enhancement of Virtual Agent Mathematics Tutor


FundingNational Science Foundation: Cyberlearning EXP 2013-16 ($550,000, with UC Davis)
Conference Proc.ICLS 2014CSCL 2015

Intellectual Merit: Digital media offer potential affordances for democratizing access to quality mathematics instruction. However, the media’s effectiveness depends on Learning Sciences research into expert teachers’ instructional practices (WHAT to program into the software) and virtual agent technological capabilities (HOW to emulate these practices digitally). This proposal offers to make progress on both fronts synergistically. When teachers communicate about mathematical concepts, they use the gesture modality spontaneously to provide critical spatial-dynamical complements to verbal and symbolic utterance, for example in explaining the idea of slope. And yet these gestures have historically been invisible to researchers, due to: (a) the pervasive epistemological fallacy that mathematical knowledge resides in static symbolic inscriptions rather than in multimodal cognitive processes; and (b) the enduring analytical challenges presented by the gesture phenomenon that impede encoding it for modeling and replication. Consequently, attention to gesture is mostly absent in academic and popular discourse about teacher practice and preparation. Unsurprisingly, then, digital simulations of teachers are absent of gesture, and thus virtual pedagogical agents still fall short of emulating human tutor discursive capability, to the detriment of students’ content comprehension. Two recent developments in research-- (1) LS insight into pedagogical gesture, and (2) animation breakthroughs in gesture algorithms—create an exciting opportunity for a Cyberlearning interdisciplinary collaboration on designing interactive cognitive tutors with multimodal capability. This project seeks to build and evaluate an embodied-agentbased mathematics learning environment that supplements tutorial interaction with gestures that are realistically executed as well as naturalistically timed vis-à-vis students’ multimodal actions and speech contributions. The design will seamlessly integrate the Mathematical Imagery Trainer for Proportion (MITP), an empirically researched embodied-learning device, with a gesturing pedagogical agent. Through designing gesture-generation algorithms that are deeply grounded in gesture and learning research, we hope to significantly improve the performance of pedagogical agents in the mathematics domain, while also developing general guidelines for generating agent gesture. Working with middle-school participants, initially as individuals and pairs and then scaling up to classroom implementations, we will experiment with a range of both interface designs (Wii-mote vs. multi-touch-screen) and agent gestural interaction patterns (e.g., pointing, representing, miming, coaching) to obtain a deeper understanding of how virtual agent embodiment impacts student learning. In turn, theoretical advances will also inform future development of pedagogical digital media across the curriculum. Finally, through the process of simulating expert teachers’ behaviors, we will gain deeper understandings of these behaviors and in particular teachers’ interactive use of multimodality. Via design-based empirical research cycles of implementation, analysis, and refinement, we hope to deliver: (i) technology: software and principles for augmenting pedagogical agents with new gesture-enriched capabilities; and (ii) theory: insights into the nature, types, and roles of gesture in educational interaction. Our team is multidisciplinary, with an LS investigator bringing expertise in mathematics education, embodied cognition, and design-based research, and a computer science investigator with expertise in the engineering and research of animated virtual agents with gesture capability.

Broader Impacts: Despite consistent reform efforts, US mathematics students still straggle behind their global peers. Digitally administered quality instruction offers potential responses to this predicament, yet only to the extent that its design builds on research into effective teaching and leverages computation to simulate this expertise. And yet currently available digital media, broadly speaking, are diluted rather than enhanced versions of effective teaching, with online options including non-interactive videotaped lessons, individualized conversation-based interactions with human tutors, and non-gesturing interactive AI tutors. By infusing interactive agent tutors with naturalistic, domain appropriate gesture, the proposed media will provide the missing link toward massive distribution of naturalistic tutorial interaction. At the same time, theoretical insights into pedagogical gesture could be leveraged in creating new content and activities for pre/in-service teachers participating in professional development. The proposing laboratories will proactively disseminate their theoretical work and design innovation via publishing in leading journals, participating in conferences and workshops, and hosting visiting scholars. These laboratories are diverse, collaborative collectives dedicated to training the next generation of LS/CS researchers.