Gesture Enhancement of Virtual Agent Mathematics Tutor

FundingNational Science Foundation: Cyberlearning EXP 2013-16 ($550,000, with UC Davis)
Conference Proc.ICLS 2014CSCL 2015

Intellectual Merit: Digital media offer potential affordances for democratizing access to quality mathematics instruction. However, the media’s effectiveness depends on Learning Sciences research into expert teachers’ instructional practices (WHAT to program into the software) and virtual agent technological capabilities (HOW to emulate these practices digitally). This proposal offers to make progress on both fronts synergistically. When teachers communicate about mathematical concepts, they use the gesture modality spontaneously to provide critical spatial-dynamical complements to verbal and symbolic utterance, for example in explaining the idea of slope. And yet these gestures have historically been invisible to researchers, due to: (a) the pervasive epistemological fallacy that mathematical knowledge resides in static symbolic inscriptions rather than in multimodal cognitive processes; and (b) the enduring analytical challenges presented by the gesture phenomenon that impede encoding it for modeling and replication. Consequently, attention to gesture is mostly absent in academic and popular discourse about teacher practice and preparation. Unsurprisingly, then, digital simulations of teachers are absent of gesture, and thus virtual pedagogical agents still fall short of emulating human tutor discursive capability, to the detriment of students’ content comprehension. Two recent developments in research-- (1) LS insight into pedagogical gesture, and (2) animation breakthroughs in gesture algorithms—create an exciting opportunity for a Cyberlearning interdisciplinary collaboration on designing interactive cognitive tutors with multimodal capability. This project seeks to build and evaluate an embodied-agentbased mathematics learning environment that supplements tutorial interaction with gestures that are realistically executed as well as naturalistically timed vis-à-vis students’ multimodal actions and speech contributions. The design will seamlessly integrate the Mathematical Imagery Trainer for Proportion (MITP), an empirically researched embodied-learning device, with a gesturing pedagogical agent. Through designing gesture-generation algorithms that are deeply grounded in gesture and learning research, we hope to significantly improve the performance of pedagogical agents in the mathematics domain, while also developing general guidelines for generating agent gesture. Working with middle-school participants, initially as individuals and pairs and then scaling up to classroom implementations, we will experiment with a range of both interface designs (Wii-mote vs. multi-touch-screen) and agent gestural interaction patterns (e.g., pointing, representing, miming, coaching) to obtain a deeper understanding of how virtual agent embodiment impacts student learning. In turn, theoretical advances will also inform future development of pedagogical digital media across the curriculum. Finally, through the process of simulating expert teachers’ behaviors, we will gain deeper understandings of these behaviors and in particular teachers’ interactive use of multimodality. Via design-based empirical research cycles of implementation, analysis, and refinement, we hope to deliver: (i) technology: software and principles for augmenting pedagogical agents with new gesture-enriched capabilities; and (ii) theory: insights into the nature, types, and roles of gesture in educational interaction. Our team is multidisciplinary, with an LS investigator bringing expertise in mathematics education, embodied cognition, and design-based research, and a computer science investigator with expertise in the engineering and research of animated virtual agents with gesture capability.

Broader Impacts: Despite consistent reform efforts, US mathematics students still straggle behind their global peers. Digitally administered quality instruction offers potential responses to this predicament, yet only to the extent that its design builds on research into effective teaching and leverages computation to simulate this expertise. And yet currently available digital media, broadly speaking, are diluted rather than enhanced versions of effective teaching, with online options including non-interactive videotaped lessons, individualized conversation-based interactions with human tutors, and non-gesturing interactive AI tutors. By infusing interactive agent tutors with naturalistic, domain appropriate gesture, the proposed media will provide the missing link toward massive distribution of naturalistic tutorial interaction. At the same time, theoretical insights into pedagogical gesture could be leveraged in creating new content and activities for pre/in-service teachers participating in professional development. The proposing laboratories will proactively disseminate their theoretical work and design innovation via publishing in leading journals, participating in conferences and workshops, and hosting visiting scholars. These laboratories are diverse, collaborative collectives dedicated to training the next generation of LS/CS researchers.


"Kinemathics": Kinetically Induced Mathematical Learning

UCB Committee on Research: Faculty Research Grants, 2008-11 [$21k]
White PaperResearch-Group("White-Paper")
Conf. Proc:  AERA2010(Empirical),  PME-NA2010(Empirical),  PME-NA2010(Design)AERA2011(Hooks-and-Shifts)AERA2011(Embodied-Learning-Symp), SIGCHI2011IDC2011, CSCL2011, PMENA2011, AERA2012(+x), AERA2012(tutor), ICLS2012-symp, ICLS2012-paper, PME-NA2013(tablets), INTED2015(eye-tracking)PME-NA2015
Journal ArticlesTechnology, Knowledge, and Learning (hooks and shifts),
Technology, Knowledge, and Learning (fostering hooks and shifts), ZDM (coordinating visualizations), ZDM (Enactivism)

Video ClipsMechanical-MITElectronic-MITCyTSE2011

We're developing and researching an embodied-interaction design for mathematics learning. Children remote-manipulate virtual objects on a computer screen in an attempt to figure out the underlying principle of the "mystery device." The underlying principle is proportionality, a mathematical conceptual system that challenges many students. Our premise is that everyday life does not occasion opportunities for children to enact what could then serve as the meaning -- the cognitive substrate -- for what thus become challenging mathematical notions. For example, we can all intuitively gesture what "addition" means and, at a stretch, what "multiplication" means, but what does "proportion" mean or look like? Our design principle is to contrive these missing everyday occasions by creating problem-based interactions wherein students' gestural solutions literally inscribe the kinesthetic image schema underlying a conceptual metaphor of the targeted notion*, such as proportional growth. If we physically support students in enacting the ambidexterous performance of proportional progression, they may develop and articulate strategies for enacting this physical coordination; if we further interpolate into the learning environment a set of mathematical symbolic artifacts (a Cartesian grid, numerals, etc.), then students may be able to mathematize these emerging strategies in the form of quantitative propositions. Indeed, this is what we are seeing in our emprirical data from 20 interviews with Grade 4-6 students, who particiapted either in individual or paired interviews.

*The EDRL embodied-design principle.


Paradigmatic Didactical Mathematical Problematic Situations

In collaboration with Betina Zolkower, Ph.D., Brooklyn College, CUNY.
AERA 2009

In the PDMPS project, we are implementing and researching an experimental design for pre-service teachers and future education researchers enrolled in graduate-level college courses on mathematics cognition, learning, and instruction. Central to this design are selected 'paradigmatic didactical-mathematical problematic situations,' i.e., unique activities evoked as contexts for collaborative inquiry into the epistemology, pedagogy, and practice of mathematics as well as into subject matter content. Our data include rich documentation from both the college classroom and the placement classrooms, where the student teachers are trying out the same problems. Building on functional-grammar analysis techniques, we are evaluating the conjecture that the curriculum's value lies in the authenticity of the multi-disciplinary pragmatic approach it fosters in future teachers. We are also interested in potential tradeoffs inherent to a problem-focused curriculum.


Seeing Chance: Fostering Student Implicit Knowledge Towards Fluency in the Domain of Probability and Statistics

Seed funding: National Academy of Education/Spencer Foundation postdoctoral fellowship to Abrahamson, 2005-6 [$65k].
Conference Proc.: AERA2007PMENA2007a, PMENA2007b, AERA2008ICLS2008, ICME2008, AERA2009SRTL2009, PMENA2009, ICLS2010,   
Journal ArticlesCognition & Instruction , Ed. Sudies in Math. (+ QuickTime), For the Learning of Mathematics,
Int. Elec. J. of Math. Ed.Int. J. of Comp. for Math. Learning: article
Int. J. of Comp. for Math. Learning: snapshot, ZDM,
CreativeIT interactive
J. of the Learning SciencesJ. of Stat. Ed., Mathematical Thinking and Learning (+ QuickTime)
Chapters: Stein&Raudenbug(Eds) , Chernoff-Sriraman(Ed.)

Consider this paradox. On the one hand, there is growing evidence of very young babies' capacity to draw mathematically sound infererence from situations involving random events. On the other hand, school students' manifest chronic challenges in learning probability concepts. What's going on? Well, perhaps school is not capitalizing on students' innate or early-developed capacity. In this project, we created learning materials --  using both traditional media (marbles, cards, crayons) and computer-based modules (NetLogo simulations) -- that, on the one hand, enable studetns to draw on the same intuitions that we know babies have, but on the other hand lend themthelsves to elaboration into mathematical models. We thus honed the tension between inference from intuitive perceptual judgments of naturalistic situations and inferences from analytical models of these same situations. We worked with students in Grades 4-6 as well as with 7th graders and undergraduate and graduate students. Across the gamut, we found, students were capable of leveraging the same sound intuitions when looking at the situation in non-analytical ways but were challenged by the rationale of the formal analysis. The challenge, it turned out, was centered on perceptual construction of random events. Namely, whereas mathematical analysis contrues compound events as classes containing the combinatorial expansion (e.g., the pair "Heads, Tails" is double as likely to occur than "Heads, Heads", because the former event has two permutations), intuitive views of the source situations construes the compound events as holistic instances with probabilistic "intensity" and no reference to order (it "feels" more likely but we're not sure why). These felt "intensities" are difficult or even impossible to inscribe and thus make use of in ways that promote understanding, if you do not accept the principle of combinatorial expansion. Yet, once students succeeded in viewing a specially arranged assembly of the sample space as holistically expressing their intuitive inference regarding the source situation, they were willing retroactively to accept the procedure of combinatorial analysis. We named this sudden appropriation of the mathematical model a "semiotic leap," because at those moments new signs were born in which disciplinary structures first bore the intuitive meanings. The model signified the intuition. (see previous and current design)


The Real World as a Trick Question: Mathematical Modeling, Knowledge, and Assessment

UCB Committee on Research: Junior Faculty Research Grant, 2006-7 [$6k]
AERA 2007

Study consisted of conducting and analyzing probability-related clinical interviews with college students majoring in statistics to explore issues of intuitive reasoning. 24 undergraduate/graduate students enrolled in mathematical programs participated in one-to-one interviews as part of a design-based research study of the cognition of probability. The students were asked to estimate outcome distributions of a very simple randomness generator consisting of an exposed bin full of marbles, half green and half blue, and a scooper -- a 2-by-2 array of concavities -- for drawing out exactly four marbles from the mix. This array formation (4-block) featured also in combinatorial-analysis materials and computer-based simulations of the probability experiment. Central to the design is the combinations tower, an assembly of the 16 unique outcomes in the form of a 1:4:6:4:1 "picto-barchart," i.e., with the outcomes themselves, not just stark columns as in regular histograms. All students said that the relatively most common experimental outcome should have 2 green and 2 blue marbles, but only 10 students initiated combinatorial analysis as a means of warranting their guess, of whom only 4 conducted it successfully. For all students, the combinations tower constituted a context for coordinating between the sample space of the stochastic device and distributions of actual outcomes in experiments with this device. I argue for the utility of guided, situated problem solving for the learning and consolidation of probability concepts.


Handing Down Mathematics: The Roles of Gesture in the Design, Teaching, and Learning

Analyzing videotaped classroom interactions to understand the roles of gesture in the design, teaching, and learning of mathematics.



Embodied Design: Framework for Creating Materials and Activities


Distributed Learning in Practice and Theory


Tackling distributed-learning theoretical models from a complexity-studies perspective to frame the design and implementation of agent-based models and their extensions that support participatory simulations in mathematics classrooms; Using agent-based modeling to study and develop theoretical models of group learning.


The Three M's: Imagination, Embodiment, and Mathematics

dowload abstract from JPS 2006

Research into the mechanisms and potential role of imagination in mathematical reasoning.


Fractal Village (Sneha Veeragoudar Harrell's dissertation work)

Journal of Virtual Worlds Research

Design-based research utilizing a critical and constructionist pedagogical philosophy in an alternative high school setting to study mathematical agency, computational literacy, and identity. 


Giant Steps for Algebra (Kiera Chase's dissertation work)

Conference proceeding papers: IDC2013ICLS204IDC2015
Conference presentations: AERA2015
Journal articles: Reflective Practice, ZDM

GS4A is a design-based research project investigating the cognittion, teaching, and learning of early aglebra. The project is centered on a technology-based activity, in which students use computer-interface resources so as to model a narrative about a giant burying stolen treasure on a desert island. The idea is for students to 're-invent' the basic principles of algebraic thinking, and in particular solving a system of two story-equations. My work led to developing a design construct I call "reverse scaffolding." This means that rather that a teacher offering support that then fades out, it is the child who initiates and builds new mathematical functions that then fade in. By 'fade in' I mean that the computer takes over enacting those functions, thus freeing up the child to work on higher-order issues. Another new construct that emerged from my dissertation is that of a SILO -- situated intermediary learning objective -- which attempts to capture a proto-conceptual type of practical knowledge about how to organize information toward solving a problem. This work built on ideas coming from constructionism as well as the related idea of transparency. Our various publications explain the rationale, early work, results from the main study, and reflections on the design process.

User login