Gestures as a tool of semiotic mediation
in 3d turtle geometry environment
Maria Latsi, mlatsi@ppp.uoa.gr
Educational Technology Laboratory, School of Philosophy, University of Athens
Chronis Kynigos, kynigos@ppp.uoa.gr
Educational Technology Laboratory, School of Philosophy, University of Athens
Abstract
In this paper we report findings from
a design-based research aiming at shedding light on the way eleven-year old students
used gestures while constructing mathematical meanings in the framework of a 3d
Turtle Geometry Environment. The results bring in the foreground the role of
gestures, as signs that mediated the mathematical notions integrated in the
computational environment, as well as the interconnection between gestures and
the embodied turtle metaphor. Different kinds of gestures were used by students
depending on their focus point during the construction processes. When focusing
on turtle’s navigation and the graphical results of this navigation, students
used dynamic representational gestures. On the contrary when viewing 3d space
or 3d objects as external observers they used abstract deictic or static
representational gestures.
Keywords (style: Keywords)
gestures, turtle geometry, embodied
metaphors
Theoretical Background
The study of gestures is a rather new field
of research in mathematics education (Radford, 2009), which is investigated
through various perspectives. In this paper the focus is set on the way students
at the end of primary school used gestures in the framework of constructionist
activities in a 3d Turtle Geometry computational environment. Gestures are
investigated as a special mode of embodied expression and communication and as
a tool of semiotic mediation of the learning process, which -in conjunction
with other modes of expression- can shed light on the processes of mathematical
meanings construction (Arzarello et al., 2009).
According to the socio-cultural theory of
learning people’s interaction with the real world is defined and formed through
the use of symbolic objects and cultural tools (Vygotsky, 1978). In this
framework gestures are investigated in their semiotic perspective, on the way
they function as signs that mediate people’s interaction with their environment
in specific cultural contexts (Radford, 2005). In particular, in mathematics
education gestures seem to acquire new dimensions (McNeil, 2000) and are
thought as a means of knowledge objectification, as a means that can help students
realise the notions integrated in the various mathematical objects (Radford,
2009). A special interest is aroused in cases that digital technologies are
used, where certain actions are conceived as new kinds of gestures, e.g.
pointing with the mouse or ‘dragging’ of hot spots in Dynamic Geometry
computational environments (Kaput, 2005). On the one hand gestures are related
to the task to be accomplished and on the other hand they may be related to the
mathematical knowledge that is to be attained. Gestures are usually contingent
to the situation determined by the solution of a particular task but they can also
play a rather pivotal role in promoting the evolution of signs from idiosyncratic
to culturally determined and crystallised mathematical signs. The relationship
between digital technologies and knowledge is complex (Bartolini Bussi & Mariotti,
2008) and a careful analysis of the evolution of gestures used by students
(among all the other signs) can offer a new perspective on the access that students
have on the embedded in these artefacts mathematical knowledge.
In parallel, the intimate relationship
between the functioning of the brain and body experience (with or without the
use of tools) even when the most abstract mathematical notions are considered
is now commonly recognised. Concepts are imminent in each concrete realisation
of experience and in its relation to other experiences. Nemirovsky (2003)
argues that ‘thinking is not a process that takes place ‘behind’ or
‘underneath’ bodily activity, but it is the bodily activities themselves’. Thus,
gestures are investigated as a window on the embodied aspects of meaning
construction processes (Anastopoulou et al., 2011), as an interface between abstract
and symbolic mathematics and mathematical metaphors (Kim, Roth and Thom, 2010) that
are on their part grounded on human sensorimotor experience and action (Lakoff
& Nunez, 2000). Research interest has also arisen recently in relation to
the role of gestures while students are constructing geometrical figures in 2d
(Latsi, 2010) and 3d Turtle Geometry (Morgan & Alshwaikh,
2009) computational environments. These studies
provided empirical support for the embodied means used by students in their
effort to carry out particular geometric tasks highlighting the connections of
certain gestures with the integrated in the aforementioned digital tools mathematical
knowledge while raising questions about the interpretation and use of the same
gestures by different groups of interlocutors.
In the research presented here our
pedagogical aim was to engage the students in navigating a moving entity, the
turtle, to construct graphical digital objects through Logo programming. Research
seems to conclude that carefully designed 2d Logo- based microworlds are an
effective medium in offering rich mathematical experiences and encouraging the
construction of meaning through the turtle metaphor (Clements & Sarama,
1997; Kynigos, 1997). Navigating the turtle requires the formation of
essentially novel methods of spatial orientation, where the reference point is
not the position of the user’s body but the turtle’s body, relative to which
the entire system of orientation may change. In this framework body-syntonicity
is a critical concept in 2d Turtle Geometry (Papert, 1980) that refers: a) to navigating
the turtle by coordinating one’s body-posture, physically or imaginary, with
the turtle-vehicle of motion and b) to solving geometrical problems drawing
upon ones embodied motional experiences. Recent extensions of Turtle Geometry
in 3d space do not offer just a new perspective in the teaching and learning of
geometry. New issues are raised related to the way the turtle metaphor is put
to use and the way deeply rooted intuitions about experiencing space and
locomotion can be exploited so as to make sense of geometric notions (Kynigos
& Latsi, 2007, Morgan et Alshwaikh, 2009). In particular our research aim was to investigate: a) students’
gestures and their role in mathematical meanings construction in the 3d
simulated geometrical space and b) the way these gestures are related to the
central metaphor in the 3d Turtle Geometry environments, turtle as a moving
entity with which the user can syntonise his/her body.
The computational Environment
MaLT is a constructionist microworld environment
that extends ‘Turtleworlds’ to 3d geometrical space. ‘Turtleworlds’ blends Logo
based Turtle Geometry with tools to dynamically manipulate procedure variables
and observe the resulting ‘continuous’ change to the respective figural constructions
(Kynigos et al, 1997). In MaLT, we used a well established method to extend
Turtle Geometry to 3d by adding two kinds of turn commands (Reggini, 1985):
‘UPPITCH/DOWNPITCH n degrees’ (‘up/dp n’), which pitches the turtle’s nose up
and down on a plane perpendicular to the one defined by right-left turns, and
‘LEFTROLL/RIGHTROLL n degrees’ (‘lr/rr n’) which moves the turtle around its own
axis. A second feature of MaLT is that we kept the ‘Turtleworlds’ feature of
variation tools. These tools recognise the procedure responsible for any
figural construction and afford dynamic manipulation of variable values
resulting in DGS-style change in the figures. A third feature also affords
dynamic manipulation but this time what is changed is the users’ viewpoint of
the Turtle Geometry space: a) in a toggle fashion by using buttons to pick
among 3 default views (front, side, top-down) and b) by dragging a specially
designed vector tool, which we called ‘the active vector’, where the user can
define the camera’s direction or position. Thus MaLT combines: a)
interactivity, b) multiple interlinked representations and c) dynamic
manipulation and dynamic visualisation of the 3d simulated space.
Figure 1 .The MaLT
computational environment
Methodology
Espousing an interpretive approach in
educational research (Cohen et al., 2007) in the study reported here we
followed a design-based research method (Van Den Akker et al., 2006), which
entailed the ‘engineering’ of tools and task, as well as the systematic study
of both the process of learning and the means of supporting it (Gravemeijer
& Cobb, 2006). A critical component of design –based research is that the
design is conceived not just to meet local needs but to advance a theoretical
agenda, to uncover, explore and confirm theoretical relationships, to create
new theoretically expressed understandings about areas for which little is
known. Thus, the analysis we have carried out does not comprise any kind of
quantification of qualitative data, but rather refers to a non mathematical
process of interpretation, carried out for the purpose of discovering concepts
and relationships in raw data and then organizing these into a theoretical
explanatory scheme.
The research took place in the 6th grade of a public primary school in Greece. The class consisted of 23 pupils,
who had totally sixteen 45 minutes teaching sessions with the experimenting
teacher over two months. The pupils didn’t have any previous experience with 3d
Turtle Geometry environments but they were accustomed to 2d Turtle Geometry.
The pupils worked collaboratively in mixed-gender groups of two or three in the
school’s computer laboratory. The tasks were designed to bring in the
foreground issues concerning the mathematical nature of 3d geometrical objects through
their dynamic manipulation and transformation in mathematically meaningful ways. In this research paper we present and analyse data
taken from the first two tasks of the activity sequence that lasted 4 teaching
sessions. In the 1st task the pupils were asked to navigate the
turtle in such a way so as to simulate the take-off and the landing of an
aircraft while in the second one they were asked to construct rectangles and to
position them in at least two different planes of the simulated 3d space, so as
to simulate the walls of a room.
In order to describe the pupils’ learning
trajectories as they happened in real time we adopted a participant observation
method in data collection while the main corpus of data included video-recorded
observational data, the experimenting teacher’s observational notes as well as
the sorting and archiving of the corpus of the students’ work on and off
computer. As far as the students’ work on the computer is concerned we used specially
designed screen capture software -called Hypercam- which allowed us to record students’
voices and at the same time to capture all their actions on the screen. Trying
to attend to the full range of the communicational forms that students used in
the meaning-making process in data analysis we followed a multimodal discourse analysis
method viewed through a social semiotic lens (Jewitt, 2009). Initially data
were transcribed in a multimodal way focusing on students’ situated choices of
resources rather than emphasizing on the system of the available resources. In
an attempt to overcome the limitations presented by a sequential organisation
of data and to present simultaneously multimodal phenomena, matrices with
columns were used. As a unit of analysis we used the ‘multimodal episode’. The
multimodal data were divided in episodes that constituted easily discernable
parts of children’s actions and interactions with a clear focus point (Noss
& Hoyles, 1996, p. 148). Thus ‘multimodal’ episodes do not represent some
quantifiable entity but are chosen to represent clearly the kind of activity
that was going on in specific time in the classroom. As episodes have been
extensively used as a unit of analysis in the framework of qualitative
researches, the term multimodal has been added so as to stress that the episodes
used in our analysis do not rely only on oral or written language but comprise
also gestures, visual images, instances of students’ symbolic work on and of
computer etc. The results presented here are based on the work of one group,
consisted of one boy and one girl, while focusing on the way gestures were used
during the construction processes.
Gestures as a means of semiotic mediation
Use of gestures and turtle’s navigation
It follows from the data that while the
students were trying to navigate the turtle in the simulated 3d space they
basically used two kinds of gestures: a) dynamic representational gestures and
b) abstract deictic gestures.
It seems that the ‘play the turtle’
metaphor (Papert, 1980) cannot be realised physically as far as the 3d Turtle
Geometry environments are concerned: In these environments the turtle moves in
all the 3 dimensions without any restriction while in the real 3d space the
human body can only move in a 2d horizontal plane. As a result students used
extensively their palm so as to represent the 3d entity and its orientation as
well as its motion in the simulated space. The palm was used as a 3d object
analogous to the computational turtle, as it has distinct ‘place’
characteristics, up-down, forward-backward and right-left, it can be moved in
all the three dimensions, while it is easily manipulated and observed. In the
following episode the students are trying to decide how to carry on the
turtle’s journey in the 3d simulated space during the first task using a series
of dynamic representational gestures. The use of the palm seems to contribute
to the visualisation of a series of successive spatial representations before
these representations are systematically articulated either verbally in
everyday language or symbolically through logo code. Thus gestures were used as
a link to the embodied turtle metaphor that underlie Turtle Geometry
environments and as an intermediary stage between lived experience and
institutional signs such as Logo code. Moreover these gestures seem to provide
the context in terms of which students verbal expressions are to be
interpreted. It is indicative that the students’ utterances cannot be
understood if not accompanied by the respective gestures.
S2 |
Now, do you know what we should do?
As it is like that, to turn it this way and to move it forward. |
 
|
S1 |
Not to move it down a bit? |
|
Episode 1: Dynamic
representational gestures
In the present research the palm was not
used only representationally but also deictically to indicate the turtle’s
direction in the 3d space. In the following episode which took place during the
1st task, the students are trying to decide how many degrees the
turtle should turn so as to take the intended position and direction. As the
focus point is not turn’s direction but turn’s measure, the palm is rather used
as an indication of the various turtle’s position for certain angular turns in
the 3 space and in particular for the left turn of 45, of 90 and of 135
degrees.
S1 |
Fine. We will turn it. Wait it is like
that. So half of it, approximately 45, 90,
approximately at 135 … |
 
|
Episode 2: Abstract deictic
gestures
It should be stressed that the students’
palm was not used deictically as far as certain concrete objects or attributes
of the context of the activity are concerned, but that these gestures
integrated abstract deictic characteristics. Children’s gestures rather
exhibited geometrical knowledge and helped students cope with the abstractness
of certain mathematical concepts such as angular turn in 3d simulated space. These
abstract deictical gestures called for mathematical interpretation and implied
a metaphoric use of the real space, where certain angular measures had acquired
spatial properties. The aforementioned kinds of gestures - as well as the kinds
of gestures that are presented in the next section- are here considered as
signs that were invented and used by the students as an auxiliary means of
solving the given tasks while using the specific digital tools. On the one hand
these gestures are related to the accomplishment of the task and on the other
hand they are rather related to the mathematical content that is to be mediated
bringing in the foreground the complex relationship between digital tools, task
and mathematical knowledge.
Use of gestures and 3d graphical objects’
construction
It follows from the data that when the
students’ focus point was on the construction of 3d graphical objects, two kinds
of gestures were used: a) static representational gestures and b) dynamic
representational gestures.
In the following episode the students are
trying to translate their intuitions in visual representations so as to represent
geometrical figures and their orientation. The gestures that are used could be
considered as representational gestures as they have a degree of resemblance to
the desired geometrical figure. In the following episode that took place during
the second task, while trying to match real to corresponding virtual 3d
objects, the students are using firstly the one palm to represent the position
and direction of the one wall/plane that they want to construct in the 3d
simulated space. Then, the intended figure and the spatial relationship between
the two walls/planes are represented through the use of both palms. So the palms
seem to be used as intermediary transitive objects between the real object and its
figural representation on the computer screen. This kind of gestures rather depicts
spatially encoded knowledge and helps students conceptualise the spatial
relationships that should be then expressed in Logo code.
S1 |
Now as it looks this way, let’s make
the one wall like that.
|
|
S2 |
Yes that’s better. |
|
S1 |
Otherwise we can do it this way. |
|
Episode 3: Static
representational gestures
The representational gestures used in the
previous episode could be characterised as static, as they constitute static
instances of the intended figures. While the children were trying to move along
from static 3d representations to their design through the turtle’s navigation
another kind of representational gestures was noticed: dynamic representational
gestures. This kind of gestures seems to represent the 3d object not as a
static instance but as a result of the turtle’s motion.
S2 |
Up (90). From
this to go this way. |
|
S1 |
The staircase should rather be this way and not that way. |
 
|
S2 |
What? Will we do it straight? |
|
S1 |
Of course. |
|
S2 |
It should move
this way. Then it should turn. … rt, no. No, there is something else, how is
it called?
This way and
then this way. |
 
 
|
Episode 4: Dynamic
representational gestures
In the above episode the two students are
discussing about the way they should construct a staircase in the 3d simulated
space of MaLT. It was a task that was carried out by the students spontaneously
at the end of the second task and while they were waiting for the other groups
of students to finish their constructions. Initially student 2 suggests turning
the turtle up 90 degrees while representing this motion with his hand. The
other student having focused not on the turtle’s navigation but on the
staircase’s inclination in relation to the horizontal level corrects the former
showing with his palm the inclination that the staircase should have, which in
any case should not be vertical to the horizontal plane. S2 reacts asking if
they should do the staircase straight, understanding the other student’s
gesture as a straight inclined line. Then, he represents both the turtle’s
motion and the staircase moving his both palms. The palms side by side
represent the horizontal and vertical planes of the staircase, while the hands’
motion seems to represent the turtle’s motion in the simulated 3d space of
MaLT. In parallel he tries to translate verbally in Logo code the turtle’s
motion trying to find out the right turn order saying indicatively: ‘Then it
should turn. … rt, no. No, there is something else, how is it called?’,
looking apparently for the ‘downpitch’ order which corresponds to his
hands’ motion. It is rather interesting that the spatial arrangement of the
plane’s as well as the angular turtle’s turns are initially represented
visually and kinaesthetically and then verbally. The use of gestures was an
alternative sign, an alternative way of embodying and organizing information
that the student was not able to express in purely verbal or formal ways. It
should be stressed that these situated gestures denoted the intended figure not
so much pictorially but through actions and as a result of it while playing a
mediating role between internal, subjective imagery and shared conventional
logo code.
Conclusions
In the present research gestures were
understood as signs/symbols that mediated the mathematical knowledge integrated
in the computational environment (Radford, 2009, Arzarello et al., 2009). Gestures
were used in order to objectify, to attribute meaning to mathematical contexts
and contents interpersonally and intrapersonally. A virtual gesture space was
created in front of the students as a result of the use of gestures, where the
various represented mathematical objects were placed, processed and
interconnected. This virtual space was ‘endowed’ with mathematical meaning that
was accessed and visualised kinaesthetically. Thus, gestures rather provided an
intermediary stage between real and computational objects that fostered imagery
focusing on images’ structure rather than on accuracy (e.g. the exact degree of
turtle’s turn) that rather reduced some of the cognitive load of problem
solving. In parallel, gestures offered a context without which students’ verbal
expressions could not be interpreted.
The gestures that students used seemed to have
helped bridging the cap between abstract mathematical notions and sensorimotor
experience. In the present research gestures were conceived as a way of
revealing unconscious aspects of concepts formation, while certain kinds of gestures
were rather strongly related to the embodied metaphors that underlie Turtle Geometry
environments. Dynamic representational gestures were used not only in order to
represent the turtle’s motion through a series of successive spatial images but
also in order to represent 3d geometrical objects as a result of the actions of
a moving entity. Moreover, it follows from the research that the students used
different kinds of gestures according to their point of focus. While they were
focusing on turtle’s navigation through body-syntonicity and the graphical
results of this navigation, the students used dynamic representational gestures.
On the contrary while they were observing 3d space as external observers and
not through the turtle metaphor they used either abstract deictic or static
representational gestures. The kind of gestures used in the various phases of geometrical
objects’ construction processes could rather be integrated in the broader
research interest on the perceptions students have in 3d virtual environment
(Hauptman, 2010) and the spatial dimensions of interactions though 3d avatars
(Petrakou, 2010). Highly visual 3d Turtle Geometry microworlds, such as MaLt,
seem to influence not only the kind of geometrical problems posed to students
but also and most importantly the way students interact with the medium and the
solution processes followed by them (Hollebrands et al., 2008; Jones et al.,
2010). In this framework gestures serve for students as signs that mediate
mathematical activity and knowledge and for researchers as a window that offers
a new perspective on how learners think and talk about mathematics.
References
Anastopoulou M., Smyrnaiou Z. xxxx. (year).
“Bringing intuitions of natural and virtual interactions into conflict: the
POLYMECHANON experience”. In E. Efthimiou, G. Kouroupetroglou, and C.
Vogler, GW 2011: The 9th International Gesture Workshop Gesture in Embodied
Communication and Human-Computer Interaction May 25-27, 2011, Athens,
Greece, pp 68-71
Arzarello, F., Paola, D., Robutti, O.
& Sabena, C. (2009) Gestures as semiotic resources in the mathematics
classroom. Educational Studies in Mathematics, 70, 2, p. 97-109
Bartolini Bussi, M. G.,
and Mariotti, M. A. (2008), Semiotic mediation in the mathematics
classroom: artifacts and
signs after a Vygotskian perspective, in: Handbook of International
Research in
Mathematics Education, second revised edition, L.
English, M. Bartolini Bussi, G.
Jones, R. Lesh, and D. Tirosh, eds., Lawrence Erlbaum, Mahwah, NJ., pp. 746-805.
Clements, D., & Sarama, J. (1997).
Children’s mathematical reasoning with the turtle programming metaphor. In L.
English (ed.), Mathematical Reasoning, Analogies, Metaphors and Images (pp.
313–338). Mahwah: Lawrence Erlbaum Publishers.
Cobb P., Confrey J., DiSessa A., Lehrer
R., and Schauble L. (2003) Design Experiments in Educational Research, Educational
Researcher, 32 (1), 9-13.
processes in a 3d Simulated Space, Informatics
in Education, 6, 2, 1-14
Gravemeijer, K., & Cobb, P. (2006).
Design research from the learning design perspective. In J. Van den Akker, K.
Gravemeijer, S. McKenney & N. Nieveen (eds.), Educational Design
Research (pp. 17-51). New York: Routledge.
Hauptman, H. (2010). Enhancement of
spatial thinking with Virtual Spaces 1.0, Computers & Education,
54(1), 123-135.
Healy L. & Kynigos, C. (2010).
Charting the microworld territory over time: design and construction in
learning, teaching and developing mathematics. ZDM the International Journal of
Mathematics Education, 42, 63-76.
Hollebrands, K., Laborde, C., &
Strasser, R. (2008). Technology and the learning of geometry at the secondary
level. In M. K. Heid & G. Blume (eds.), Research on Technology in the
Learning and Teaching of Mathematic, Volume 1: Research Syntheses (pp.
155–205). Greenwich CT: Information Age.
Jewitt, C. (Ed.) (2009). The Routledge Handbook of Multimodal
Analysis. London: Routledge
Jones, K., Mackrell, K., &
Stevenson, I. (2010). Designing digital technologies and learning activities
for different geometries. In C. Hoyles & J. Lagrange (eds.), Mathematics
Education and Technology: Rethinking the Terrain (ICMI Study 17) (pp. 47-60).
New York: Springer.
Kaput, J. (2005) Building intellectual infrastructure to expose and understand
ever-increasing complexity, Proceedings of 29th PME
Conference, Melbourne, 1, 147 – 154
Kynigos, C. (1997). Dynamic
representations of angle with a Logo - based variation tool: A case study. In
M. Turcsanyi-Szabo (Ed), Proceedings of the Sixth European Logo Conference (pp.
104-112). Budapest, Hungary
Kynigos, C. & Latsi, M., (2007)
Turtle’s Navigation and Manipulation of Geometrical Figures constructed by
variable
Lakoff, G. & Núñez,
R. (2000). Where mathematics comes from: how the embodied mind brings
mathematics into being, New York: Basic Books.
Latsi, Μ.
(2010), Estimating the measure of angular turns of analogical clocks’ hands. In
proceedings of the 2nd Panhellenic Conference of Educational
Sciences, Athens (In greek).
McNeill, D. (Ed.)
(2000). Language and Gesture: Cambridge: Cambridge University Press.
Morgan, C., Alshwaikh, J. (2009). Mathematical
activity in a multi-semiotic environment. Paper presented at the Sixth
Congress of the European Society for Research in Mathematics Education, Working
Group 6 Language and Mathematics.
Nemirovsky, R. (2003). Three
conjectures concerning the relationship between body activity and understanding
mathematics. In: N.A. Pateman, B.J. Dougherty & J.T. Zilliox (eds.), Proceedings of PME 27, 1, 103-135.
Papert, S. (1980) MindStorms –
Children, computers and powerful ideas. The Harvester Press Limited, London
Radford, L. (2009). Signs, gestures,
meanings: Algebraic thinking from a cultural semiotic perspective. Proceedings
of the Sixth Conference of European Research in Mathematics Education (CERME 6). Université Claude Bernard, Lyon, France
Radford, L., Edwards, L. &
Arzarello, F. (2009). Beyond words. Educational Studies in Mathematics, 70(3), 91 - 95.
Reggini, H. C. (1985). Ideas y formas: Explorando el espacio con Logo. Buenos Aires: Galápago
Van den Akker, J., Gravemeijer, K.,
McKenney, S., & Nieveen, N. (2006). Educational
Design Research. New York: Routledge.
Vygotsky, L. (1978). Mind in
Society: The development of higher psychological processes. Cambridge University Press
