Part IV: Can you interact with nonexistent physical objects and spaces?

Using embodied cognition as a framework, the preceding post revealed the probable role that the gestures and body movement played in creation of the objects. In this post, we focus on the causes of the accuracy of designer’s interaction with the objects and spaces. It will address the unanswered question,

How could designers corporally interact with a virtual, physically non-existent object that they created in their mind’s eye? And that too, with surprising accuracy?

About this post

It is planned to rely on multiple sources to answer these questions. First, we will depend on research literature, particularly on embodied cognition. This does not deal specifically with gestures in imagery and sketching. So, some elements of speculation cannot be denied, but at least, it offers a broad framework to support the ideas. Second source is the data that is available as series of experiments. We will also take support from anecdotal example, more to understand ideas than to support the conclusions.

So, the discussion that follows has following aims. 1] To offer a plausible explanation to the participants relying on specific kinds of gestures and movements during creation of these objects and spaces. It argues on why and when gestures and body movements vigorously mimic and do not mimic the physical characteristics of the shapes. 2] To explain the source of accuracy of designer’s interactions with the physically non-existent virtual objects while creating the shapes, and 3] To explain why, during summing up their solutions towards the end of the experimental sessions, the participants are able to repeat the identical gestures and body movements.

The creators selectively chose gestures and/or body movements to suit the design task at hand. The post will argue that their decisions to deploy or not to deploy them were prompted by the useful contributions that gestures and movements made to the evolving thoughts. Let us try to develop these arguments further.

During discussion in the preceding blog on embodied cognition, there were references to the close relationship between visual system and the motor system. This suggests that cognitive processes also encode the sensory-motor aspects as series of movements.1 The argument is built on this central idea of the partnership between the visual and the motor system. So, we first address the partnership related question,

Can the creator’s actions be viewed as close-knit partnership between visual decisions, movements and the thought process?

If the answer is yes, it will give us a framework to answer the questions that we started with.

The casual support to the idea of this partnership comes from several examples in our every day routine tasks that demand relying on mix of motor and visual system. They seem to work together in partnership. We perform such tasks efficiently because these two systems seem to work in sync. But are all partnerships same? For the purpose of this paper, we plan to categorize the partnership into two broad groups. We are specifically focus on the second, which is of interest to us in the context of this paper. So we only briefly touch the first and move on.

1] Creative partnership where gestures and movements are based on convention or tradition

In this category, the planned visual effects are achieved through ‘pre-designed’ body postures, movements and visual expressions. The visual and the motor systems work in sync. They balance the cerebral and the corporal.

The major difference is that the gestures and movements are not free to evolve, but were defined by codes. They are motivated by convention or tradition, and are often formally learnt, practiced and produced. These events give ample evidence of how visual and motor systems work in sync.

Typical day-to-day example would include writing and production of your own signature and learning to write new script.2 Formal example to understand the category would be live performances of classical dance, street play, fashion shows and even performances of acrobats in circuses.3 These gestures and movements are either controlled by tradition (as in classical dance forms) or by convention (as in writing, plays and fashion shows).

These categories of gestures and movements are unlike those produced in our experiments. Explanations of learning, producing and use of these gestures are available as footnotes cited above. This ensures that we maintain our focus on the second category i.e. the spontaneous and intuitively produced gestures and movements.

2] Supporting the partnership with intuitive and spontaneously produced gestures and movements

A typical example to explain the idea would be the gestures accompanying natural speech. These gestures are spontaneously produced to support the evolving thoughts. They reflect the thinking in some ways. They show how the cerebral and corporal work in partnership.

Better and more convincing examples in this category are the gestures produced when giving route directions to a remotely located third party. They are truly embodied creations used everyday. The gestures clearly offer a corporal support to the cerebral processes.

The synchronization between the visual and motor systems continues like in earlier cases. It is not dictated by practice, but by the emerging thoughts and ideas. Our focus is on corporal actions (gestures and body movements) as supports to the cerebral actions (thinking process). Spotlight on this is justified because we feel that this is how the designers and architects have used gestures in our experiments.

A formal and compelling example is a live music concert. The singers move their hands and torso. Accompanying artists too move their bodies in sync. These gestures and movements support the primary task of creation and production of live music. (Also see ‘Can we prevent designer from sketching’ in this blog series)

Such gestures are very similar to the way they occur in our experiments. They are in sync with the evolving thoughts and ideas and so, reflect the thought process. They are very much a part of the process of creation. Such spontaneous use of gestures would be of interest to us in the current discussions.

Like in the live concert example, in our experimental sessions, the gestures were spontaneously produced. They appear to respond to the creator’s evolving thoughts. Most of them seem to incrementally shape the ideas of objects and spaces. They often mimic the shapes and/or indicate the locations of features in the space, as if these exist in front of them.

Let us focus on these experiments with architects and designers that we reported in this series of posts. The participants in the experiments were not briefed and had the option of using or not using the gestures. They were free to deploy gestures and movements, if and when they chose.

Identifying partnership to interact with the physically non-existent shapes and spaces

Transcripts show these gestures and movements do not appear as consciously ‘designed’, but look spontaneous and well integrated with the act of creation. They had clear roots in the thought process that resulted into in creation of objects/spaces. Obviously, individual differences do exist in the way movements were produced. Yet they display internal consistency.

Such occurrences are also true of most speaking gestures. The creators seem to develop the rules of their own and sometimes improvise on the spot. Such individualized gestural behavior is also common in live music during concerts. Yet, there is a difference between gestures in music and in design.4

By revisiting events in the experiments reported in the earlier posts, we plan to understand these spontaneous gestures and body movements of the designers and architects during the act of creation. Because there are differences in the nature of the assignments handled as well as in the way the designers and architects used gestures and movements, it is better to discuss them separately.

Revisiting SP’s actions: Casserole case

Let us look at SP’s performance under experimental condition one, where he was blindfolded. (See ‘Part II: Reflecting on gestures as design behavior’ in this blog series) Video transcripts show large segments, where SP moves his hands to shape the casserole, responding to his evolving ideas and thoughts.5 The transcripts suggest that he was describing the shape to himself and experiencing it. But there is much more to these simple actions.

The actions dealing with the shape issues show close correlations between the decisions dealing with evolving image and the accompanying gestural kinesthetic sequences. (See video 1 and 2) SP’s palms and fingers are actually seen mimicking the shape in 3D space as truthfully as he could. (See also video 4 in the next section)

Video 1 : The segment defines the overall shape. Watch how the palms and fingers define the curved surface. The relative angles define the angular relationships of surfaces with each other.

Video 2 : In the later part, the segment discusses complex detailing, where the legs are conceptualised to be folded-up during packing.

On the face of it, SP seems to be mentally shaping the evolving visualization of 3D geometry as an image in his short-term memory. This mimicking of the shape with hand gestures perhaps ensures a close partnership between, 1] sets of sequential (gestural) kinesthetic actions in 3D space, leaving traces of motor system in action, 2] the image in the mind’s eye, which he continues to modify, visualize and consolidate through visual system. Both processes seem to happen simultaneously.

Accurate recall of form of the object

SP could repeat identical movements while revisiting the feature for modifying at different points of time during designing, as well as during final description of the shape created. So accurate was the description of his final image and reproduction of hand movements and gestures that shaped that image, that even other designers could decode them accurately and reconstruct the casserole and bowl shapes from the video. The accuracy in performance is surprising.

Holding a stable image in the mind’s eye as well as reproducing gestures that shaped the image with surprising accuracy is not a trivial fit, particularly considering that the gestures were spontaneously produced on the spot, they were shape dependent and were not practiced, nor bound by conventions.

The visual processes and motion processes are working in sync and together they are responding to the emerging ideas and thoughts. The close synchronization and partnership between the two systems seems to have led to stronger and stable creations that you could recall and modify at will.

Revisiting architects’ actions

Architects, when blindfolded, moved in the real world space and used gestures differently. The large sizes of the enclosing wall envelopes and other architectural elements were beyond the grasp of their hands. So, they figured out the way to compensate. Two of them visually built spaces by walking around, as if they were on a real site. They developed the spaces around themselves and used body movements to create the layout and feel the scale of the spaces.

It appears that they chose to rely more on the visual system to generate the walled enclosures in the mind’s eye. Simultaneously, they depended on body movements and the motor system for the geometry of the layout and to scale the spaces. To indicate relative positions of features and walls, they selectively used gestures and virtual locations of their body, on the virtual site that they visualized. In such cases, gestures played a secondary role, more as a complement to the body movements.

In two out of four architects, their physical movements in the hall could be mapped to the layout of spaces they were creating in their mind’s eye. These physical movements in the hall could have left interlinked memory traces. Transcripts show that their movements match the walkthroughs in space when they were creating in their mind’s eye. That is how they could effortlessly move in and out of these built spaces, ‘feel’ them, and to get the ‘view’ from the position they visualized they were standing at. This explains why, when asked what their built-form looked like, they raised their head and even stepped back to ‘see’, though the image was only in their mind’s eye! (See video 3)

Video 3 : The architect was asked to describe what his building looked like. While answering, he took a step back, looked up as if he was scanning the face of the building.

Accurate recall of architectural design

As a part of protocols of the experiment, the architects were asked to describe their final solutions after they completed their design. Some preferred to walk the talk while describing; and these walking movements were identical to what they had performed when they conceived these solutions. This also explains why during descriptions of the final solutions, two architects could walk accurately through the spaces that they had created. Another architect when asked, could walk back to the virtual entrance without error.

None of these performances are trivial. Probably, their actions may have led to strong visual and motor memory traces that they could fall back on.

When gestures? When body movement?

Architects and designers appear to spontaneously produce gestures and movements when they are a convenient support in 1] Development of the image in the mind’s eye and/or operating on it, 2] Conceptualization of enclosure spaces around them, 3] Exploring spatial relationships of elements and scaling the shapes and spaces.

Preference to depend on visual system, and use or not to use motor system (gestures and/or movements or mix the two) depended on individual capabilities, design context as well as the complexity of the design assignment or its details. It also depended on the context of the experiments. (i.e. Sitting or allowed to move) SP’s actions are restricted by the fact that he was sitting with a table in front. Besides, his design assignment was also a tabletop object and so, within his grasp. He used hand gestures to shape all the features.

It is interesting to note that the choices between the gestures and body movements were largely interchangeable. There is also sufficient evidence of effortless switching between the two. For example, architects, who generally walked and used body movements, could adopt quickly to the use gestures, when they found this efficient. For example, when the idea of entrance demanded a sculpted shape and flow of surfaces, she used the hands and palms almost the way SP used them. (See video 4) Similarly, some occasionally went back to their traditional methods and scaled down the creations to see the built spaces as a small table-top model, being inspected from top and even shaped it using gestures to match their evolving thoughts. (See video 5)

Video 4 : The segment where the architect shapes the entrance with her palms, as if she is sculpting it by hands. The limitations of the grasp are clearly visible.

Video 5 : The architect switched to treating the creation as a scale model that she visualized and partly shaped it using gestures.

Exception to the rule

Though most of the designers and architects chose to use gestures, to construe that they are essential part of the thinking process may be too hasty. As we saw earlier in the context of mental rotation, individuals either use visual system, favour the motor system or a mix of the two, to perform such tasks. They choose what they consider appropriate for the task or what they are comfortable with.

Amongst several experiments conducted, there is one unique example in this series where the architect sat on the chair throughout, with hands on the hand rests of the chair and developed and spoke of his ideas without any gestures or body movements. He produced quality work and was as detailed as others, who used gestures. He walked through the building virtually, describing people populating the spaces, the ambiance of light and shades and so on. He seems to have depended on visual system completely. When the experiment was over, he was asked about it. He replied that he could have used gestures, but was able to visualize it clearly in imagery equally well. He did exercise his choice to depend on visual system.

In all my initial research writings on these experiments, I explained gestures and body movements by referring to them as ‘thinking with body’. Later, I hinted to its roots in hinted of spatial intelligence.6 I had no clear idea of their relationships with evolving thoughts and ideas. It was during the writing this blog, that I realized the full force of spatial intelligence perspective. Later, I realized that embodied cognition and embodied design offer a even more effective framework to explain why the creator’s gestured and moved their body. If we look at these events and actions through this new lens, lot of events and actions are easily explained.

These are of course post-facto explanations and there are some speculations involved. But, I thought they are intuitively appealing. However, it is best leave it to the cognitive scientists to explain how and what prompted these actions.

Summing up

Have we answered these questions that we started with?

How could designers corporally interact with a virtual, physically non-existent object that they created in their mind’s eye? And that too, with surprising accuracy?

We now have reasonably clear explanations to these questions. In doing that we have also identified several unique, non-trivial and surprising performances of the designers and architects that need a special mention. We have listed them below.

Non-trivial discoveries in design behaviour

1] Our results of the experiments reported so far seem to support the idea that the visualization of objects/spaces by the visual system was assisted by motor system that contributed to creating, altering and maintaining the continuously transforming shape and spaces. These two linked systems seem to lend stability to the image and allow the designer to revisit them at will, and interact with them with accuracy.

2] Video transcripts give a feeling that the creators were treating the creations as if they were real events unfolding in front of them. There is sufficient evidence of the creators being present in the situation and taking decisions, accounting for the location and orientations of the features. Because of the visual and motor systems working as partners in the process of evolving and modifying objects emerging in their mind’s eye, designers seem to accurately and almost effortlessly, interact with these nonexistent entities using their gestures and body movements

3] Creators are able to recall and reconstruct the current states as an image with a fair accuracy. They could also describe their images in details with certainty. This suggests that the images that they recall in their mind’s eye are relatively stable at any point of time. Even the physical actions that shaped these creations could be recalled and repeated without error.

It appears that the entire history of the gradual evolution of these shapes and layout spaces is available to the creators.

4] The actions like gestures and movements of the body accompanying the speech strings were so accurate that could be decoded by third party to understand and reach the final design in the creator’s mind.

None of these are easy to explain, but an attempt is worthwhile. The reasons not only include the cognitive aspects, but go beyond. So, in the next post we will attempt this.

Preview of the next post

To effortlessly perform these non-trivial actions at different points of time in the design process is a surprising fit. We have tried to attribute this to stability to the otherwise fragile mind’s eye images.

Can you attribute the performance in these non-trivial tasks directly to the linking of visual and motor systems at a cognitive level? Or are there also other factors in these actions?

To perform these actions at different points of time in the design process is a surprising. We have tried to attribute it to the duel encoding that may have given stability to the otherwise fragile mind’s eye images. In the next post, let us explore other answers to this. Though we can never be sure, based on the evidence before us, we can at least come up with some conjectures.

Notes and references

1     Pande, P., & Chandrasekharan, S., (2017) Representational competence: towards a distributed and embodied cognition account. Studies in science education, 1-13

2     In many routine tasks, along with visual system, the motor system (movement memories) is also primarily used during learning as well as in performance. Most common example is the reproduction of our own signature. While signing, the flow is monitored by visual system as well as by the motor system. They have to work in sync and as partners.

Another example. In traditional method of learning to write, the children are given learning templates that indicate starting point/s and the direction of pencil movement. They are also encouraged to pronounce the sounds. The directions on the template control the final appearance of the marks on the paper as well as the correct movements of the hand. The practice also ensures creation of movement memories.

Video 6: Child practicing script. The appearance of the letter is controlled visually and the movements by the instructions given on the template. (video courtsey Santosh Khirsagar)

That explains why, little children can play the game of letter recognition by writing on each other’s back with finger. These are clear case of cerebral and corporal working together.

3     Creative partnership, where gestures and movements are based on convention or tradition

In classical dance, the gestures and body movements are directly responsible for the primary visual effect. So, the visual and the motor systems work in sync. The planned visual effects are achieved through ‘pre-designed’ body postures, movements and visual expressions. The freedom to evolve and change is within the framework of tradition and convention. The partnership has to be consciously and painstakingly learnt through practice till they are well synchronised.

Modern dance has the freedom to innovate new gestures and movements. Choreographers of modern dance and street play can explore new creative directions. Because they are driven by creator’s thoughts during conceptualiaation, they are closer to our experiments in design. They balance the cerebral and the corporal.

Closer to our example is when the modern dance performance is being choreographed, particularly modern dance. We could have included such cases to balance the findings. However, we are forced to exclude them from the current discussions because we did not have the opportunities to study the practices and actions involved during choreography.

4     In live music, the singer produces gestures spontaneously. The listener is aware of the gestures and uses them to understand and appreciates the performance. He may even synchronize his bodily responses to these gestures. In design, the final creation of objects and spaces stand independently and do not even offer clues to the gestures that were used to create the object. They play no visible role in the appreciation of the final output. In the act of design, gestures act more like backstage tools.

5     That the transcripts also show different kinds of gestures and all the gestures do not necessarily mimic the shape. For classification of these gestures refer to reference 6 below.

6      Athavankar U. A., (1999) Gestures, Imagery and Spatial Reasoning. In: Garo JS, Tversky B (eds) Visual and Spatial Reasoning. Preprints of the International Conference on visual and spatial Reasoning, (VR 99) MIT, Cambridge, June 15–17, 1999, pp 103–128.