Re-thinking the interface: What is the meaning of Alan Kay's"Doing with Images makes Symbols"

Alan Kay created a slogan for development of user interfaces: Doing with Images makes Symbols. This reveals the essence of the Graphical User Interface (GUI) which millions of people now use all over the world. This paper examines how GUI, based on Kay's slogan, has changed our communication with the computer. In order to explore this topic, I focus on Azuma and Turkle’s argument of "interface value" intuit for thinking about the relationship between our action and new sign surfaces. Next, Kay’s slogan is considered from the viewpoint of its formation process. Finally, I refer to Sperber and Wilson's Relevance theory. I conclude that GUI demands that we, along with computers, act for ostensive-inferential communication in a mutual cognitive environment.

What You See Is What You Touch? Double description of seeing and touching in ‘Display Acts’

What you see is what you touch?

View more presentations from Masanori MIzuno.

There is, in fact, almost no formal theory dealing with analogue communication and, in particular, no equivalent of Information Theory or Logical Type Theory. This gap in formal knowledge is inconvenient when we leave the rarified world of logic and mathematics and come face to face with the phenomena of natural history. In the natural world, communication is rarely either purely digital or purely analogic.1

This is Gregory Bateson’s words. I consider that user interface has the same problem. Computer is digital. Computer is controlled by formal theory. However, We, human, is not purely digital. We, are analogic rather than digital. Therefore, user interface should be the place where purely digital computer becomes analogic one, and analogic human becomes digital one in order to communicate each other.

Display Acts

When we use a computer, what do we do? Almost all of us look at some images on an electric display, grab and move a mouse, and type on a keyboard, then our right hand holds the mouse in order to point to an image called an icon on the display. This is very 'natural' for us. If our body makes some actions with a plastic object, then the images on the electric display change. However, this relationship between our body and the image did not exist until the computer, and especially until the Graphical User Interface (GUI), appeared. I call this phenomenon 'Display Acts': action which is formed by connecting our body action with the change of images on the electric display.2 However, we do not understand this action well because we have only separately studied body actions and images in man-computer communication.

In order to make clear the relationship between our body action and images in the man-computer world, this presentation focuses on the cursor in the electric display and the mouse in the real world. We see the cursor in the electric display: The cursor is "a small mark on a computer screen that can be moved and that shows the position on the screen, where, for example, text will be added."3 Moreover, the shape of cursor is almost arrow (→) which is "used to show direction or position."4 We just see the arrow cursor in order to know where it is and point some images on the display. Then, we also hold the mouse.

Exemplification

What does the cursor show us? It shows us the position and direction, however, the cursor often points nothing. When the arrow cursor points nothing, it does not work. We don't care this. Why? Because we not only see the images in the display, but also grabbing the mouse. What is a relationship between seeing and grabbing? In order to make clear this relationship between the cursor and mouse, I refer to Nelson Goodman’s idea; Exemplification. According to Goodman;

Exemplification, though one of the most frequent and important functions of works of art, is the least noticed and understood. Not only some troubles about style but many futile debates over the symbolic character of art can be blamed on ignoring the lessons, readily learned from everyday cases of the relation of being-a-sample-of, that mere possession of a property does not amount to exemplification, that exemplification involves reference by what possesses to the property possessed, and thus that exemplification though obviously different from denotation (or description or representation) is no less a species of reference.5

Possession means that some samples show us their own properties, and we interpret their properties for our interests. The user interface is not work of art, but I consider that the images in the electrical images with the mouse or other input devices show us almost same functions of work of art.

We see the shape of cursor as a sample in order to know functions of cursor. We get a pointing function from the arrow shape, therefore, we label this pointing function to the arrow cursor. However, the arrow cursor points nothing. As a result, we think this arrow possess another functions; Maybe it is moving for pointing because the arrow is shape for pointing. Only when we see the electric display, we can understand that the functions of the cursor is only pointing. However, user interface has the mouse. We touch and grab the mouse, then we see that the cursor moves and point the image. This touching is another function of the cursor because the cursor is designed for connecting with the mouse in order to work in the display. Only when we see the display, we can just know pointing and moving functions of the cursor (fig. 1). Only when we touch the mouse, we can understand the cursor is able to touch the image and change images in the display (fig. 2). Only pointing can not change anything in the display. Changing needs touching. In short, we not only see the images on the display, but also grabbing the mouse when we use the computer. Therefore, we have to consider what a relationship between seeing and touching is.

　　　　　　　　　　　　fig.1

　　　　　　　　　　　　fig. 2

Moreover, the arrow cursor sometimes suddenly changes into the spinning wait cursor or something else. When we see transformations of the cursor's shape, the mouse we grasp does not change. However, the moment the arrow cursor changes into the spinning wait cursor, our action with the mouse changes. Before transformation, we use the arrow cursor and the mouse in order to point to an image on the display, and therefore the cursor exemplifies the function of pointing and our body action with the mouse is formed for pointing. After transformation, the spinning wait cursor exemplifies the function of, not pointing, but showing the computer status. As a result, our action with the mouse is not formed for pointing. After a moment, we realize that what we can do with the mouse in hand is just moving the cursor in the electric display. The spinning wait cursor exemplifies that the computer is still working, but it does not leave us alone; the cursor and the mouse are connecting, please wait. Throughout the whole process, we grasp the same plastic object, even though the cursor's shape is changing.

Double Description

In user interface, the image we see is changing while the object we touch is not changing. Moreover, this transformation of image is sometimes out of our control. But we naturally accept this phenomena. In order to make it clear, I refer Bateson’s Double Description. He quotes Shakespeare’s Macbeth and says that:

This literary example will serve for all those cases of double description in which data from two or more different senses are combined. Macbeth “proves” that the dagger is only an hallucination by checking with his sense of touch, but even that is not enough. Perhaps his eyes are “worth all the rest.” It is only when “gouts of blood” appear on the hallucinated dagger that he can dismiss the whole matter: “There’s no such thing.” 6

We need Double Description of seeing and touching in order to consider and prove what the user interface is. However, user interface mainly focuses on seeing. For example, What You See Is What You Get; WYSIWYG. From Wikipedia, the main attraction of WYSIWYG is the ability of the user to be visualize what he or she is producing.7 Here, ‘to be visualize’ is emphasized. One more example. Ben Shneiderman proposed a very important idea for user interface: Direct Manipulation. From Shneiderman, Direct Manipulation’s central idea is to be visibility of object of interest.8 Here, only visibility is emphasized.

I want to focus on Direct Manipulation from anther point of view. According to George Lakoff and Mark Johnson. Direct Manipulation leads us not only seeing but also touching because we see and touch the object in order to create something. And they say that the Direct Manipulation makes the concept of CAUSATION.9 Furthermore, Søren Pold connects GUI with the concept of CAUSATION. He says that WYSIWYG interface has many buttons, and buttons are essential part of controls in GUI10 and also continues that:

When pushing a button in a interface ─ that is, by movement of the mouse, directing the representation of one’s hand onto the representation of a button in the interface and activating a script by clicking or double-clicking ─ we somehow know we are in fact manipulating several layers of symbolic representation and,…… It is a software simulation of a function, and this simulation aims to hide its mediated character and acts as if the function were natural or mechanical in a straight cause-and-effect relation.11

According to Pold, pushing button in the user interface makes a straight cause-and-effect relation in the electric display, although this cause-and effect appears by hiding manipulating several layers of symbolic representation.

Another view of the cursor and mouse. A media artist, Masaki Fujihata says computer does not have a question of materiality.12 This means that computer has different principle from us because we live in the material world. Fujihata considers that if the repeated experience are provided us, through interactive experience, Images are made into object.13 According to artist’s intuition, we go through new phenomena: Image are made into object.

What You See Is What You Touch?

However, this ‘object’ in the display is not real object. According to Pold, this ‘object’ in the display shows us cause-and-effect, therefore, we feel as if we directly manipulate it by our own hands. On the other side, Fujihata says that the ‘object’ does not have its own materiality. We, of course, see the ‘object’ but can we touch it? Maybe, yes. Maybe, no. I consider that this ‘object’ is something like “switch” as Bateson defined;

We do not notice that the concept “switch” is of quite a different order from the concepts “stone,” “table,” and the like. Closer examination shows that the switch, considered as a part of an electric circuit, does not exist when it is in the on position. From the point of view of the circuit, it is not different from the conducting wire which leads to it and the wire which leads away from it. It is merely “more conductor.” Conversely, but similarly when the switch is off, it does not exist from the point of view of the circuit. It is nothing, a gap between two conductors which, themselves exist only as conductors when the switch is on. In other words, the switch is not except at the moments of its change of setting, and the concept “switch” has thus a special relation to time. It is related to the notion “change” rather than to the notion “object.” 14

Like the concept ‘switch’, the ‘object’ is related to the notion “change” rather than to the notion “object”. The cursor is given a special relation to time by the mouse.

Why is this happen? Because the computer is purely logic machine which is made by analogic human in cause-and-effect world; “the if ... then of logic in the syllogism is very different from the if ... then of cause and effect.” 15 Bateson makes a question: Can logic simulate all sequences of cause-and-effect? His answer is maybe no.16 The reason is that “the if ... then of causality contains time, but the if ... then of logic is timeless. It follows that logic is an incomplete model of causality.”17 However, we want to re-made the computer as cause-and-effect machine via Double Description of seeing and touching. Introducing the special relation to time with the mouse, user interface makes new images with coupling cause-and-effect and logic in the electric display. We are making pseudo-cause-and-effect with logic in the user interface. Therefore, we touch the same plastic object, even though the ‘object’ in the display is changing.

                                             fig.3

Returning to the cursor and the mouse. The cursor looks like showing cause-and-effect, however, it stands for the logic in the computer: there is no time. But, we live in time: cause-and-effect world. Pold tells us that the user interface make us believe there is a straight cause-and-effect relation, even though this cause-and effect appears by hiding manipulating several layers of symbolic representation. On the other hand, Fujihata says that if we see cause-and-effect behind the image, then we can touch it. Almost all us do not understand what the logic in the computer is, but we use it with the display, the keyboard, the mouse, and something else. The reason is that we believe that the computer is not only logic, nor only cause-and-effect, but just ‘something’. Therefore, when we consider about user interface, we should not confuse ‘if ... then of logic’ with ‘if ... then of cause-and-effect’. We see the change of image in the electric display; it shows us the logic in the computer but we feel something like cause-and-effect, which is made from computer’s pure logic. We touch the same object while the image is changing: our grasping mouse introduces the special relation of time to the computer and makes pseudo-cause-and-effect in the computer (fig. 3). Therefore, only seeing is not enough and only touching is neither. In order to made clear ‘something’; the pseudo-cause-and-effect with logic, we have to consider the double description of seeing and touching in the user interface. In short, the computer has changed the relationship between our seeing and touching. It means that the computer changes our notion and experience for touching object: Touching not “object” but “change”.

References
1 Gregory Bateson, “Steps to an Ecology of Mind”, The University of Chicago Press, 2000, p.291.
2 Masanori Mizuno, “The formative process of “Display Acts” on the establishment of GUI”, Doctor Thesis,2009, p.3.
3 Oxford Advanced Learner’s Dictionary (electric version), Oxford University Press, 2000
4 Ibid.
5 Nelson Goodman, “Way of Worldmaking”, Hackett Publishing Company, 1978, p.32.
6 Gregory Bateson, “Mind and Nature”, Wildwood House, 1979, p.73.
7 Wikipedia, http://en.wikipedia.org/wiki/WYSIWYG (9, April 2009 access).
8 Ben Shneiderman, ‘Direct Manipulation: A Step Beyond Programming Languages’ in Noah Wardrip-Fruin
and Nick Montfort ed., “The New Media Reader”, MIT Press, 2003, p.486.
9 George Lakoff & Mark Johnson, “Metaphor We Live By”, The University of Chicago Press, 2003, pp.75-76.
10 Søren Pold, ‘Button’ in Matthew Fuller ed., “Software Studies: a lexicon”, MIT Press, 2008, p.31.
11 ibid., p.33.
12 Masaki Fujihata, “Masaki Fujihata: The Conquest of Imperfection ─ New Realities Created with Images and Media”, Center for Contemporary Graphic Art, 2007, p.178.
13 ibid., p.178.
14 Bateson (1979), pp.108-109.
15 ibid., p.58.
16 ibid., p.58.
17 ibid., p.59.

Origina text: Proceedings of 4th International Symposium on Multi-Sensory Design

We are groping around for the courtship action to the digital world which is not able to see and touch

I think that we have aimed to bring our action in the physical world into the virtual world as it stands until now. However, PINCH of iPhone may make us know that we do not necessarily bring our action into the virtual world as it stands. Although PINCH does not make something big or small in the physical world, we control two points of a diagonal line in order to draw a quadrangle in the virtual world like Adobe Illustrator. PINCH makes a movement of diagonal, therefore we may superimpose this hand movement and the movement for drawing a quadrangle in the digital world. As result, we naturally accept the relationship between PINCH and Zoom in and out. I consider that it is big problem to explain what makes us naturally accept the relationship between our gestures and the image action or feel something strange for it. We have to be careful and aware of what our unconscious gestures mean and correspond them to the image movements and actions in the virtual world.

However, it is our contention that gesture ─ certainly descriptive or 'iconic' gesture ─ necessarily involves indexical links to the material world, even though these links are rarely established or explicated in the communicative situation itself. Rather, in conversational contexts that are detached from the talked-about world, participants must fill in encyclopedic knowledge (ranging from universal bodily experiences to highly specific cultural practices) to see and recognize gesture. Phenomenally, there are only motions of the hands. What is perceived, however, is typically not motions but actions and, simultaneously, implied objects acted upon.

Curtis LeBaron, Jürgen Streeck, 'Gestures, knowledge, and the world' In David McNeill(Ed.), Language and Gesture, pp.118-138 (2000)

Finally, I think that a big change will happen because the digital which is no material and shape intervenes in our gesture communication based on the material and shape of our body. To show our will to the digital world. Conversely, the digital world shows its own will to us. I formerly called this correlation 'Display acts' which the image on the display changes corresponding to our actions. Now, I consider that 'display acts' is, connected with 'display' means the bird's action of the courtship, a courtship dance or action between the human and the digital furthermore, the human and the human through the digital. We are groping around for the courtship action to the digital world which is not able to see and touch. I want to feel the touch of the digital world.

Is multi-touch gesture poetic language or prose?

Is multi-touch gesture poetic language or prose? Maybe, it is wrong to ask this question, but I want to consider about it. When the computer came into our world, "metaphor" played an important role: the desktop metaphor. The desktop metaphor arranged the images on the display and the mouse in the order of our body sense. Now we used to use them, then the multi-touch gesture is coming into the world. Is the multi-touch gesture some metaphors for something. The metaphor connects two things which are normally separated. Moreover, it is baed on our body sense according to George Lakoff and Mark Johnson. If the mouse as a physical object gives us the feeling of "grabbing" for the images on the display in the desktop metaphor, what does the multi-touch gesture bring into the relationship between the human and the computer. I think that multi-touch gesture makes one-to-one relationship between us and the logical machine, which is like a prosaic sense or a sense of grammar/dictionary. However, my friend pointed out that the multi-touch gesture is like a poetic language. And I felt that his idea is also interesting so I have to re-consider about the multi-touch gesture.

iPhone introduced the PINCH:

With the iPhone multi-touch, zooming in and out of a picture never been easier. Steve Jobs introduces what he called a “pinch”. Bring your thumb and index fingers closer together or move them further apart to scale images.  Have a look at Steve Jobs presenting the feature, the audience is amazed by this.        

http://www.iphonefreak.com/2007/01/iphone_and_stev.html

 This thumb and index fingers action had made nothing to zoom the image or the physical object before the iPhone. Now, we think that our two fingers action have been able to zoom the image on the display from ancient time. This is the problem; Pinch is new action for us but we used to do it. Does iPhone just connect our thumb and index fingers action to zoom the image? Does the human have a sense for this finger action make zoom the image or the object? If there is only artificial connection our fingers action and zoom, this is one-to-one relationship like the dictionary. If we have natural sense for the link between fingers action and zoom, the Apple design and engineer team feel this sense and give it shape. Anyway, this it time that the digital device makes our action rule, therefore we must consider about these new action and its meaning in the long human history. I think that a new language based on our body, the image, and the digital device is arising now. I use "Language" as metaphor. We has changed the our own language in order to adjust our recognition for the environment. In digital age, I think that this change occurs at our body level very calmly.

I think that multi-touch gesture reduces our three-dimensional action to two-dimensional one. In order to reduce that, our body sense for the three-dimensional world have to be discarded. And, Pinch action for zooming the image shows this switchover. Pinch action need just two points of contacts in three-dimentional fingers action in order to zoom the image in the display. Even though we see and feel our own finger action which is three-dimentinal, iPhone need just two points of contact in the its surface. We live in three-dimantional world, so we always feel that our action for iPhone is the three-dimentinal one but the reduction happens there. However, we can not notice this reduction and feel "natural" for Pinch action. Pinch action is the artificial but we feel and make it "natural". The relationship is that "natural : artificial = poetic : prosaic"

A possibility of 10/GUI: Smooth but sticky

A possibility of 10/GUI. 10GUI suggests new interface with multi-touch gesture. We has used the interface with the mouse for over thirty years, so it is the time for change. However, I think that it is important for us to "grab" something physical like the mouse in order to operate the image on the display which we can not grab or hold. I consider that "grabbing" and "touching" are entirely different actions in human history. Therefore, our body may feel "grabbing" the mouse and multi-"touch" when we use the computer. But, I get used to the MacBook's glass trackpad and operate the image on the display like using the mouse. I do not superficially feel incongruity. When I pay deep attention to my body, however, I feel something strange between using the mouse and the trackpad, especially scrolling. Using the mouse, I feel "grabbing" the scroll bar for up and down. With trackpad, I feel that my fingers "stick to" the image on the display. Therefore, I feel odd when the scroll bar moving up and down because I do not "grab" it but just "touch". Although the human has "grabbed" something in order to move it for the long time, our fingers seldom stick to something in order to move it in daily life. I write "Stick to", however, it doesn't work if our fingers actually stick to the trackpad. Consequently, the trackpad must be smooth. I think that Apple made the glass trackpad for multi-touch due to pursuing the smooth and many people enjoy its feelings. While we use the trackpad, our body feel that it is smooth even though sticking to it in the bottom. "Smooth but sticky" makes us curious because of its newness and gives us something pleasant. On the other hand, it may cause the rejection in our body. In my case, touching trackpad is pleasant to my fingers and body in almost time but I sometimes feel something strange.

Back to 10/GUI. 10/GUI is good design for the image on the display in order to operate with our fingers. Therefore, many people interested in it. At first I think that  it is difficult to operate with multi-touch, but many user would get used to do soon. What it the material for 10/GUI's keyboard which is like a big trackpad? Is it the glass like MacBook's one? Our finger has high sensibility for touch, so the material which we directly touch is very important: we need always smooth. However, I don't know from this concept movie.

Moreover, if our body get used to multi-touch, we will get new body sense for the moving image. Once we get this new sense, we don't need to touch something in order to operate the image on the display, in other words, the computer like Tom Cruise in Minority Report. Anyway, I'm looking forward that multi-touch gesture open new future for us.

Not to be taken the image from the world, but to pile up the image on the world.

This moving image is an experience of Phillips Design Electric Tattoo. This is more like Media Art than Concept moving image. Watching it, I feel ticklish on my hand. I understand that the image is projected on the hand, however my body may feel that it is directly drawn on the skin. This image is very low resolution, so I must undergo this design work in order to what happens on the skin. But, I have to say that the image will be piled up on our skin and not just 'piled' but 'sticked fast' to our body in the near future. Fujihata Masaki, Media Artist, insists that "we can control the image which is (like) material as if the printings" due to the appearance of the organic electroluminescence(OEL) display. I think that not only OEL but also the projector technology will make the images change the material everywhere. We have taken the image from the world, however will pile up the image on the world. When this would happen, might we feel something material with the image? We "see" the image on the skin and feel ticklish because of seeing. At the same time, our skin may feel nothing. In the future, our body, five senses will be dismembered. Now, our body is becoming loose because we already have watch a lot of images with only touching the plastic mouse. Therefore, a relationship of seeing and touching becomes loose now. However, we can't feel this separation. We have to think why we can't feel that.

To connect our desire to digital technologies

Philips Design Electric Tattoo. When someone touches someone's skin, a tattoo is drawn from the place he/she touches. Phillips design says "Simulated by touch, an Electronic Tattoo traverses across the landscape of the body, navigated by desire." The word "navigated by desire" reminds me of drawing something with my fingers on the beach. When I draws something with my fingers on the beach, I feel more directness than drawing with pencil or pen. I become more free to draw. I don't know what the trained painter think about this, however I feel more direct and free when I draw with my finger on the beach. Finger is not tool like a pencil or pen but a part of my body, therefore this may bring me a fundamental feeling for drawing. Because of the fundamental feeling, I don't care what I draw, I just draw. Although I can't understand what I draw, it is very good feeling. Good or not is the problem for me. When I begin to write about Phillips Design Electronic Tattoo, I consider the above. Now, we have to connect our desire to digital technologies in order to consider about the fundamental relationship between us and our environment for  generating good feeling from our body. If the Electric Tattoo reminds us of the fundamental relationship, the Tattoo which we draw marks on our skin brings us to our ancient feelings in our body. Moreover, the digital technology may make the ancient feelings vivid for us now. Even though We insist on own identity with wearing clothes or hair styles now, the Electric Tattoo will play that role in our life. Not drawing tattoo on our skin, but displaying air tag or something around our body through AR technology play the important role to show own identity.

fluid-solid body

We can say that the body as substance is sometimes ‘not there’ in everything smooth. Our body is fluid-solid like a plastic in smooth information flow.

the new realm of tactility with the 'light-colour' entity and the smooth materiality

After seeing 'light-colour' for a long time, we just begin full-scale investigation for the new realm of tactility with the smooth materiality.

David Katz writes, ﾒWhat has been touched is the true reality that leads to perception. [Emphasis in original] (Katz, 1989, p.240)

We re-train our body with the smooth materiality in order to touch and generate new reality of 'light-colour'.

Before, we touch something physical in order to feel the weight (in the flow of cause and effect).

Now, we touch something smooth in order to give the weight to the 'light-colour' entity (for making its smoothness in the information flow).

This demand us to believe our bodyﾕs weight and density in the 'light-colour' world.

And this belief creates new diverse bodily sensations in ﾔDisplay Actsﾕ.