The (pseudo) causal chain

   http://zig.media.mit.edu/Work/G-stalt

In last entry, I considered why a gesture interface like G-stalt is studied in Tangible Media Group. Prof. Ishii says that:

Interfaces that control the computer with a mouse are like a black box, and it's difficult to understand what's going on inside. If gestures are used, however, one can clearly see the causal chain, as with a violinist's response to the conductor's baton. Research into gesture interfaces is attempting to hammer out a new paradigm by seamlessly merging body, objects, and space."

MIT Media Lab Tangible Media Group's Gesture Interfaces: Dynamic Interaction in Hybrid Space in AXIS 2009.8 vol.140

Although G-stalt seamlessly connects body, objects, and space, why the image is not in this relationship? The thing that is moved by the gestures is not the object itself but the image on the display, isn't it? Are 'body, objects, space and images' seamlessly merged in G-stalt? However, Prof. Ishii does not say the image. Why? Maybe, the images have already become the objects in our life. If we think that, I can smoothly connect the gesture interfaces with 'Radical Atoms' which is next vision of TMG. Now, we rarely think that the image is the 'image'. The image is something like the object, so it is tangible. Therefore, we can feel that the image is the object and the objects is the objects at the same time now. This idea makes totally new paradigm. This paradigm may be related with plasticity, I think. 

I want to focus on Prof. Ishii' word " If gestures are used, one can clearly see the causal chain". I consider that the mouse and the cursor on the display make "pseudo-cause-and-effect" between our action which is ruled by physical cause and effect and computer logic which is not ruled by one. Then, The pseudo-cause-and effect changes the images on the display into the 'object [entity]' which is something tangible. Even though the image which we see on the display has already become the 'object', it is supported by the pseudo-cause-and-effect. However, the gesture may interfaces remove  'pseudo' from the pseudo-cause-and-effect because the gesture is closely connected our body. Therefore, Prof. Ishii is maybe right. Our action will be tightly connected with the image on the display because the gesture is very close to our body. In the mouse and cursor, 'to grab and move something' is transformed into  'to point and move something' and this transformation makes the pseudo-cause-and-effect on the display. 'Grab' and 'point' are very general actions for us, then we can do many things with the combinations of these two actions on the display. As result, there is the ambiguous relationship between the image on the display and our action. However, the gesture requires one to one relationship with the image on the display. This one-to-one relationship make us think why an action is connected with the change of image on the display. Connecting our gesture with the image on the display in one to one relationship, the image may be more close to the object. Then, may the ambiguity of our action be reduced in turn?  

Atom → Bit → Pixel → Atom

I feel something strange that Tangible Media Group(TMG) studies the gestural interface which is not like the tangible. And more, TMG leaded by professor Hiroshi Ishii shows a new vision 'Radical Atoms'; Study the interface based on the material which can freely change like the pixel on the display. There is a stream of idea; Atom → Bit → Pixel → Atom. We have touched Atom for a long time. Then, we have to touch Bit like touching Atom. We have to touch Pixel like touching Atom. In the mean time, we have to touch Atom like touching Bit and Pixel. The first Atom does not change. It is just solid material. However, the second Atom can freely change. It is like fluid material. Now, we can change Bit and Pixel at our will but Atom does not. Prof. Ishii feels strange this solid Atom, therefore he give us the vision of 'Radical Atoms' for study the interface based on the idea that we can freely change the Atoms. I think that the second Atom is the new entity based on the appearance of Bit and Pixel which is always changing. Bit and Pixel is freely changing therefore we have to make a new interface in order to correspond them. It is g-stalt. This gestural interface make us re-think our actions to Bit and Pixel. This thought leads us new actions for the new fluid Atom. I think that this is 'Radical Atoms'.  In this way,　I understand why TGM studies the gestural interface like g-stalt.

You are a game controller

Project Natal Xbox 360. "You are a game controller." You don't need any controller. You can control the image on the display via your gestures. Talking of the gesture for game, I have Nintendo Wii which needs Wii remote for playing the game. We normally do actions with something physical like a racket for table tennis or a sword for Kendo. In fact, I can easily play those sports in Wii Sports resort. How about playing them without something physical to keep holding?  I think that I would have the strength for it. Although we hold Wii remote in order to play Wii games, it is different between the feelings for hitting a ball with a racket and for hitting a ball image on the display with  Wii remote which gives us sound and vibration. However, many people do not care this difference and feel as if Wii gives us 'real' feelings. We do not hold anything physical in Natal, therefore we can not feel some vibrations. We have to control our actions based on only the image on the display and sound. The sense of sight and hearing controls our whole body actions.

I consider about the feeling of swishing the air in soccer when I watch demonstration movie of Natal. It is like something empty. When try to kick the ball which is the physical entity, my body gets some tenses for a feedback from the ball.  Then, I miss my kick. Our body tenses can not go to the ball and return to my own body. If I understand that there is nothing physical contact, my body do not get any tenses. As a result, we can do our actions without any body tenses because there is nothing physical feedback in Natal. Maybe, we use our sense of sight and hearing as a substitution for our body tenses to the contact of something physical. Even though we do the actions in Natal as same as the ones in the physical world, how our brain reacts in this situation? If we keep doing this no physical feedback actions, our body will store new kind of experience and what will happen for us.

Writhing this entry, I remember that I did a practice swing without a racket many times when I belong to a tennis club in a junior high school.  Maybe, this practice teaches me 'form' for hitting the ball in the tennis. However, there is the difference between a practice swing with a racket and without racket because our body tenses change for the weight of it. 'Form' reminds me of 'NOU' or Pantomime. Natal and Pantomime. However, there is no 'weight' or physical 'resistance' in Natal in comparison with good pantomime. Anyway, we have to play long time in no physical feedback environment like Natal long time in order to understand its effects for us.

Magic Mouse Scroll with momentum.

Magic Mouse Scroll with momentum. iPhone does the scroll with momentum and it is comfortable for us. Magic Mouse, too. iPhone and Magic Mouse do the same scroll with momentum, therefore our finger action for them and the movement of the image on the display are also the same. However, I feel something different between iPhone and Magic Mouse. With iPhone, our finger move lightly on the smooth glass surface and the image on the display sticks to our finger movement. There is Smooth and Sticky. With Magic Mouse, our finger move along the gently slope on the sticky plastic surface and the image on the display sticks to our finger movement. There is Sticky and Sticky. Firstly, I think that "Smooth and Sticky" and "Sticky and Sticky" are a big different between iPhone and Magic Mouse, then the slow slope of Magic Mouse would be important for use because I feel very comfortable when my index finger go and come along it. However, a biggest different between them is that we touch the image directly in iPhone even though our finger and the image separate  in Magic Mouse.

Anyway, this scroll with momentum brings the momentum law of our physical world into the logical world of the computer, although the momentum in the display is generated from the logic. After going through many world, we feel good when the image on the display represents the law of our physical world which we are familiar with. There is something different, isn't it? The computer can not represent our whole physical world. It just cut out only the momentum from the physical world and gives it to the movement of the image. Moreover, the movement of the image with the momentum is adjusted for us in order to fit our finger action. As a result, we feel that something new happens, even though the image on the display represents just the law of physical world  which we are familiar with.

At first, I consider about a relationship between the scroll with momentum and the shape of surface; the slow slope of Magic Mouse fits our daily action for the slope because we get the momentum for going up the slope and the momentum is given us when we go down the slope. Therefore, the slope of Magic Mouse compensates the sticky plastic surface. However, I don't know whether our body feeling for the slope is reflected in our finger action for the scroll with momentum of Magic Mouse, even though the image on the display fits our finger action with the momentum which is cut out from the physical world. Although the slope of Magic Mouse may be just the shape for easy to grab in practical manner, it is the fact that the plastic surface of Magic Mouse has the slope in comparison with the glass surface of trackpad is flat. Finally, I understand that we cut out the law of physical world in order to bring it the logic world of the computer and the relationship between the material like a plastic or a glass and the shape; The glass - Smooth-Sticky-Flat. The plastic - Sticky-Sticky-Slope.

We are groping around for the courtship action to the digital world which is not able to see and touch

I think that we have aimed to bring our action in the physical world into the virtual world as it stands until now. However, PINCH of iPhone may make us know that we do not necessarily bring our action into the virtual world as it stands. Although PINCH does not make something big or small in the physical world, we control two points of a diagonal line in order to draw a quadrangle in the virtual world like Adobe Illustrator. PINCH makes a movement of diagonal, therefore we may superimpose this hand movement and the movement for drawing a quadrangle in the digital world. As result, we naturally accept the relationship between PINCH and Zoom in and out. I consider that it is big problem to explain what makes us naturally accept the relationship between our gestures and the image action or feel something strange for it. We have to be careful and aware of what our unconscious gestures mean and correspond them to the image movements and actions in the virtual world.

However, it is our contention that gesture ─ certainly descriptive or 'iconic' gesture ─ necessarily involves indexical links to the material world, even though these links are rarely established or explicated in the communicative situation itself. Rather, in conversational contexts that are detached from the talked-about world, participants must fill in encyclopedic knowledge (ranging from universal bodily experiences to highly specific cultural practices) to see and recognize gesture. Phenomenally, there are only motions of the hands. What is perceived, however, is typically not motions but actions and, simultaneously, implied objects acted upon.

Curtis LeBaron, Jürgen Streeck, 'Gestures, knowledge, and the world' In David McNeill(Ed.), Language and Gesture, pp.118-138 (2000)

Finally, I think that a big change will happen because the digital which is no material and shape intervenes in our gesture communication based on the material and shape of our body. To show our will to the digital world. Conversely, the digital world shows its own will to us. I formerly called this correlation 'Display acts' which the image on the display changes corresponding to our actions. Now, I consider that 'display acts' is, connected with 'display' means the bird's action of the courtship, a courtship dance or action between the human and the digital furthermore, the human and the human through the digital. We are groping around for the courtship action to the digital world which is not able to see and touch. I want to feel the touch of the digital world.

Is multi-touch gesture poetic language or prose?

Is multi-touch gesture poetic language or prose? Maybe, it is wrong to ask this question, but I want to consider about it. When the computer came into our world, "metaphor" played an important role: the desktop metaphor. The desktop metaphor arranged the images on the display and the mouse in the order of our body sense. Now we used to use them, then the multi-touch gesture is coming into the world. Is the multi-touch gesture some metaphors for something. The metaphor connects two things which are normally separated. Moreover, it is baed on our body sense according to George Lakoff and Mark Johnson. If the mouse as a physical object gives us the feeling of "grabbing" for the images on the display in the desktop metaphor, what does the multi-touch gesture bring into the relationship between the human and the computer. I think that multi-touch gesture makes one-to-one relationship between us and the logical machine, which is like a prosaic sense or a sense of grammar/dictionary. However, my friend pointed out that the multi-touch gesture is like a poetic language. And I felt that his idea is also interesting so I have to re-consider about the multi-touch gesture.

iPhone introduced the PINCH:

With the iPhone multi-touch, zooming in and out of a picture never been easier. Steve Jobs introduces what he called a “pinch”. Bring your thumb and index fingers closer together or move them further apart to scale images.  Have a look at Steve Jobs presenting the feature, the audience is amazed by this.        

http://www.iphonefreak.com/2007/01/iphone_and_stev.html

 This thumb and index fingers action had made nothing to zoom the image or the physical object before the iPhone. Now, we think that our two fingers action have been able to zoom the image on the display from ancient time. This is the problem; Pinch is new action for us but we used to do it. Does iPhone just connect our thumb and index fingers action to zoom the image? Does the human have a sense for this finger action make zoom the image or the object? If there is only artificial connection our fingers action and zoom, this is one-to-one relationship like the dictionary. If we have natural sense for the link between fingers action and zoom, the Apple design and engineer team feel this sense and give it shape. Anyway, this it time that the digital device makes our action rule, therefore we must consider about these new action and its meaning in the long human history. I think that a new language based on our body, the image, and the digital device is arising now. I use "Language" as metaphor. We has changed the our own language in order to adjust our recognition for the environment. In digital age, I think that this change occurs at our body level very calmly.

I think that multi-touch gesture reduces our three-dimensional action to two-dimensional one. In order to reduce that, our body sense for the three-dimensional world have to be discarded. And, Pinch action for zooming the image shows this switchover. Pinch action need just two points of contacts in three-dimentional fingers action in order to zoom the image in the display. Even though we see and feel our own finger action which is three-dimentinal, iPhone need just two points of contact in the its surface. We live in three-dimantional world, so we always feel that our action for iPhone is the three-dimentinal one but the reduction happens there. However, we can not notice this reduction and feel "natural" for Pinch action. Pinch action is the artificial but we feel and make it "natural". The relationship is that "natural : artificial = poetic : prosaic"

"Weight" of Magic Mouse

I tried Apple's Magic Mouse. My first impression is thin and sticky. I consider that we have to grab the mouse in order to use it, therefore I pay attention to the thinness of Magic Mouse. According to Twitter, however, many people don't care it or get used to the thinness of it. Gizmode says that "Compared to ergonomic mice, the Magic Mouse is really low and aerodynamic, which means it doesn't contour to your hand and doesn't give the sensation that the mouse is a part of your hand". In my personal view, I not 'grab' the Magic Mouse but put my hand on it when I use it. Put my hand on it and move my fingers on its surface. So, I think that the surface of Magic Mouse it very important because my fingers can freely move on it. The glossy plastic is looks so smooth that I expect it is nice for my fingers. However, it is stickier for my fingers than MacBook's glass trackpad. This stickiness more bothers me than the thinness of Magic Mouse. As I already wrote, I consider that it is important for the multi-touch gesture to be smooth on the surface for freely finger movement. The smoothness is that our fingers which touch not information but an object don't feel some holds because information doesn't have weight and physical friction. We feel good due to let our fingers freely move on the smooth physical surface at the same time that the image on the display, information moves with our fingers like sticking each other. Although I mentioned this good feeling as 'Smooth and sticky,' we feel that there my be no weight. If our finger sweat, we feel some frictions between the fingers and the plastic surface. This frictions make something strange for us. Moreover, we already know the new standard for the smoothness of glass trackpad, so we feel that the plastic surface is sticker than the glass. This stickiness makes us feel "weight". This "weight" may be not physical but the tension of our muscle. It is like standing ready for controlling the image on the display, in other words, information. This is just my first impression for Magic Mouse, the first multi-touch gesture mouse. Therefore, I have to use it for a long time and gather many user's comments to it in order to consider about the meaning of Magic Mouse and the multi-touch gesture mouse.