Multitouch Support

Introduction

The computer mouse has been the predominant input device for graphical user interfaces for about 30 years. The interactive component of Cinderella depends on it, as it provides a one-to-one correspondence between the movements of a physical device and a point on the screen. On the other hand, the mouse restricts the gestural expression of a human being to a two-dimensional translation. We have become used to that restriction, but in real life we prefer to work with both hands and several fingers.

Technology advances, and one of the big steps forward are multitouch interfaces as found in modern smartphones or tablet computers. Even the touchpads of laptops are able to recognize several fingers simultaneously, and screens with touch recognition have become affordable and are supported by the latest computer operating systems.

For a geometry tool, multitouch is more than a nice-to-have feature. Mathematically spoken, the interface changes from "translation only" to "multi-transformation capable". For good reasons there is no mode to add a "free line" in Cinderella. Instead you have to use lines by two points or lines through a point. If you are using a mouse as input device, only those two tools define a meaningful interaction between the user and the construction. The situation changes drastically in a multitouch environment. Movements of two fingers are already powerful enough to encode a similarity transformation. In such a scenario it is physically possible to use two fingers to get the full interaction for a line: You can rotate and translate it at the same time.

From a programmers' point of view, multitouch introduces some complexity. With the mouse there is a defined pointer position at any time, and the button interaction is always a press-drag-release sequence. This is no longer true for multitouch interaction; obviously there might be any number of current positions (even none, if you don't touch), and the touch-move-release sequences may overlap. As a user, you should not have to care, but if you are going to work with CindyScript, this will be of importance. See [KD10] for a discussion in more detail.

Using Multitouch With Cinderella

Currently, Cinderella supports multitouch events through the community standard TUIO protocol [KBBC05], see also http://www.tuio.org external link

. This cross-platform standard has the advantage that there are corresponding drivers for almost every multitouch device. For smartphones and tablets you can find apps on the TUIO home page that will send the multitouch events to your computer.

TUIO settings in the Inspector

You can enable TUIO support in the mode and language settings of the Inspector. Check the TUIO support checkbox and Cinderella will listen on the standard port 3333 for TUIO events in Move mode. While TUIO support is on, regular mouse events will be ignored in Move mode to avoid double input if used with standard TUIO drivers on Windows or Mac OS X. Other modes of Cinderella still require mouse operation and do not support TUIO events. If you check Creation gestures, you can create points and lines by touching the plane with one or two fingers. As this can lead to a lot of unwanted points and lines this option is unchecked by default and should be considered experimental.

The Only frontmost window option should be checked or unchecked depending on the device you use for multitouch gestures. If the device displays the screen contents it usually should be unchecked, as the touches on the screen are mapped to the individual windows of Cinderella automatically. If the device does not have a screen or does not show the Cinderella windows, as it is the case with touchpads on laptop computers or TUIO apps on mobile phones, it might be better to check this option. In that case all events on the touch surface are mapped to the visible view area of the frontmost window of Cinderella.

Programming Multitouch With CindyScript

Multitouch input for geometry software is still a research area, and besides the obvious behavior in move mode, there are no user interaction standards yet. While we will work on the multitouch integration in other modes for future updates of Cinderella, we also decided to make the multitouch events accessible to CindyScript.

If you enable TUIO support in the Inspector, the mouse event slots Mouse Down, Mouse Up and Mouse Drag will be triggered by TUIO events instead of regular mouse events. The mouse function will report the location of the touch event. The mouse() location is uniquely defined as every touch event reports about one of the multiple touches only. This implies that you cannot rely on the fact that successive mouse events belong to the same gesture. This is a programmer's nightmare: How can you keep track of the different touches and assign them to the right actions? For your convenience, CindyScript includes the necessary tool, as we describe below.

Declaring touch-local variables: `mtlocal(<var1>,<var2>,...)`

Description: All variables declared to be mtlocal are local variables for a touch sequence. This function is only useful in the Mouse Down slot. If a variable is defined as touch local, then for all touch sequences a unique instance of that variable is created. If you use the same variable in the Mouse Drag slot or Mouse Up slot you will work with the instance of that variable that is assigned to the touch-move-release sequence.

Example: We will illustrate the use of the mtlocal function with a small example. By placing the following code in the Draw, Mouse Down, Mouse Up and Mouse Drag slots you can draw polygons withs the mouse. While the mouse is pressed, the new polygon will be drawn in red, when the mouse is released, the new polygon will be added to a list of polygons that is drawn in blue.

// In Draw:
forall(polygons,drawpoly(#));

// In Mouse Down:
polygon=[mouse()];

// In Mouse Drag:
polygon=polygon++[mouse()];
drawpoly(polygon,color->red(1));

// In Mouse Up:
polygons=polygons++[polygon];

This code relies on the fact that a polygon will be created first in Mouse Down, then points will be added in Mouse Drag, and finally the polygon is complete and will be added to the list of polygons in Mouse Up.

When the same code runs in a multi touch environment, the (global) variable polygon will be used for all touches. Also, if any touch is released, the new polygon is added to the list of polygons, but still other drag events will add points to it. The code is not multitouch-compatible and will badly fail.

By adding a single call to mtlocal we can fix the code:

// In Draw:
forall(polygons,drawpoly(#));

// In Mouse Down:
mtlocal(polygon); // make sure every touch gets its own polygon
polygon=[mouse()];

// In Mouse Drag:
polygon=polygon++[mouse()];
drawpoly(polygon,color->red(1));

// In Mouse Up:
polygons=polygons++[polygon];

Now a new polygon will be created for each touch, and the drag and up events will use the correct polygon variable that is associated to the gesture.

Drawing three polygons at once using multitouch

If you try this code example, you will see that the repainting of the red polygons does not work as expected due to the automatic clearing of the screen at every event. We can fix that by explicitly keeping a list of new polygons and drawing them in the draw event only. You find the final code below.

// In Draw:

forall(polygons,drawpoly(#));
forall(newpolygons,drawpoly(#,color->red(1)));

// In Mouse Down:
mtlocal(polygon);
polygon=[mouse()];
newpolygons=newpolygons++[polygon];

// In Mouse Up:
newpolygons=newpolygons--[polygon];
polygons=polygons++[polygon];

// In Mouse Drag:
newpolygons=newpolygons--[polygon]; 
polygon=polygon++[mouse()];
newpolygons=newpolygons++[polygon];

Alternatively, you could use layers and the autoclear flag to achieve the same effect without storing all the new polygons.

Contributors to this page: Kortenkamp , Richter and Akira Iritani .
Page last modified on Friday 02 of September, 2011 [09:01:48 UTC] by Kortenkamp.

The content on this page is licensed under the terms of the License.

similar

Home

[-] Wiki [+]

Multitouch Support