Project/Multitouch Interaction/User Interface
From Drexel Smart House
2008 Design Projects
Summary: To create a practical user friendly interface for a MultiTouch that will be used in the Drexel Smarthouse or any commercial residential dwelling.
Team Members:
Academic Advisor: Cody Ray cody@drexelsmarthouse.com
Problem Statement
Design user-friendly interface for a MultiTouch system for the Drexel Smarthouse.
The next generation of computers lies in the MultiTouch screen. However, the interfaces that are on computers now do not take advantage of what the MultiTouch system has to offer and therefore will not be as efficient and easy as it could be. With the new technology the MultiTouch screen has to offer, a brand new interface needs to be developed that will be able to browse through the programs of the computer quickly and effectively, while still being easy enough to be used by any person accessing the computer. During a survey we took, about 80% of college students interviewed believe that the common windows interface is inefficient and should be improved for a touch screen type computer.
Possibilities
Because of the capabilities of the MultiTouch system, there are almost an ifinite number of possible interfaces that could be used. However, three potential solutions we found include having all the programs on a wheel and rotating them to choose the program to run, a pen based navigation system where the pen is used to write the name of the program and that program opens, or using hand gestures to navigate through the system.[1]
The Finished Interface
In the end, we are hoping to have an interface that is simple enough able to be used by anyone with a touch screen computer, or at least most users of the system. It will need to be faster to navigate than the modern windows interface where all the programs are listed in the start menu and navigation is through a mouse. Any constraints for our solution are based with the MultiTouch screen and what it will be able to detect and decipher, yet still simple and relatively quicker than the today’s windows interface. This interface will be integrated onto the Drexel Smart House MultiTouch screen that is being built right now, but hopefully it will be able to be used on any MultiTouch screen in homes around the world.
References
[1] Anonymous "Finding touch screens that work," Machine Design, vol. 67, pp. 118, Oct 26. 1995.
[2] K. Greene, "Wall-Size Touch Screens," Technology Review, vol. 110, pp. 23, Apr. 2007.
State Of the Art Review
Right now most home computer interfaces consist of a point and click, mouse to monitor interface while commercial computers such as cash registers are instating touch screen monitors. It is inconvenient for a cashier to have to use a mouse to point and click through necessary information to complete a customer transaction so it is much easier to use touch screens. As technologies arise it becomes a bigger assumption that soon home systems will also used touch screens as well and while these technologies advance quite steadily it will not be too far off to where a user-friendly multi-touch will be the norm in a residence.
I. MultiTouch Systems (MTS) General
The MTS technology is based on its unique 2D sensor array composed of proximity sensors. These sensors capture images of the fingers and hands on or near the surface. Touching one finger at a time is recognized as a keystroke command. Simultaneously placing two fingers on the surface initiates pointing. Simultaneous thumb and fingertip motions become gesture commands. Effortless computing is the characteristic that allows the user to perform typing, mouse movements, and gesture commands with the smallest amount of pressure to the surface. MTS’s ability to be a seamless interface, which the user can easily switch among typing, pointing, scrolling, and gesture commands, eliminating redundant motions between multiple input devices (e.g. mouse and keyboard). There have already been several devices that have been using the touch screen technology for the past several years. All of them have used such technology as capacitive or resistive overlays, piezoelectric, light beam interruption, and surface acoustic waves to detect where the user is touching the screen, but were not able to detect several touches at once [1]. The interface do not need a mouse and can make a keyboard on screen and therefore do not need as many input devices as a regular desktop computer to navigate. They have been especially prominent in hand-held devices and cell phones. However, because the screens cannot detect multiple touches, like the MultiTouch screen is able to do, the interface of the MultiTouch screen could use some of the ideas from the single touch interfaces, but should not be prohibited to that. There have also been some MultiTouch-like interfaces that are in some appliances now. For example, the iPhone and iTouch both have a few MultiTouch features, such as being able to manipulate the size of pictures by grabbing the corners of the picture and moving them [3]. Also, the Microsoft Surface, shown to the right, is a completely MultiTouch screen that has an interface that works, but was made to be used at companies like restaurants because of the expense of the MultiTouch screen (around $5000-$10,000 right now) [4]. To be used in a home, there will need to be some developments in order for the interface to be efficient for an everyday use, but many of the Microsoft Surface’s functions could be useful in the interface of the home based interface. Recently there have been many advances in multi-touch technology allowing for the realistic vision of reasonably priced consumer applications of the technology. This technology has existed for some time but historically has required the kind of computing power not available until now [8]. We’re now seeing the first inclusions of multi-touch technology in widely available consumer electronics on a small scale i.e. the apple iPhone and iPod Touch. The technology for the multi-touch we are designing will have to work on a larger scale, and be more adaptable to all information, rather than work for specific purposes. In the retail area, Microsoft is set to release its Surface interface for restaurants and shops and the technology used is very interesting and intuitive for its uses, namely opening up new avenues for commerce to take place. The interface allows for very quick connection of other devices (such as phones and cameras), and intuitive interaction with them. The interface is at its heart based on Windows Vista, heavily modified for the touch commands. It works very well for its retail oriented purpose but may not be practical for home consumers because of the lack of functionality on a personal level and price. A Surface machine may cost upwards of $10,000.00 to buy. One interesting idea that can be worked with is the idea of a highly graphic interface. The multi-touch technology makes it possible to have an interface where all information can be displayed graphically instead of in hard typed information. A user could manipulate a “cube” of information, resizing it, moving it, and arranging it. It could be turned to display other information, twisted to combine it, and crumpled to delete it. Using ideas like that, a highly interactive, stimulating, and graphic interface can be adapted. Another approach that could be taken is the incorporation of RFID tags and the multi-touch technology. The tags could be implanted into items, cards, and even people for instant identification and interaction.
II. Where MultiTouch is in Use Today
II.i Camera-based systems:
The Visual Touchpad uses a stereo camera to track fingertips. The disparity between the image pairs determines the height of the fingers above the touchpad; a threshold is set to coarsely detect contact with the surface.
II.ii Thinsight:
ThinSight, as shown in figure 2 and 3, is based around a 2D grid of retro-reflective optosensors, which are placed behind an LCD panel. Each optosensor emits light that passes right through the entire panel. Any reflective object in front of the display (such as a fingertip) will reflect a fraction of the light back, and this can be detected.
Figure 2: ThinSight detecting a finger Figure 3: The ThinSight Display
- A human fingertip typically reflects around 20% of incident IR light and is therefore a quite passable ‘reflective object’. - IR light is not visible to the user, and therefore doesn’t detract from the image being displayed on the panel. - Using these established techniques, fingertips are sensed to within a few mm, currently at 11 frames per second. Both hover and touch can be detected, and could be disambiguated by defining appropriate thresholds. A user can therefore apply zero force to interact with the display.
Figure 4: Examples of finger-painting with the MultiTouch Screen
III. Applications
III.i Painting
Finger painting, such as the examples in figure 4, may be the most obvious application of the system. Finger pressure is used to control the opacity. The glossy surface of the panel causes a high static friction, which makes the finger momentarily stick. Selection. A selection mechanism that mimics grasping can be implemented similarly to painting: The small disk formed by one finger tip or the extended ellipse shape formed by several fingers is used to select icons, adding to the existing selection.
III.ii Disk Scratching
The pressure sensitivity can be used to gradually slow down, accelerate or arbitrarily rotate a spinning disk, identical to what DJs do when “scratching” a disk. The more pressure is applied, the more the disk is subjected to the motion of the finger, not its regular spinning motion.
III.iii Rotary Knobs
In real life, knobs are grasped and twisted with the thumb and the index finger. Lacking two-finger control, typical on-screen knobs do not offer an obvious interaction mode. With the rotation measurement described before, one can use the thumb and the index finger like in real life.
III.iv Pan, Zoom, and Rotation
A map can be panned with one finger and zoomed in or out and rotated with a two-finger gesture. Sensing the loss of one or both contacts, the system can still freeze the transformation in a reliable state. The pressure sensitivity leads to the following new interaction mode: By touching at one position and then adding pressure at a second position, one can slowly zoom in.
V. Touch Screen Technologies
There are different types of ways to make touch screens. The interface technology must be efficient and reliable in order to have multiple access points being invoked simultaneously. Different style of touch acceptor could mean a different user interface.
V.i Resistive Membrane
Resistive membrane is a glass panel with a resistive coating and a transparent foil underneath will act like an electrical conductor. When the surface is touched the conducting foil will touch the membrane and an electrical voltage will be out put and the exact location and movements can be traced using a calculation of the position and force of the voltage. The accuracy of this system could be potentially be dead on and the simple electronics allow for inexpensive construction. This system is a contact and while it can follow and locate position the action is not initiated until the finger is removed.
V.ii Capacitive Panel
A capacitive coated glass plate with 4 electrodes at each corner. Those electrodes connected to oscillators and when contact with the screen occurs the disturbance in the phase and frequency shifts are calculated from the amount of change. A lot of electricity is required to power the system and a nonconductive stylus would not be able to be used with this system. Simply works the same as the resistive membrane but different composition.
V.iii L.E.D. Array
This screen picks up position of touch by using two sets of L.E.D.s that usually use infrared and two sets of photo sensitive transistors in order to pick up the arrays. The beams of light face each other that create a matrix like lattice. When the matrix is broke the point of access can be found. Accuracy can be infinitely accurate depending on the amount of transmitter/receiver pairs used. This system is considered a contact-less touch screen. Considered this because as soon as a beam is broken the receptor picks up the location, long before the screen is touched. This will also allow tracing but will have direct feedback as well during the selection and contact [12].
V.iv Piezo Electric Touch Panel
This style uses a force distributions found at point of contact. This piezo ceramic sensor will be placed at each corner and the sensor will be placed underneath the screen between the panels and what holds the screen together. When touched the screen picks up the force and changes it into an electrical signal which is then digitized and transmitted to a microprocessor. This system requires careful calibration in order to use analog and digital filtering systems to determine which is a touch and what is a vibration. After calibration the touch system requires very little cost because no mechanical alignment would be required after. A piezo touch panel could be dirty, scratched, drawn on, etc. and still work properly because the surface doesn’t matter. There are no moving parts so reliability is extremely high as well as the technology allows for any sized application.
VI. Patents
The technology empowering the new user interface being developed is Multi-touch technology. MultiTouch technology is a touch screen that allows for multiple points of contact. The most known multi-touch screen used today is the iPhone. The iPhone is a basic multi-touch and has already shown what the technology can do if software is developed properly. There are already several different patents that apply to the technology. In patent number 6,570,557, different gestures using a multi-touch screen are explored. The patent deals with the new application of modifier keys such as the Shift or Alt key using multi-touch. The patent proposes that different combinations of four fingers can replace these keys. For example, the greater used keys such as shift would be relegated a more intuitive position while a less used key would be delegated a position that uses an open hand in order to operate. This way the software gets more commands intuitively out of a four fingers. Another patent, patent number 6,271,835 also deals with data input through the multi-touch screen. The patent deals with alphanumerical entry. When a key is pressed corresponding keys are shown thus increasing the efficiency of entering the data. The last patent that deals with using multi-touch gestures is 7,030,861 and this patent deals with using gestures and entering data using only a single hand. For example twisting the hand would correspond to a particular type of entry on the screen. Through this method multiple users could correspond with the screen each using one hand to interact with the screen.
References
[1] S. Sherr and R. C. Durbeck, “Computer Engineering/Input and Output,” Electrical Engineering Handbook, vol. 89, pp. 14, 2000.
[2] C. K. Yang, “Input / Output Devices”, The Engineering Handbook, second edition, vol. 146, pp. 6.
[3] W. S. Mossberg “Multitouch Interface Is Starting to Spread Among New Devices,” Wall Street Journal, Jan. 31, 2008.
[4] T. Bishop “Microsoft Surface brings computing to the table,” Seattlepi, May 30, 2007.
[5] “The Microsoft Surface screen at a restruant,” May 7, 2007. Online image. “Microsoft Surface could be a great Rich Internet Application platform,” March 01, 2008 <http://blogs.zdnet.com/Stewart/?m=200705 >.
[6] Derene, Glenn, " Buzzword: Multitouch," Popular Mechanics, Feb. 20, 2007.
[7] Han, J. Y., " Low-Cost Multitouch Sensing through Frustrated Total Internal Reflection," NYU, 2005.
[8] Mossberg,Walter S., " Multitouch Interface Is Starting to Spread Among New Devices," Wall Street Journal, pp. B1, Jan. 31, 2008.
[9] Frett, E. J. and Barner, K. E. 2005. Accuracy and frequency analysis of multitouch interfaces for individuals with Parkinsonian and essential hand tremor. In Proceedings of the 7th international ACM SIGACCESS Conference on Computers and Accessibility (Baltimore, MD, USA, October 09 - 12, 2005). Assets '05. ACM, New York, NY, 60-67. DOI= http://doi.acm.org/10.1145/1090785.1090799
[10] Loviscach, J. 2007. Two-finger input with a standard touch screen. In Proceedings of the 20th Annual ACM Symposium on User interface Software and Technology (Newport, Rhode Island, USA, October 07 - 10, 2007). UIST '07. ACM, New York, NY, 169-172. DOI= http://doi.acm.org/10.1145/1294211.1294239
[11] Hodges, S., Izadi, S., Butler, A., Rrustemi, A., and Buxton, B. 2007. ThinSight: versatile multi-touch sensing for thin form-factor displays. In Proceedings of the 20th Annual ACM Symposium on User interface Software and Technology (Newport, Rhode Island, USA, October 07 - 10, 2007). UIST '07. ACM, New York, NY, 259-268. DOI= http://doi.acm.org/10.1145/1294211.1294258
[12] Gungl, K.P. (IBM Lab. E+F, Boeblingen, West Germany) Source: Proceedings. VLSI and Computer Peripherals. VLSI and Microelectronic Applications in Intelligent Peripherals and their Interconnection Networks (Cat. No.89CH2704-5), 1989, p 2/98-100 Database: Inspec
[13] Todd, M.S. (John Fluke Manuf. Co. Inc., Everett, WA, USA) Source: Northcon/84. Mini/Micro Northwest-84. Conference Record, 1984, p 9/1/1-11
[14] Penna, D.E. Source: Electronic Technology, v 19, n 5, May 1985, p 84-7
2008 Design Projects
