Virtual Shelves: Interactions with Orientation Aware Devices

Mobile devices can require several keystokes to utilize an application based on the context of the application. This requires the user to visually engage the display even when it would be beneficial to not have your eyes preoccupied with the device such as using a phone while driving.  Virtual Shelves is an attempt at alleviating this problem by using the user's spatial awareness and gestures as input to a device, thereby not requiring visual interaction with a device.  A hemisphere of options directly infront of the user can be activated by moving your arm into specified quadrants.

Two experiments were done to test the efficacy of Virtual Shelves:

1. Measuring Kinesthetic Accuracy - studying how adept people inherently are at landing in a specific quadrant given the approximate theta and phi coordinates
2. Efficacy of Virtual Shelves - The hemisphere infront of the user was divided into regions whose size was based on the general accuracy obtained from experiment 1(+/-15 degrees).  Each region was assigned an arbitrary application and when the user hits the center button on their cellphone when in a certain region it activates that application without the user having to navigate menus with the D-pad.  The efficacy results were compared with navigation using the D-pad.

The results showed that past the initial training use, people were able to access applications faster using Virtual Shelves.

Discussion: 

The author mentions this setback, but it does require your cellphone to utilize motion capture in order for Virtual Shelves to function.  Also, motions needed to utilize Virtual Shelves are gross motor skills as opposed to the fine motor skills needed to navigate menus using a D-pad.  This could be socially awkward for user, especially in a crowded environment.  People generally do not like drawing attention to themselves in order to use everyday devices.  Virtual shelves, however, makes one look like they're guiding in an airplane when they just want to launch their mobile device's music player.  I think this could be a serious short coming of Virtual Shelves but could be minimized if the user didn't have to outstretch their arm to select a region.

Disappearing Mobile Devices

Summary:
Mobile devices have seen a trend of growing smaller and smaller as technology permits.  This can be seen in the progression from notebook computers and PDA's to interactive watches and rings.  However, the limiting factor seems not to be the size of the circuitry, but rather the size of the physical user interface.  The paper explored the possibilities and limitations of "disappearing mobile devices" which do not have a large physical user interface, but recognize gestures across it's surface as input.

When devices are miniaturized, several physical aspects become impractical.  For example, capacitive touch pads become touch sensors when miniaturized.  Also, pressure sensing becomes skewed at small scales especially when the sensor is on human skin because pressure sensors generally require placement on hard surfaces.  Motion sensing using two techniques were implemented for the user study: EdgeWrite, where the shapes of letters are modified to minimize multidirectional(circles, arcs, etc) strokes, and Graffiti, where letters are still done in single strokes but are more natural and therefore rely on relative positions.

EdgeWrite

Graffiti

The user study found that EdgeWrite results on a disappearing mobile device was competitive with larger devices, but the error rate for Graffiti nearly doubled and may not be practical.  Based on the findings of the study, they found that miniaturization could still yield meaningful interaction even on such a small scale.

Discussion:

I think the idea of mobile devices becoming smaller and smaller is a very popular idea, but I don't really believe it's the future of electronics.  Especially within the cell phone market, we only temporarily saw a trend toward smaller phones. Instead the idea of smart phones was embraced more readily.  Rather than phones continuing the trend of becoming smaller, we've seen them grow more powerful and subsequently larger.  However, computers seem to be following the trend of growing smaller.  With the advent of the netbook, portability seems to be increasingly more attractive. I do not however foresee micro-laptops the size of a dime being the future.  Rather, I see a collision between phones and computers being the future with some device bridging the gap between smartphones and netbooks.

Enabling Always-Available Input with Muscle-Computer Interfaces

Summary:

Our fingers allow us to be one of the most dexterous creatures on the planet, allowing us to interact with our environment in very intricate ways.  However, one limitation that we've come to experience is that when interacting with computer devices we must always use a physical control as a medium such as a keyboard, mouse, joystick, etc.  In this paper, the concept of utilizing an EMG muscle-sensing band around the forearm to determine finger movements is explored for the purpose of potentially interacting with devices without the need to physically touch the device.  One example is using an mp3 player while running. It does not matter where the mp3 player is placed on the body, the player is incredibly inconvenient to manipulate while moving often causing the user to stop running to change a song or increase the volume.  Therefore, it was proposed that the user, wearing the muscle-sensing band, could make a series of gestures that correspond to certain functions of the mp3 player.  This would allow the user to keep their hands free or at least free to do something else.

The user study was carried out in three parts, each testing to see whether or not certain gestures could be recognized with differing grips:

1. small or no object in hand
2. tool in hand
3. heavy load in hand.

The results yielded a 79% accuracy while not holding anything, a 85% accuracy while holding a travel mug, and an 88% accuracy while carrying a weighted bag.

Discussion:

I feel like this concept could easily be extrapolated to other areas as well. If the accuracy of the EMG devices could be enhanced then sign language could be translated to text incredibly easy or prosthetic limbs could be more easily removed if the muscle sensors were external rather than embedded.  I understand that the user study was specified for a very particular set of gestures and grips, but I definitely agree with the author's statement that thorough and robust testing could yield much more significant results.
So, I may have chosen to read this paper because it's incredibly close to my senior design project which I thought was convenient.

Sikuli: Using GUI Screenshots for Search and Automation

Summary:

Sikuli is an attempt to utilize GUI snapshots instead of text to enhance computer human interaction. Sikuli is composed of two features: Sikuli Search and Sikuli Script.

Sikuli Search allows the user to take a clip from a screen shot and plug the actual image into a search engine to search forums, blogs, help manuals, the product websites in addition to the built in help database to find a information about the GUI element they selected.

This concept was tested by making a graphical database of screenshots compiled from online tutorials, official documentations, and computer books. Then, a test group of 12 people used the program and reported an average query time of less than half what it took compared to the keyword queries. However, relevancy decreased, probably due to the size of the database which was being queried.


Sikuli Script is geared more towards the developer clients. It allows the developers to create automated scripts using clips of screenshots rather than having to program using text commands.

Sikuli script finds GUI patterns on a screen using an algorithm adapted from that which is used to distinguish cars and pedestrians in a street scene.  In the example above the script finds all pdfs, then opens the hard drive and drops all the pdf in the documents folder of the harddrive.  The script below minimizes all windows on the screen.  The average search time for a 100 x 100 area on a 1600 x 1200 screen on a 3.2 GHz Windows PC was less than 200 ms.

Discussion:  This concept seems to follow the trend of usability at the expense of performance.  While making it extremely easy to create scripts through use of GUI screenshots, it seems like the script would be very processor intensive.  An interesting extension of the concept would be an integration with Google search to be able to search by image instead of just for images.  I think that would solve Sikuli's problem of such a limited database to query from.