Sikuli: Using GUI Screenshots for Search and Automation

Summary:

Sikuli is an attempt to utilize GUI snapshots instead of text to enhance computer human interaction. Sikuli is composed of two features: Sikuli Search and Sikuli Script.

Sikuli Search allows the user to take a clip from a screen shot and plug the actual image into a search engine to search forums, blogs, help manuals, the product websites in addition to the built in help database to find a information about the GUI element they selected.

This concept was tested by making a graphical database of screenshots compiled from online tutorials, official documentations, and computer books. Then, a test group of 12 people used the program and reported an average query time of less than half what it took compared to the keyword queries. However, relevancy decreased, probably due to the size of the database which was being queried.


Sikuli Script is geared more towards the developer clients. It allows the developers to create automated scripts using clips of screenshots rather than having to program using text commands.

Sikuli script finds GUI patterns on a screen using an algorithm adapted from that which is used to distinguish cars and pedestrians in a street scene.  In the example above the script finds all pdfs, then opens the hard drive and drops all the pdf in the documents folder of the harddrive.  The script below minimizes all windows on the screen.  The average search time for a 100 x 100 area on a 1600 x 1200 screen on a 3.2 GHz Windows PC was less than 200 ms.

Discussion:  This concept seems to follow the trend of usability at the expense of performance.  While making it extremely easy to create scripts through use of GUI screenshots, it seems like the script would be very processor intensive.  An interesting extension of the concept would be an integration with Google search to be able to search by image instead of just for images.  I think that would solve Sikuli's problem of such a limited database to query from.