Can your robot play Angry Birds? On an iPhone? Mine can. I call it “BitbeamBot”. It’s a creation that’s been about ten years in the making. As originally conceived, it was just an art project for fun. As I built it and gave it more thought, I came up with a much more practical application: mobile web testing. Let me tell you a bit about how I got here.
A Pin Art display
About ten years ago, on my desk sat a Pin Art display. Pin Art, or Pinscreen, is a popular desk toy containing an array of pins that you can push your face or hands into and leave an image.
One day when playing with the Pinscreen, I had a thought: what if you could motorize and computer control each individual pin? If you could do that, you could create a display that shows 3D computer graphics on a real honest-to-goodness 3D display. At the time, I was playing around with 3D graphics simulations on my computer and was frustrated by the irony of viewing 3D objects on my very flat 2D computer monitor.
As I saw it, my project involved three big components: software, electronics, and mechanics. Since I had a background in software programming, I tackled the software side first. About 8 years ago, I created a Python and OpenGL-based software simulation. I dubbed it “PinThing”. To make the demo more accessible to more people, last year, I ported the Python version to JavaScript and HTML5 and now host it at pinthing.com. (Source code: https://github.com/pinthing/pinthing.com)
PinThing.com
With the software side moving along well, it was time to turn my sights to mechanics. After a few failed attempts to make a mechanical prototype with wood and metal, I turned to my favorite prototyping material, Lego Technic. Lego Technic is like regular Lego bricks, but with holes for attaching wheels, or gears, or pulleys. You can create almost any structure or mechanical device with Lego Technic beams, pegs, and axles. After a few tries, I had a working Lego prototype for one PinThing pin. While I was excited to have a workable design, I was sad when I realized how expensive it would be to buy enough Lego to scale this up to as many pins as needed — dozens to hundreds of pins for a really big display.
A few days later, I realized the solution to the cost issue – make my own Lego! Of course, that was easier said than done. Fortunately, I had stumbled upon the grid beam project at Maker Faire 2010 in San Mateo, California. Grid beam is a modular design for building things, like tables, chairs, shelves, and even vehicles. Grid beam’s system is composed of square beams of wood or metal, with holes evenly spaced. (More information about grid beam is available at http://www.gridbeamnation.com ).
I ported grid beam dimensions down to Lego proportions. I bought pieces of basswood at the local art supply store, and took a class on laser cutting at Tech Shop in San Francisco. I laser cut holes into the wood to create what I now call “bitbeams”. (More info at http://bitbeam.org)
The PinThing prototype
It then was easy to port over my Lego PinThing design to Bitbeam.
Here’s my first prototype pin made out of Bitbeam. If you look closely, there are a few Lego Technic pieces in there, too.
In my day job as CTO at Sauce Labs, my job is to lead the product direction of our cloud testing service. We have a cloud-based array of desktop browsers like Firefox, IE, and Chrome available for test automators to drive using the Selenium API as an integral part of their continuous integration processes. For a while, we’ve been planning how and when we’d tackle the problem of mobile testing.
After playing with the PinThing/Bitbeam pin prototype, I noticed something. The mechanics required to move a pin up and down are the same to touch the screen of a mobile phone. If it can touch a screen, it can be used for mobile testing. And if it can be used for testing, this just went from a silly weekend art project to something that might be useful back at work.
But how could a bunch of wooden Lego-like beams be useful to software testing?
For the confidence that your mobile app truly works, you need an end-to-end test on the actual device. This means the full combination of device manufacturer, operating system, data network, and application. And since the devices were meant to be handled with the human hand, you need something like a real hand to do real end-to-end testing. At some point, after lots of repetitive manual testing, the inevitable questions is asked “Can we / should we automate the testing of the old features, so I can focus the manual testing effort on the new features?”
BitbeamBot the mobile testing robot
That’s where the BitbeamBot comes in. To turn the PinThing into a robotic button clicker I needed to do two more things. First, I added motors to move the pin thing in any direction left, right, forward, or backwards over the screen surface. Second, to touch the screen just a like a real finger, I duct taped conductive foam to the end of the pin. (MacGyver would be proud.)
At the moment, BitbeamBot is only a prototype and far from production-ready, but it can play games with simple mechanics, like Angry Birds. However, it’s not very smart; it can’t yet “see” where objects are on the screen. From my computer, I send electrical signals to two motors to move the pin over any point on an iPhone screen. I then use a third motor to move the pin down to the screen surface and click or drag objects. This open loop, flying-blind approach to automation is how automated software testing was done years ago. It’s the worst way to automate. Without any sense of what’s actually visible on screen, the script will fail when there’s a discrepancy between what’s actually on the screen and what you expected to be on screen at the time you wrote the automation script.
A better approach to testing with BitbeamBot will involve closing the feedback loop and have software determine where to click based on what is actually on screen. There are two styles I’ll experiment with: black box and grey box. Black box testing is easier to get started with, but grey box testing is more stable long term.
Black box testing requires no internal knowledge of the application. It treats the application like a metaphorical “black box”. If something is not visible via the user interface, it doesn’t exist. To get this approach to work with BitbeamBot, I’ll probably place a camera over the phone, send the image to my computer, and use an image-based testing tool like Sikuli. Sikuli works by taking a screenshot and then using the OpenCV image recognition library to find elements like text or buttons in the screenshot.
The other style is grey box testing. It’s a more precise approach, but it requires access to and internal knowledge of the application. I’ll implement this approach by extending the Selenium library and tethering the phone to my computer via a USB cable. With the USB debugging interface turned on, I can ask the application precisely which elements are on screen, and where they are before moving the BitbeamBot pin to the right location.
I am documenting my building progress at bitbeam.org. The hardware, software, and mechanical designs are open source and available at https://github.com/hugs/bitbeam.
Author Profile – Jason Huggins
Jason Huggins co-founded Sauce Labs and currently leads product direction. Prior to Sauce Labs, Jason was a Test Engineer at Google where he supported the grid-scale Selenium Farm for testing Google applications such as Gmail and Google Docs. Jason’s experience also includes time at ThoughtWorks in Chicago as a software developer.
While at ThoughtWorks, Jason created the Selenium testing framework out of the need to cross-browser test a new in-house time and expense system.
When not programming in Python or JavaScript, Jason enjoys hacking on Arduino-based open source hardware electronics projects. Although he was born in New York City, grew up in L.A., and travels in a gravitational orbit around San Francisco, Chicago is his kind of town.
No comments yet.