Seven benefits of automated GUI testing

Introduction: I first wrote this article about automated GUI testing many years ago, but I find that it still holds today.

I just wrote most of the following note on the Apple Mac Java-dev mailing list, and I'd like to share it here as well, because I think it captures my thoughts on the benefits of automated GUI testing and GUI testing software.

I ran automated GUI tests part-time (4-6 hours per week) on a project with 8-12 developers, and saw some good benefits. True, in the 80/20 rule, 80% of the problems were due to UI changes and communication, like “We forgot to tell you we split the Name field into First Name and Last Name,” but with a good automated GUI testing tool, one test may fail, but the rest of the automated GUI test suite keeps running (see Fowler’s continuous integration). Furthermore, with a good GUI testing tool, something like this is also a minor change to get the test running again.

Benefits of automated GUI testing

The biggest benefits I’ve seen from automated GUI testing are:

1. Finding regression errors.
2. Letting human GUI testers (“domain experts”) get back to their normal jobs instead of running tests a robot should be running.
3. Finding errors when the test suite ran at night, and telling testers in the morning, “You can avoid ABC, because we have some known problems there.”
4. Covering for application areas that didn’t have unit tests. This can be a temporary situation, or if a system wasn’t written to be tested easily, a more permanent thing.
5. Detecting differences in behavior between platforms, i.e., Windows/Mac/Java versions, different web browsers, different operating systems.
6. Pounding on the system (more or less randomly), and finding memory/resource leaks.
7. Being able to use the same tool and scripts for performance tests that you use for automated GUI testing.

I don’t recommend automated GUI testing on every project, but if you’re paying people to test your application’s GUI, it may have enough merits. In the end, it’s all about quality, and ROI. I know that my automated GUI tests found problems that humans didn’t, simply because the tests were boring or because we didn’t have enough testers.

Keys to automated GUI testing and continuous integration

To me, the keys to successful automated GUI testing and continuous integration include:

1. Being smart about how you a) move through the app and enter data, and b) detect state changes and failure conditions. Stated bluntly, a worst practice is to rely on pixel coordinates too much (though admittedly, sometimes that’s all you can do).
2. Having a tool that lets you change your tests fast, so you can easily keep up with the UI changes. (Worst practice: Using a recording/playback tool that requires a new recording for minor UI changes.)
3. Obviously, you need a tool that supports continuous integration. Your tests should be able to run unattended. It’s a great feeling to come into work in the morning and seeing a report (or email) saying all the tests ran successfully.

Beware automated GUI testing software sales pitches and recorders

(Note: This section was written ten years ago, and my software is out of date now.)

Because I learned these benefits and best practices in the battlefield, I developed my own Agile/Automated GUI Testing software with all these ideas in mind. It’s an open source tool, there’s no salesperson here, and frankly I’m creating it for myself, not for you. Your using it won’t change what I do on a daily basis (unless of course you come up with some great new ideas). I happen to think it’s a heck of a good tool, but that’s my opinion.

In fact, I highly recommend you go to this Wikipedia page and find other tools, but please, keep everything I’ve written here in mind. In particular, if someone is trying to sell you down the road of a GUI recorder/playback system, and that system relies on pixel coordinates, your tests are going to break easily and break a lot, and you’ll have to re-record your tests constantly — and that’s not agile at all.