Automated GUI Testing Software: I just wrote most of the following note on the Apple Mac Java-dev mailing list, and I'd like to share it here as well, because I think it really encapsulates my thoughts on the benefits of automated GUI testing and GUI testing software.
I ran automated GUI tests part-time (4-6 hours per week) on a project with about eight developers, and saw some good benefits. True, in the 80/20 rule, 80% of the problems were due to UI changes, like "We forgot to tell you we split the Name field into First Name and Last Name", but with a good automated GUI testing tool, one test may fail, but the rest of the automated GUI test suite keeps running (Fowler's continuous integration). Furthermore, with a good GUI testing tool, something like this is also a minor change to get the test running again.
Automated GUI testing - benefits
The biggest benefits I've seen from automated GUI testing are:
- Finding regression errors.
- Letting human GUI testers ("domain experts") get back to their normal jobs instead of running tests a robot should be running.
- Finding errors when the test suite ran at night, and telling testers in the morning, "You can avoid ABC, because we have some known problems there".
- Covering for application areas that didn't have unit tests. This can be a temporary situation, or if a system wasn't written to be tested easily, a more permanent thing.
- Detecting differences in behavior between platforms, i.e., Mac/Java versions, different web browsers, different operating systems.
- Pounding on the system (more or less randomly), and finding memory/resource leaks.
- Being able to use the same tool and scripts for performance tests that you use for automated GUI testing.
I don't recommend automated GUI testing on every project, but if you're paying people to test your application's GUI, it may have enough merits. In the end, it's all about quality, and ROI.
Keys to automated GUI testing and continuous integration
To me, the keys to successful automated GUI testing and continuous integration include:
- Being smart about how you a) move through the app and enter data, and b) detect state changes and failure conditions. Stated bluntly, a worst practice is to rely on pixel coordinates too much.
- Having a tool that lets you change your tests fast, so you can easily keep up with the UI changes. (Worst practice: Using a recording/playback tool that requires a new recording for minor UI changes.)
- Obviously, you need a tool that supports continuous integration. Your tests should be able to run unattended. There's nothing better than coming into work in the morning and seeing a report (or email) saying all the tests ran successfully.
Beware automated GUI testing software sales pitches and recorders
Because I learned these benefits and best practices in the battlefield, I've developed my own Agile/Automated GUI Testing software with all these ideas in mind, but it's up to you whether you want to use my software or not. It's an open source tool, there's no salesperson here, and frankly I'm creating it for myself, not for you. Your using it won't change what I do on a daily basis (unless of course you come up with some great new ideas). I happen to think it's a heck of a good tool, but that's my opinion.
In fact, I highly recommend you go to this Wikipedia page and find other tools, but please, keep everything I've written here in mind. In particular, if someone is trying to sell you down the road of a GUI recorder/playback system, and that system relies on pixel coordinates, your tests are going to break easily and break a lot, and you'll have to re-record your tests constantly -- and that's not agile at all.