Automated GUI testing tools: best practices

Automated GUI testing tools FAQ: I've read that you've done a lot of work with automated GUI testing tools, can you share some "lessons learned" about your automated GUI testing tools experience?

I'll come back and update this article from time to time as I run into more "lessons learned", but after writing my last article (Seven benefits of automated GUI testing), I also wanted to share these ideas on "Automated GUI testing best practices."

1) Don’t rely on an automated GUI testing system that ties you to pixel coordinates

Don't rely on a recorder/playback system in an automated GUI testing tool that requires re-recording for minor GUI changes. If the GUI testing tool is specifically generating a lot of X/Y pixel/coordinate information, this is a very fragile approach.

2) Make sure you have a powerful scripting language

As a corollary to the first rule, do use an automated GUI testing tool with a simple, but powerful scripting language. Scripts like this can be maintained much more easily than any X/Y recorder tool:

# good example
click_new_user_link

c 'first name'
type 'Alvin'
tab

c 'last name'
type 'Alexander'
tab

c 'zip'
type '99654'
tab

# and so on ...

That script is much better than this innocent-looking script that ties you to x/y pixel coordinates:

# bad example
click 400, 300

c 'first name'
type 'Alvin'

click 400, 330
c 'last name'
type 'Alexander'

click 400, 360
c '99654'
type 'Alexander'

With one change in the GUI font size, the entire second example must be re-worked, but in the first example, only your initial command has to be changed. But notice that since it is a command, when you change it in one place, it will automatically work again in all of your scripts. (Or, if you relied on image recognition in that command, it may not have to be updated at all.)

(Note: The 'c' command in those examples mean "comment".)

3) Do use an automated GUI test tool that supports continuous integration

Some automated GUI test tools may let you run only one test "automatically". You want a tool that can run 50 or 100 tests (or more) in succession at night.

The continuous integration automated GUI testing pattern is simple:

1. Set the system in a known state for the next automated GUI test.
2. Run the test.
3. Regardless of the whether that test passes or fails, go to Step 1.

As I've mentioned elsewhere, there's no better feeling than coming to work in the morning and seeing that a suite of regression tests ran overnight and they all passed.

(Walking into work and seeing that some tests failed isn't always a bad feeling either.)

4) Do use an automated GUI testing tool that supports “random” testing

Once you get through those initial hurdles, now you get to a point where you just want to beat on your application with random tests to see if you can find any resource leaks, like memory leaks, or database connection leaks.

Again, if your GUI testing tool has a powerful scripting language, and you encapsulate your actions as "commands", you can do this. As a great example, we once found a memory leak in the menu system of a Java/Swing GUI application that no human tester had found.

In short, you need a system that can grow to your needs, so again, you need a powerful scripting language in your automated GUI testing tool.

5) Do use an automated GUI testing tool that supports performance testing

Some people might disagree with me here, but doesn't it make sense that your automated GUI testing tool could also be used to run performance tests against your application architecture? If you have a terrific service-oriented architecture, you can run a lot of performance tests without a tool like this, but if you really want to simulate dozens or hundreds of users, how would you like to do that?

Here are a few possibilities:

1. You can run tests against all your services, and since those services work, you can assume the overall system works. (Hint: Not a smart idea.)
2. You can have dozens or hundreds of people stop what they're doing, and test your application simultaneously.
3. You can buy a specialized performance testing tool.
4. Or you can use the scripts and commands you've created for your GUI tests to also stress-test your application.

There are benefits and drawbacks to each approach, but I don't see many people talk about this approach, yet with a good automated GUI testing tool, it's entirely possible.

My own automated GUI testing tool

I feel qualified to write these "Automated GUI testing best practices" based on my own GUI testing experience, and having developed my own Automated/Agile GUI Testing software. All the code I've written has had these best practices in mind, again, based on my own experience.

Frankly, I started down the GUI recorder/playback road many years ago, realized how bad an idea that was, threw out all that code, and started over. And I'm very happy with where the code is these days.

My GUI testing software is free and open source, so the sales pitch stops here. If you're interested in checking it out, just follow this link.

If you're not interested in using my software, that's cool, I'm going to keep developing it anyway for my own purposes. Just please bear in mind this list of automated GUI testing tool best practices, as they are indeed "lessons learned" from working in the GUI testing field.