Pleasures and Pitfalls of Automation

I remember the excitement I felt when I first started to get my mind around the idea of testing using automation. I had started a testing job right out of university and after a while I was transitioned towards doing some cleanup work on the regression system.  After figuring the system out and glimpsing some of the power of automation, I started to get really excited about this idea of test automation.  There seemed to be so much about it that promised to do away with all the parts of my job I was growing to dislike.  No more boring monotonous work!  No more manually executing the same test cases from one build to the next.  No more putting features into 'beta' mode since testing hadn't gotten to them yet. Automation seemed like the answer to all the problems we as an under-resourced (of course) test team faced.  However, several years later the automation that we have tried to put in place hasn't lived up to all our expectations.  We still have to do boring work.  We might not have to re-run as many test cases by hand any more, but how many times were we updating the same test case in the automation system?  And we are still having features go into beta due to lack of testing resources.  What happened?  Why is automation failing us?

I suspect that a part of the answers lies in a lack of understanding what automation is good at.  Are we perhaps trying to automate the wrong things?  Are we trying to automate too much?  Or maybe we haven't committed enough resources to developing a more robust system?  Where is the balance between automation and manual testing?  I order to help myself answer some of these questions I would like to first list some of the reasons that automation is good, along with some of the reasons it is bad.  Hopefully a list like this will help clarify when a test should be automated and when it should not be.

Benefits:

• Machines don't get bored.  Humans do.
• Machines don't make mistakes.  They will repeat the exact same actions over and over
• Machines are really really fast at computation or comparison
• Better protection against project regression
• Can provide faster feedback. 

Pitfalls

• Can be expensive to maintain
• Machines don't make mistakes.  They will repeat the exact same actions over and over (yes I know that is on the benefits list as well....some times automation misses things that would be obvious to a human tester)
• Can give false sense of confidence
• Can take some time and training for people to be able to use the automation system.

I'm sure there are many more items that could go on each list and many of the items could be broken down into several parts as well.  These are just some of the obvious ones I could come up with in after a few minutes of thought.

Although my initial enthusiasm about automated testing may have waned a bit over the years, I do still feel that it is a useful tool in the testers tool pouch, but I think I still have a lot to learn about how to properly wield it.   Next time I will try to look a little more closely at how we can measure the costs and benefits of an automated test.