How to Choose Which Tests to Automate?

How do you choose which tests to automate and which tests to leave for manual testing?

Before you start automating a test, you need to see what benefits you get by automating the test after you factor in the time, effort and resource invested in test automation.

Below are some factors to consider to help identify which manual tests should or should not be automated. As the old saying goes, just because you can automate something doesn’t necessarily mean that you should.

Here are some guidelines to help identify good candidates for test automation:

Tests that should be automated:

  • Business critical paths – the features or user flows that if they fail, cause a considerable damage to the business.
  • Tests that need to be run against every build/release of the application, such as smoke test, sanity test and regression test.
  • Tests that need to run against multiple configurations — different OS & Browser combinations.
  • Tests that execute the same workflow but use different data for its inputs for each test run e.g. data-driven.
  • Tests that involve inputting large volumes of data, such as filling up very long forms.
  • Tests that can be used for performance testing, like stress and load tests.
  • Tests that take a long time to perform and may need to be run during breaks or overnight.
  • Tests during which images must be captured to prove that the application behaved as expected, or to check that a multitude of web pages looks the same on multiple browsers.

Generally speaking, the more repetitive the test run, the better it is for automation.

Also remember that tests are not the only candidates for automation. Tasks such as setting up or creating test data for manual exploratory testing are also great candidates for automation.

Tests that should not be automated:

  • Tests that you will only run only once. The only exception to this rule is that if you want to execute a test with a very large set of data, even if it’s only once, then it makes sense to automate it.
  • User experience tests for usability (tests that require a user to respond as to how easy the app is to use).
  • Tests that need to be run ASAP. Usually, a new feature which is developed requires a quick feedback so testing it manually at first
  • Tests that require ad hoc/random testing based on domain knowledge/expertise – Exploratory Testing.
  • Intermittent tests. Tests without predictable results cause more noise that value. To get the best value out of automation the tests must produce predictable and reliable results in order to produce pass and fail conditions.
  • Tests that require visual confirmation, however, we can capture page images during automated testing and then have a manual check of the images.
  • Test that cannot be 100% automated should not be automated at all, unless doing so will save a considerable amount of time.

Can you think of any other candidates for automating (or not automating) a test?

3 Replies to “How to Choose Which Tests to Automate?”

  1. Your article doesn’t do any justice to the subject of test automation – people coming to this article would only see the same “reasons” that were written before the year 2000 – why don’t you write something different instead of copying old stuff and add a new dimension to our industry?

    Anyway on to your article, I have a few observations and a different way of looking at this…

    When you begin your test automation efforts there really is only one question you have to answer: Why am I being asked to automate our existing tests.

    The answer to this question will tell you what the most pressing points the business is trying to solve and that automation may or may not be the answer.

    You talk about “…what benefits you get by automating…” but you don’t go on to explain what you mean by “a benefit” – can you expand on that point?

    Instead of talking about benefits, why don’t you mention outcomes? That would help to frame how automated testing can be viewed as a value added service (to the business) and not just an expensive overhead with no real benefits. (Remember businesses only deal in profit and loss)

    You also mention “time, effort, and resource” as things that can limit or even halt your automation efforts – and it’s true they will kill your automation efforts if you haven’t done the proper upfront analysis to determine the best approach to the “Why?” question.

    But, for now, lets follow your thought pattern through regarding these limits…Did you consider not having enough people or the right skills within the existing team (is that what you mean by resource?), and what about not having the correct infrastructure (to run your tests on) or even a decent architecture for your framework?

    Finally you say this…As the old saying goes, just because you can automate something doesn’t necessarily mean that you should.

    I’ve been involved with test automation for a very long time and I’ve never heard that sentence defined as a “saying”. In fact it was quoted in a couple of books and conferences around the turn of the century.

    You may take my comments as unfair criticism, and that’s your choice, but in reality they are my observations on what you’ve written and not necessarily what you know.

Leave a Reply