Article by Amir Ghahrai
Automated Testing serves its best purpose when it comes to tests that are run frequently such as regression test suites to give quick feedback to the development team about the status of the current build. As more and more companies move to agile development methodologies (Scrum, XP) where 2-4 weeks releases are the norm, automated regression tests form the heart of QA. These are both Unit and System level regression tests.
Before starting to run any test (automated or manual), the tester’s task is to ensure that the correct environment is setup and ready and also any test data that are going to be used in the tests are prepared and available. During analysis and design of tests, we identify test data to state the expected results and during test execution we use the test data to compare our stated expected results against actual results. For example, if I need to design a test case to test a successful login to a web application, my expected result after login in successfully is to see the name of the user that was used to login. In order for the test to pass, I first need to identify what test data I will be using so I can state what name I should check against after I have logged in successfully.
Since we want to run automated tests frequently and rely on the results of the tests, the automated scripts have to be self contained, easy to maintain, easy to execute and most importantly the tests have to be repeatable and be executed with known test data. By repeatable, I mean being able to re-run the same test after it has just finished and the results of the second run will still be identical to the results of the first run.
During a test, it is quite common to “use up” the test data to be able to perform a positive test and if actual results meet expected results, the tests are passed. Re-run the same test again and it could fail because the test data that was used in previous run were “used up” and no longer available. Or our tests may create data in the first run, but in the second run it fails because the data is already present. Consider the following scenario as an example:
Suppose I have a test to purchase an item from an e-commerce website. In order to ensure that I purchase the correct product (rather than any product) I first need to define the attributes of the product (such as it’s name and price) that I’m interested to purchase and use that as test data for my checkout journey test. My test steps are as follows:
1 – Navigate to the e-commerce website
2 – Search for product “mountain bike 101″
3 – Click on the product
4 – Add product to basket
5 – Login as “user01″ and “pass02″
6 – Add a delivery address “Mr A, 1 Park Street, London, SE1 1PQ”
7 – Place order
The first time I run the test with above steps I get a pass. Great!
The second time I run my test it fails at step 2 and cannot go to next step and the test ends as a fail. Why? Because the first time I run my test it was able to find the product “mountain bike 101″ and proceed to checkout, but when I purchased the product it was no longer available! i.e. when I searched for “mountain bike 101″ at step 2, no products were returned and hence was not able to carry on. So, to make the test pass again I had to recreate my test data.
Ok, now I have recreated my test data and I run the script for the third time. This time the search found the product “mountain bike 101″ and was able to carry on to next step. But again my test failed, this time at step 6! It attempted to create the address but it failed because the same address already exists from the first run.
So, to make the test pass again I had to delete the created data.
I re-run the same test for the fourth time and all the steps passed and I got identical results to first run.
The above example gives us two very essential and fundamental aspects of any test execution practice. First, before running any test make sure that the test data are available and secondly, once the tests have completed the run, clear any created data.
To make the automatged tests repeatable and produce consistent results and without much maintenance, the design of tests should be such that the start up script should check for existing test data, and if they are not available, create the test data prior to test execution AND tear down script to delete any data that are created during the test execution. Only then we could rely on our automated test scripts to give us valid and consistent feedback and results.
Loading ...