What is a test script defect?
During a testing phase, it is not uncommon at all that testers face issues other than application failures. It is believed that during the first cycle of testing, about half the number of reported defects are related to test scripts; i.e. test script defects count about 50% of all the reported defects, while the other 50% are due to software failures and incorrect environment setup.
What causes test script defects?
The root cause of the defects found in test scripts can be attributed to the following:
- Not fully understanding the requirements or design or any other source documents that the test script is derived from and based on.
Designing test cases require thorough understanding of the application subject to test. Therefore, it is imperative that the test designers should have a clear understanding of the requirements and design flow documents, so they can write correct test cases.
- Not working with the latest version of the base documents.
Proper change control and configuration management activities are the absolute necessary efforts in order to prevent the pitfalls of working with old or wrong versions of documents.
- Not properly translating requirements and design flows in to test cases and breaking them down to test steps.
The same way a programmer needs to translate software requirement or software design into code, a tester must also be able to analyze a requirement and derive test cases from it. This in turn requires understandable and testable requirements.
- Not realizing the person executing the test script could be someone from outside without knowledge of the application under test.
A lot of the time, when testers design test cases, they assume that the only people who will execute their scripts will be their team mates or peers and will have familiarity with the applications.
Therefore, the steps are condensed or merged which could appear to be vague for someone less experienced with the applications and thus unable to follow the script to execute the tests.
Severity of Test Script Defects
Issues found in the test script can be categorized in three levels of severity:
- Issues in test script stopping the tester to carry out the execution.
This is a serious issue with a high priority, since the software cannot be tested if the test script is majorly defective, e.g. the workflow of the test case and steps do not matchup with what is written up in the requirements or design specifications.
An example could be that the workflow and behavior of an application depends on a set of test data and values that the tester should set before the execution, but the script does not contain or list the required test data and thus the tester cannot verify the workflow.
A defect should be raised and logged and the changes and corrections to the test scripts must be made immediately during the execution phase and the test should be carried out with the new version of the test script.
- Issue in test script with a workaround.
This is a moderate issue with a medium priority. Of course, if too many workarounds have to be made during the test execution phase, the priority for fixing the test script defects becomes high.
For example, an application requires username, password and a random generated key to verify user credentials. The script only asks username and password to be entered while the application is expecting the random number to be entered as well.
The tester can enter the random number as well as the username and password and carry on with the rest of the script.
- Test script with cosmetic errors / suggestions
Spelling mistakes, missing step numbers or missing information on a section of the documents, e.g. reference to source documents are all minor issues and usually of a low priority.
How to prevent Test Script Defects?
Before the testing phase begins, all test script documents (amongst other documents) should be subjected to formal reviews to prevent the above issues appearing during the formal testing phase.
If at all possible, there should be a “dry-running” of the scripts before the formal test execution begins. This gives the testers a chance to raise any uncertainties or doubts about the nature of the scripts and to minimize the number of issues listed above.
Also testers writing the test scripts must have a thorough understanding of the applications and workflows in order to write effective test cases and to maximize the exposure of defects.