Start Testing Before Testing Starts
Article By: Terry Anderson
It’s a familiar scenario: the development phase of a project is behind schedule and the project manager has to decide whether to give the developers more time or take the incomplete build into formal testing on its planned date. A number of factors will influence this decision.
Achieving major milestones is a key measure of project progress and one of the most significant is the move from development to formal testing. This could be either from in-house development to system/ integration test or from a supplier (who should have conducted their own testing) to the customer’s acceptance test. It is the first time that stakeholders can see a tangible result of their investment.
Consequently the project manager can be tempted to prioritise the visibility of achieving this milestone over the quality of the product delivered. In doing this acceptance criteria may be overlooked. The code is delivered incomplete, with known defects and potential defects in areas that have not been tested. However, the project manager will try to justify this from a risk management perspective by arguing that specialised test teams are better at finding defects and therefore it is more efficient to release the code to allow them to start work. Finding the defects “earlier” gives the project a better chance of staying on track. In fact there are a number of problems with this assumption that put the project at risk.
Firstly, the test team will have estimated the duration of testing based on an assumed level of quality in the release and this will have been reflected in any acceptance criteria. If these are not met the estimate may be inaccurate. If the duration of testing is increased the implementation date is at risk. The test team can be strengthened and longer hours can be worked but the impact of these measures is limited if fundamental defects continue to be detected. If testing has been suspended and is awaiting a new release these resources will be idle.
Secondly, development and unit testing is a more efficient phase in which to detect and fix defects. The code can be fixed as soon as a defect is detected without the need to compile and package formal builds. Under these circumstances completing developer or supplier testing against the planned code release is likely to have a better impact on the project timeline than releasing it to formal testing before it is ready.
Finally, multiple unscheduled releases into formal test phases can cause difficulties. Again, this may not have been allowed for in estimates and may also require additional resources to do this unplanned work. A further problem is that this obscures the quality of the release as a whole and the progress of testing. In a three month test phase the plan may include three cycles of testing, i.e. one release each month. Accepting a partial release at the beginning of this will mean that only certain features can be tested immediately. If the remaining features are released two weeks later there will be a need to repeat certain tests as well as test the new features before the next formal release. This is a simple example to demonstrate a point and in reality it is often more complicated than this. Each time code is released regression tests should be run and previously passed tests should be repeated to ensure that the latest changes and fixes have not caused problems elsewhere. All of this will have an impact on testing effort and duration as well as developer/supplier support. It will also be difficult to completely assess the quality of code and therefore the risk to the implementation until it has been possible to run all planned tests at least once.
So how can the project manager avoid this situation? In looking to take the release before it is ready he or she has rightly identified the benefits of starting testing early but has approached it in the wrong way.
The answer is to test earlier by leveraging the skills of the formal test team throughout the development lifecycle. They can review the requirements, design and functional specifications.
Their brief would be to ensure that all requirements can be traced through each deliverable. They can also be used to check for ambiguities in documents that could cause problems during later phases and that requirements are phrased in such a way that it is possible to verify through testing that they have been satisfied. They can provide a similar service to the development team or supplier testers by reviewing test documentation to ensure that there is sufficient coverage of functionality and requirements.
All of this will help to reduce the risk of late changes or major defects due to misinterpretation of scope or missing functionality not being detected until formal testing when the cost to the project of fixing these and the impact on milestones is greater.
Additionally they can provide support to test execution by developers or suppliers over and above their planned quality assurance and control activities. This includes supplying critical test cases from their test plan that must pass in this phase before the code can be released to formal test. They could actually sit with the developer or supplier tester at certain points to witness successful execution of these cases.
Once the tests against all modules have been passed and the developers or supplier have merged the code ready for release the test team should execute a subset of their tests in the developer or supplier environment. This will give them greater confidence in the quality of the code before the release and ensure that previously successful tests against individual modules continue to pass once the code is merged.
These initiatives have the advantage of combining the efficiency of the earlier phases with the structure and rigour of the later.
The key to all of this is to ensure that the project plan includes these activities, plus time for rework, and that acceptance criteria between relevant phases allow for them. Superficially this may seem to increase the duration of the project but it does reduce the risk of overrun and rework due to poor quality. Additionally, the earlier involvement of the test team should reduce the effort required in the formal test phases due to greater confidence in the quality of releases. Over time this should result in a reduction in estimated effort for the formal testing phases.
When all else fails acceptance criteria should not be ignored. They are there to ensure the quality of deliverables and to sacrifice them at the expense of achieving a milestone is a false economy that puts a project at risk and creates a false expectation among sponsors and stakeholders.
About the Author
Experience: Terry Anderson has 14 years in test analysis, management and consultancy in financial services and other industries.
Company: He is the Founder of Testing Strategies Ltd which focuses on providing advice and consultancy on quality and testing issues to small companies and those with small IT functions.
