7.2 Exit Criteria
Section 7, Item Pass/Fail Criteria of a test plan has four sub-sections: 7.1 Evaluation Team, 7.2 Exit Criteria, 7.3 Evaluation Process, and 7.4 Requirements Traceability Matrix.
Exit criteria for testing are part of the “Prepare to Test” step of the Testing Process. They indicate a testing phase is finished. They are the necessary conditions that have to be in place before the testing phase is run. Normally they are included in the clause 6 “Approach” of the test plan.
Without Exit Criteria the end of testing could be defined as having run out of time, or resources to continue. This says nothing about the quality of the system, and may be an indication of the quality of the testing that has been applied.
Testing Team’s Responsibilities
The testing team are the people who are mainly involved in defining and meeting the Exit Criteria. There are five main criteria to consider:
- Main functionality available.
- Important faults cleared.
- Other faults recorded.
- Documentation updated.
- Test Summary Report produced.
Main functionality available
This is the most important of the criteria, because if the software does not deliver the key functionality then there is not much point in going any further. What needs to be agreed is what form this proof should take. It could be agreement before hand on what the main features of the system are and the proof, including testing documentation, that needs to be given to show that they work. It could even be a demonstration of the main functionality of the system to the next stage of testing.
Important faults cleared
Faults which are classified as Severe, or High, and affect major areas of the system with no reasonable work rounds must be cleared before the next stage of testing takes place. With this grade of fault in place it will be difficult if not impossible to do meaningful testing in the next stage. Those faults that affect non-major areas need to be considered to see what affect they will have on the next stage of testing, but ideally they should also be cleared.
Other faults recorded
Any other faults should ideally be fixed, but if they have not been then they must be recorded along with the affect they have on the system, and the possible date for when they will be cleared. They also need to be judged to see what affect they will have on the next stage of testing.
A planning point to note is that if a system is to be tested in separate modules over a period of time then it makes sense to fix all the faults in the next module to be tested rather than concentrate on all the high level faults across all modules.
Documentation updated
The documentation that must be updated is any that is necessary for the next phase of testing. If this is not done then the next testing team will be unable to come to a judgement about the results of the testing and how it will affect them.
Test Summary Report produced
This is the most important document to be produced as part of the exit criteria. Whereas having the documentation updated means the working documents are available, the Test Summary Report gives an overview of the quality of the software which testing has shown. Without the report it is very difficult just from the working documents to find this out.
Entry Criteria Importance
The Exit Criteria set the quality boundaries for the completion of a phase of testing and they should dovetail with the next testing phase’s Entry Criteria. With this set of gateways in place it is possible to test a software system effectively and efficiently, or even on some occasions abandon it early due to poor quality.
Sample Text for 7.2 Exit Criteria
UAT will end when the following occur:
- All Incidents with business impact have been cleared.
- Remaining incidents are recorded, risk assessed, and have workarounds.
- All Business Scenarios have been tested at least once.
- Plans are in place to update development documentation as a result of UAT.
- An Acceptance Decision has been made.
- A Test Summary Report has been produced, reviewed, and approved.
Other Articles on QUALITY
- 7. Item Pass/Fail Criteria gives an overview of how to evaluate the testing data to make a pass/fail decision.
- Basics of Project Plans gives further information on item 2:"Long or Unrealistic Timescales" and item 4:"Scope Creep" as two of the five key characteristics of a project plan and the importance of balancing them.
- IEEE 829 Test Plans - where to put the criteria in a test plan.