Running User Acceptance Testing
You are working for an organisation and are told one day that you will be doing User Acceptance Testing on a new system being brought in for your operations. What do you do?
You may consider, and reject, several possibilities such as:
- Panic - however this does not do your health any good.
- Say it is not your job - but you will be using the system after it is delivered.
- Claim you are too busy - but you will still have to use the system after it is delivered.
Finally you will have to accept that you need to do it yourself, as efficiently and effectively as possible. Because after delivery, it is your responsibility.
So the question remains, what do you do?
UAT Definition
Firstly you must be aware of what User Acceptance Testing (UAT) is about. It is not about seeing if the system works according to specification. Instead it is about finding out if the system will deliver a capability your organisation needs to meet its business objectives.
The definition of UAT used by the ISTQB in their Glossary of Terms used in Software Testing is:
"Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system."
Lets take this bit by bit:
- "Formal Testing" - means that the tests need to be prepared and planned in advance, results recorded, and the tests repeatable. In other words it is not about playing with the system.
- "with respect to user needs, requirements, and business processes" - means the focus is on what capability the organisation wants from the system, not the "cool" things the system does.
- "conducted to determine whether or not a system satisfies the acceptance criteria and" - means that you decide in advance what the criteria will be for acceptance, rather than just saying that it seems good enough.
- "to enable the user, customers or other authorized entity to determine whether or not to accept the system." - means a decision is made about the acceptability of the system to the people who require the capability the system is to deliver.
This entire definition covers the key differences between UAT and any other types of testing.
UAT Development Process
How do you go about it?
A process of six steps is illustrated in the diagram below. The core four steps are: Develop Tests, Prepare to Test, Run Tests and Review Test Results. These four steps are controlled by Plan Testing and Change Management.
Develop Tests
This involves a number of activities shown in this diagram in order to develop the formal tests required to run UAT:
- Analyse Requirements - Means understanding them and looking for what is missing and inconsistent with what the system is to do for the organisation.
- Develop Scenarios - Means the preparation of business scenarios that use the system, using techniques such as Use Case.
- Derive Acceptance Criteria - Once the previous two activities are underway or completed then the set of questions to be asked about the system to see if it matches the capability needs of the organisation are prepared.
- Construct Test Cases - Test Cases are the set of specific inputs and expected results which enable one or more Acceptance Criteria to be proved.
- Write Test Scripts - Test scripts are the operational instructions for running Test Cases including: what has to be done to the system, what has to be measured, and how to do the measurement.
- Review Documents - This is a key quality process of checking all documentation produced during the development of the system.
Prepare To Test
These are the activities other than developing the tests that are required to allow testing to take place:
- Preparing the environment to run the tests - Making sure that the people, processes, hardware, software etc. are all in place to enable the testing to take place.
- Preparing Test Data - Building the data files that are required to run the test cases.
- Creating three key documents:
- Entry Criteria - What must be done before UAT starts.
- Test Procedure - The instructions about how to run the tests on the test system including the Test Scripts.
- Test Item Transmittal Report - The instructions from the developers about the system version released for UAT.
Run Tests
Run Tests is about running the tests and recording the results:
- Running the tests involves using the input and expected results from the Test Cases and applying the Test Scripts and other elements of the Test Procedure to run them.
- Recording the results involves recording in the Test Log the activities that were done in what order, and the events that happened when the test was run. Any that have actual results that differ from the expected results have the information recorded in an Incident Report. The Incident Severity is also decided at this point.
Review Test Results
When the tests have been completed then the acceptability of the system is assessed. A simple method is to check how many outstanding Incidents there are and their severity. However this is not sufficient as a simple count of Incidents does not give any idea about their impact on what the organisation wants to achieve with the system. A flawed system which delivers capability to an organisation is much better than a perfect system that does not. Therefore the test results need to be checked and traced to see what effect they have on:
- Business Scenarios,
- Requirements and their
- Business Impact.
This analysis enables a balanced decision to be made about whether to accept the system and to make recommendations about its use. The results of all this activity are then recorded in a Test Summary Report.
Plan Testing
There are two processes controlling these other four. The first is Plan Testing. This is where the scope, timescale, resources, quality and risk are decided in advance, and kept up to date throughout the UAT.
Change Management
The other controlling process is Change Management where an impact analysis made for any changes of their effect on the UAT. As the system gets nearer the stage of starting, or even during UAT, then changes can have major influence on how effective UAT will be.
Skills Needed for UAT
The above process requires a number of skills. The good news is that many people tasked with performing UAT find they have similar skills already from the job they do. What they need to do is to see how they can be effectively applied.
A well conducted UAT will give your organisation a clear view of the risks and opportunities of accepting a new system. Although it will not eliminate surprises when a system will go live, it should find about the big problems before they damage your organisation.
This site has several resources to help organisations run effective User Acceptance Testing. These include some free information on this website, as well as the on-site UAT course suitable for groups of up to ten people.
Give yourself and your colleagues the confidence to understand what has to be done in accepting a new system into your organisation by contacting me, Phil Coley.