Success rate assessments are a good way to identify flows that work well or badly on a website. It is also good to know why you assess the website. Do you want to know how you are progressing in a project, or are you planning on making changes if the results go down?
The purpose of the assessments
The purpose of the Success rate assessment is to see how well a workflow or function on a website is working. How many of the users could finish a task without problem and how was their experience whilst doing so? And finally, how satisfied were they with the experience?
Identify functions and workflows to be tested
If you suppose that a change or improvement is about to take place on a website it is important to figure out where the largest changes will take place. Is for example a new ordering system about to be implemented, in that case, will it affect the buying process? Are new products about to be launched that will change the structure of the website? Will a new search engine be implemented that will change the search results? Include these workflows in the test.
It is also important to consider what the central parts of a website or business is. If we’re talking about an online bookshop, the process of finding a book to buy is important. A task could then be to “Find the latest book by… and put it in the shopping basket”. The payment process can be another workflow.
If the website is a source of information, then a task could be to “Find information about… and figure out how…”
The important thing here is that it’s possible to test the same thing the same way in both the baseline assessment and in the following assessments. If you change the task you give the user the results are not comparable. It would be like comparing apples and pears.
How to do the assessments
These assessments are easiest done in user tests. You tell the user that he or she will be given a number of tasks, but if they get stuck or have a question they won’t receive any help. The user is then given the tasks, one at a time in clear instructions, either orally or written down on a card. Orally is a bit nicer, but whichever way you choose, use the same method in all the different assessments.
- The user starts the task.
- Measure the time it takes for each task. If the user is not able to finish a task, the time is not measured.
- After each task the user finishes, ask him or her to assess how difficult the task felt, on a scale from 1-5. 1 represents hard and 5 easy. (How to measure feelings is another article).
- Write down which problems those who got stuck had, since this would be a good to-do list of things to fix. If a user says they have finished, but arrived at the wrong answer, it counts as a non completed task.
How many users should be used?
Because these assessments take a relatively long time to complete it’s hard to achieve numbers that are statistically reliably. This assessment method is a qualitative assessment rather than a quantitative assessment.
Even if this makes it hard to convince the ”Excel people” – who often need quantitative results to believe they are doing the right thing – the test gives very good indications to which workflows work and which do not.
If you test 10 users you will get very clear results. Though, you should be careful to say that “60% of the users will be able to do this”, since it can be read as statistically reliably. It’s better to say “6 of the 10 users that were assessed were able to do this”.
When should you do the assessment
If you are about to make an improvement or a change on the website it is good to first do a baseline assessment on the existing website – before any changes or improvements have taken place. That assessment will then be used to compare the results to later assessments. When a change has taken place on the website a new assessment can be carried out right away, you don’t need to wait for the user to get used to the new feature on the website, or learn a new work flow, because the work flows should be intuitive enough for the user to complete a task.
How do you treat and analyze the results?
It is important to stress that the result is qualitative, as in any user test and not statistically reliable. If you have set goals for how many users that should be able to complete a task, you should also follow up the results.
Bear in minds that goals should be realistically enough for you to be able to achieve them. Perhaps you want to set a goal of 5 for experience – because you want everyone to feel really good about the work flow – think again. Some personalities are not happy even if the sun is shining, and will never rate anything a 5. That is not something you can do anything about on your website.