Debriefing Phase
The purpose of the Debriefing phase is to process the results of the User Day and incorpoate findings in the development of the product.
What to Bring to the Debriefing Phase
- Tests logs.
- Further notes containing observations, problems, ideas and open questions.
- Log from the Usability Round Table with user feedback.
- Plus/minus list from the Usability Round Table with suggested solutions for the negative aspects.
Discussing the Test Results
The presentation of the course and results of the parallel tests should be sorted so that you can see how many users had a certain problem in each area of the tested application.
Sort the problems by their importance (central function, typical task scenario) and frequency, collect the solutions, and discuss them in order of importance.
Usage problems can occur at the task level (goal level) or operating level (system interaction). Both types of problem can occur as long as you are testing with a set of realistic tasks. It is important to recognize the level at which a problem occurs and its origin, so that you know what you are discussing (concepts, structure, interaction design, or visual design):
|
Problem description
|
Problem level
|
|---|---|
| When the user returns from a dialog window, the last data objects selected are no longer selected. The user thus loses his or her current work context and is forced to select the same object again and again. | Detailed Interaction Design on application level |
| While working, the user expresses a desire to have an overview of the results of his or her work. | Structural Interaction Design Mismatch with information needs |
| A user does not use a function even if it was on the screen. | Visual/Interaction Design |
| A user explains that this application would not be of value for his or her work unless it is able to display aggregated data. | Conceptual Design |
| A user complains about the 3D-look of the buttons. | Visual Design |
| The user has problems scrolling in the table control. | Detailed Interaction Design on generic control level |
Consolidating Other Notes
During the User Day a lot of notes are collected, not all of which can be discussed individually. Some of them will overlap with observations made by other colleagues.
You should therefore create clusters of related observations together with the team. The recommended method is to copy the observations on post-it notes and create an affinity diagram. This is done as follows:
- All participants stick their post-it notes on a board in no particular order.
- All participants read the post-it notes, remove post-it notes with similar remarks, and hang them in columns on another board.
- This is done until all the post-it notes have been transferred from one board (unsorted) to the other. Of course, existing columns must be taken into consideration.
- The columns are then assigned titles, and unsuitable remarks are removed from the columns. A good column size is 3-6 post-it notes.
- You should combine several columns to form one term of a higher level.
Only the clusters or problem areas are discussed at this point. Direct design ideas will always be developed during the tests and discussion. You can now analyze them again for consolidated problems.
The rule for ideas and solutions is: the higher a solution is placed in the affinity diagram, the better it is.
Updating the User/Task Analysis
If the test and discussions produced large deviations to the user roles and scenarios existing in a user/task analysis, these should be enhanced or changed. This should be done with care because direct observations from customer visits are always more realistic than what the customers report about their work.
Source: User Day Toolkit