Running a User Day

The Usability Round Table

The Usability Round Table is a separate focus-group-like discussion at the end of the parallel testing phase. Users have the opportunity to exchange their experiences and have a chance to discuss with the development team their concerns and needs. For the development team this is a great chance to establish a first consolidated list of the major issues revealed in the User Day.

Introduction

The Usability Round Table collects users' impressions of the application. It can be applied to all testing methods that are performed in parallel sessions in one User Day with multiple users.

Typically the Usability Round Table takes up the last onbe to two hours of the User Day and users share their experiences of the software application with the development team. The moderator of the Usability Round Table draws up a plus-minus list for the tested application. The entries on the minus side are discussed in detail and suggestions are collected as to how these points can be avoided or corrected. At the end, the users can optionally fill out questionnaires on the application tested and about the User Day method in general.

Participants and Roles

To avoid giving the users the inhibiting feeling of being watched, the discussion group should not be too large. Therefore the users and moderators should outnumber any other participants. With the recommended number of six users, one to two moderators, and about three to four members of the development team, there should not be more than ten to twelve participants. Developers should mainly be passive observers. Their task is to help the moderator if application-specific questions come up.

Users

The users report their findings about the application being tested and make contributions to the discussion.

Moderator

A moderator supervises the Usability Round Table and leads the discussion. You can also have a pair of moderators instead, in which case they also share the task of keeping the log (see Logger).

The moderator initiates the discussion with his or her questions and is responsible for documenting the contributions to the discussion in a comprehensible form and language so that developers who are not participating can understand them. A usability coordinator is suitable for this role since both expertise in user-integrated procedures and knowledge of the application and its context are combined here in one person. Training in moderation techniques is recommended.

Logger

This person writes down keywords from the discussion. For example, the reasons for and explanations of the user contributions are important since they are not recorded on post-it notes or cards and are only mentioned in the conversation with the participants. The log can be used to inform interested persons and members of the development group not taking part in the discussion about the contents and results. Important detailed information contained in the conversation should also be recorded for use in the design revision. For example, a user rejects the planned use of a drop-down list box for a field because too many entries appear. Until receiving this information, the developers might assume that not more than ten entries would be necessary at a company.

Developers

The developers take a passive part in the discussion. The developers themselves do not make any contributions to the discussion, but are available for consultation. If a developer does not understand the user's contribution to the discussion, he or she asks questions for his or her own understanding. He or she may not comment on the user's statements, on the functionality, or on the design of the user interface in future versions of the application you are testing.

Other Stakeholders

People outside the development team with an interest in the results of the User Day can participate in this discussion. Typically, documentation writers, quality managers or sales and marketing people might be involved. They act in the same manner as the developers.

Objectives of Usability Round Table

The discussion has the following objectives:

  • The experiences of users during the testing sessions are communicated, collected and handled in a structured manner. It is important that the users say as much as possible to provide the development groups with sufficient information. The objective is to generate a large number of individual contributions without having to worry about the qualitative basis of each individual statement.
  • The qualitative aspect of individual user opinions is tested automatically in the Round Table, as the reaction of other test users will indicate whether it is really an individual opinion or a group opinion.
  • The users discuss aspects that were not mentioned when working with the system, which creates more useful material for the development group.
  • Contradictory statements and different user opinions can stimulate discussion. Record the reason for each opinion expressed and assessment made. The objective is not to find the "right" opinion.
  • Depending on the focus of your User Day, you should emphasize topics related to either the conceptual, the structural, or the detailed user interface design.

The discussion does not have the following objectives:

  • The discussion is not about functionality that is missing, would be useful, or is planned for the future.
  • No decisions or promises should be made about open design questions during the discussion, but you should encourage discussion of a solution from the point of view of the user.

Resources Required

  • The room for the discussion should be equipped with a (mobile) projector and a computer so that any statements made by the users can be examined using the application you are testing.
  • Use at least two flip charts for comments on positive and negative experiences.
  • Provide pens for all users.
  • Check (and ask the users while preparing for the User Day) whether a video camera may run for the purposes of logging the results. The video camera is focussed on the pin boards and is primarily for documenting the conversation. The advantage of using a video recording is that it is no longer necessary for someone to write the log.

Schedule

One to three hours are needed for the discussion; Typically this begins in the afternoon when the users and other participants return from the lunch break. It is also possible to schedule parallel test sessions and subsequent Round Table in half a day.

Introduction

In this phase, the moderator explains the general course of the discussion, the objectives, and the desired results. The participants also introduce themselves if required.

  • Duration: 15 minutes.
  • Material: Name cards, pens.

Creating the Plus-Minus List

The plus-minus list is used to structure the discussion. The procedure is extremely simple for both the moderator and the users and helps to structure the Round Table. The moderator lists all the positive and negative aspects expressed by the users on the flipcharts. You should allow 15-20 minutes for this.

The moderator should first collect all positive aspects experienced in the test sessions. Although you may switch between positive and negative aspects, it is usually hard to return on positive aspects once you start to discuss negative aspects.

If collection of positive aspects slows down you may move on by collecting negative aspects. The negative aspects are the starting point for a discussion of the solution in the next phase. Consequently, you should clearly separate the collection of negative points and the discussion about possible solutions.

The moderator points out to the users that ugly and uncomfortable aspects are also negative aspects that would otherwise be considered to be unimportant. The objective is to gather as many aspects as possible in the plus-minus list. Quantity has priority over quality or (apparent) irrelevance of a user experience.

By asking appropriate starter questions the moderator has the control and responsibility to influence the collection of plus/minus points into a direction which fits to the focus of the User Day. The moderator can stimulate the feedback by asking prepared questions that focus on concept, structure, functionality, and user interface detail of the product. You may also use the terms of ISO 90421 Part 10 to structure the discussion. It is a good idea to hand out those questions as general focus questions prior to the test sessions to encourage the test user to focus on such problems.

To support the authenticity of individual opinion against group dynamics, you may also ask users to write down their personal plus/minus list first on a paper before collecting those points all together.

Finally, you should take care to check and log how many users agree with this point.

Discuss Negative Aspects and Brainstorm Possible Solutions

If there are a lot of negative aspects, you should prioritize them, i.e. each user should mark the two aspects that he or she considers most important. You can then deal with the aspects according to the number of times they are marked.

The discussion should result in a consolidated and comprehensive problem statement for each negative point so that interested persons who are not participating in the User Day can read about the discussion and use the results. Adhere to the general rules for collecting feedback, i.e. the users' remarks should not be commented upon and developers should only ask comprehension questions.

  • The moderator discusses each negative aspect in turn and asks all users to explain if and why this is a negative point. The person writing the log (second moderator) notes down this explanation.
  • The user who submitted this negative point might be asked to explain the problem again to create a shared understanding. In no way should there be any evaluation about the quality of the feedback. Questions from the developers to increase their comprehension are particularly welcome.
  • After having a clear understanding about all negative aspects, the discussion of solutions begins. The moderator must keep an eye on the time schedule and use the time available efficiently (normally there is approximately 60 minutes). The objective is to discuss proposed solutions for every problem identified. It is very important that the moderator divide the discussion based on the different types of problems (concept, structure, interaction design, visual design) and that he or she avoid long discussions about things that cannot be decided anyway. Try to stay close to the original problem as occurred in the test session.
  • If the users have no proposals for a solution at the beginning of the discussion, the moderator can prompt the user by asking the following questions:
    • How they would have expected the application to work?
    • What would have helped them handle the problem?
    • How the application could have helped more?
  • The users discuss these solutions in the presence of the developers. Not only the results of the discussion but also the different contributions and ideas for solutions are recorded. Other users are asked to present a proposal for a solution. The developers also provide spontaneous ideas for a solution and ask whether the solution would solve the user's problem.

    At the end of the discussion, the users or the moderator should note the proposed solution or alternatives and competing solutions should be noted briefly.

  • The moderator makes sure that the developers do not discuss whether the suggestions for improvement can really be implemented. The primary aim is to collect as many solutions as possible from the user. Future (planned) release versions should not be mentioned either. In short, the discussion should not go into details.
    • Duration: 90 minutes
    • Material: Pens

Summarizing and Saying Goodbye

The summary gives the users a final opportunity to comment of the application.

  • The moderator gives a final overview about results of the discussion. The plus list is used to mention that there were some positive aspects as well. The negative list is used to outline the opportunities of the product.
  • The moderator thanks the users for their participation, gives them a present, and says goodbye. He or she may also explain the further activities, if any are planned in the end phase.
    • Duration: 15 minutes
    • Material: Cards, for each user, pens, pins, thank-you gifts for participants

Additional Phase (Optional)

After the User Day is finished, many development teams use the time to enhance the user's experience of the application. For example, the development group can present planned new developments or this phase can also be used to ask the users what they thought of the User Day. Further marketing and sales aspects can also be included here, for example special presentations, a product-related questionnaire or an overall evaluation of the user interface.

 

Source:  User Day Toolkit