Skip to content

Project checkpoint 5

Think-aloud Usability Evaluation

Possible Points Due Date
40 Points April 7th - Before Class

Description

In this Project Checkpoint, your project group will conduct a usability evaluation study of your revised web app from Project Checkpoint 4 to identify 5 usability issues. The usability study will

  1. Involve 4 participants, who may be friends, family, coworkers, or students in this class but may not be members of your project group
  2. Consist of a single task that should last 10 - 20 minutes and must be at least 5 minutes long. Your task should have an overall objective but might enumerate several inter-related objectives (e.g., plan a home remodeling project, including listing materials you need to purchase and the steps you will take)
  3. Collect think-aloud data and a screencast (w/ audio) of each participant. You are free to use whatever screen recorder you would like to use. For ease of use, you can conduct these studies over Zoom and record the screen using Zoom's built in tools.
  4. Identify and report critical incidents that occur during the tasks
  5. Collect subjective reactions from participants through a brief open-ended post-task interview conducted immediately following each task.
  6. Analyze critical incidents and participant reactions to identify at least 5 usability issues.

Designing the Study

You should first design your study taking into consideration the following:

  1. Design a task that exercises one or (ideally) more of your web app's use cases. Your task should describe a concrete, fictional scenario and a specific objective for the participant to accomplish within this scenario. A task might involve interacting with software other than your web app (e.g., searching the web, and using your app to record notes) to understand how your app fits into a larger context, particularly if your use cases can be accomplished very quickly. For example, if you app is a todo app, you might have the user use your app to brainstorm a plan.
  2. Prepare a one or two paragraph description of your scenario and task.
  3. Conduct a pilot test to ensure the task is understandable and of the right length and difficulty, revise your task, and repeat (as needed).

Conducting the Study

Next, conduct your study with 4 participants. For each participant, ensure that you

  1. Introduce the experimenter(s) and the purpose of the study
  2. Answer any questions the participant has about the purpose of the study.
  3. Begin the task
    1. Give participant the task description
    2. Prompt participant to think-aloud
    3. Start the screen recorder
  4. Observe the participant as they work
    1. Take notes on participant behavior
    2. Prompt participant to continue to think-aloud, when necessary
    3. Note critical incidents as they occur, jotting down both the time and the context of what occurred
  5. Conduct a post-task interview
    1. When unclear, ask participants clarification questions about what they did and why they did it
    2. Ask open-ended questions such as what they liked best, what they found most challenging, and what they think might be improved.
  6. Wrap up the study
    1. Thank the participants
    2. Stop the screen-recorder and reset the study setup for the next participant (if any)

Analyzing and Reporting Results

First, review your notes and the screencasts to identify and report critical incidents. For each critical incident, you should report:

  1. A participant letter (A - D) and a critical incident number (1 - n) (e.g., A5)
  2. Problem statement: summary of problem and effect on user (but not a solution!)
  3. User goals: what was the larger goal that the user was working towards
  4. Immediate intention: at the moment in time when problem occurred, what was user trying to do
  5. Possible causes: speculate on what might have led the user to take the action they did.

Next, analyze the responses to the post-task interview questions. For each participant, build a short writeup summarizing responses to the interview questions.

Finally, from the critical incidents and interview responses from all of the participants, group common issues and try to find an underlying cause. For the 5 issues you believe to have the largest impact on the usability of your app, write a one to two paragraph description that summarizes the usability issue, citing all of the related critical incidents (e.g., A1, B5, C3, C6) and any related interview responses.

Note that we expect each team will find at least 5 usability issues from this exercise if the study is designed correctly. However, if you do find fewer than 5 issues, you should write a brief reflection on the study, discussing whether the design was adequate, and whether a different design may have led to the discovery of more issues. It is important to note that piloting the study is important here, because a study that is too simple may not uncover many usability issues. Therefore, it is important to pilot your study and revise the task/use cases to make sure that they are complex enough to discover potential usability issues. As noted above, a task might involve participants using other software (e.g., a search engine) in addition to your web app.

Submission Instructions

Submit your assignment through Blackboard below as document file containing your (1) final task description, (2) descriptions of the critical incidents for each participant, (3) summary of the interview responses for each participant, (4) descriptions of the 5 usability issues you identified. If you are working in a group, only one person per group should submit on behalf of your group.

If you are working in a group, each group member must separately submit a Statement of Contributions document with your name, list of group members, and a one or two paragraph statement describing how each of the group members (including you) contributed to this Project Checkpoint.