Problem

Papa John’s approached us to evaluate their web-based ordering functionality. The current version was put together quickly and they’re confident there is much room for improvement. Their key concerns are the discoverability of their site from a google search, clarity of the pizza ordering process, and ease of creating or signing in to an account.

Approach

Determine Which Usability Evaluation to Conduct
Due to a short, four-month schedule, we determined that a single round of summative testing be performed near the end of the current redesign phase. This approach allows us to test with actual Papa John’s site users to determine whether the site requirements and goals have been achieved. This task-based approach includes a script for consistent prototype testing among all participants that focus on the key areas where we want them to succeed.

Screeners & Tasks
Working with an advanced user demographic group, we created a series of screener questions and key tasks for users to work through. Screeners determine appropriate participants for usability studies. Designations like continue and terminate are assigned to each screener to narrow down test participants and ensure they meet the advanced demographic. Tasks are developed to ensure usability for key workflows. They will inform the designers of critical areas that are crucial for the site to perform effectively.

Quantitative Measurement
For this test, both success and error rate will be analyzed based on key tasks the user performs on the site. By focusing on success and error rate we will be able to gain a larger sample size of data more quickly. While a large number of participants need to be recruited, these sessions can largely be conducted remotely and unmoderated and still produce useful valid results.

The biggest limitation is that this approach needs a substantial number of participants which can be costly. But, to ensure the results are accurate we need to test a great number of people and duplicate results to be sure the data produced is reliable and valuable.

Research Protocol
A research protocol was developed so the team could perform several individual usability sessions in the limited time provided. This document ensures that consistent questions are asked in each session so that the data collected is similar and can be analyzed and synthesized.

We asked participants to order three specific pizzas, sign up for deals with their email only, and contact corporate with a complaint.

Moderate a Usability Session
A series of interviews were conducted with target users who had previous experience ordering pizza online. These interviews were conducted digitally using web conferencing software. Both audio and video were recorded to aid in note-taking and follow-up observations after the sessions.

To gain a baseline for this study, four participants who had previous online pizza ordering experience were recruited to assist in learning the usability of the Papa John’s site.

Analysis of Data
Using a spreadsheet, the team created an interview patterns document to discover similar areas of success and failure when testing web-based, pizza-ordering tasks on the Papa John’s website. The analysis of these tasks allowed us determine recommendations for improvement.

Results

Final Report
Overall, the expectations of this study were met largely due to our participants having previous experience ordering pizza online. However, because only basic functionality was covered in this study, it is recommended that to address additional usability issues, like account management and using coupons or activating special deals, more participants should be recruited and interviewed.