Jill Cao

one-on-one & group usability testing remote & local usability testing

Time: 2014 - 2015

Project goal: Tab-Completion improves users’ programming efficiency. When a user has entered the starting characters of an identifier, he/she may hit the Tab key and MATLAB will bring up a list of matching identifiers that the user may choose from without having to type the entire identifier.

This project enables Tab-Completion on the properties and methods of an object in the context of a class file.

Usability testing challenge & approach: Tab-completion is something at the user’s fingertip. Most of the time, users don’t think hard or even consciously about using it. Thus, a challenge in testing the design was how to simulate a closer-to-natural testing situation while making sure users give us the kind of data we wanted to collect.

To solve this problem, I adopted a graduated prompting approach. Users came into the usability sessions without knowing what we were testing exactly. They were told they would be working with MATLAB class files but not tab-completion. If they did not voluntarily use tab-completion by a certain point during testing, they’d be prompted to use it by answering several questions with each getting more specific about what the directions we wanted them to take. This approach made sure that users acted as naturally as possible for as long as we could allow.

Challenge working with the team & my approach: Initially, the team was resistant to usability testing. They believed that it would be “artificial” to test a feature at the user’s fingertip and that there existed no way to get untainted data through usability testing for any design. I proposed the graduated prompting technique to address the “artificial” concern and explained the value of testing with real users admitting testing could never be 100% “natural”.

In addition, I leverage a shared concern about the design to emphasize the value of testing. Tab-completion worked in some situations but not in other situations. If we did not test, we would not understand how will users be impacted by this inconsistency. We would not have any ideas as to whether users would continue to embrace tab-completion or whether they would get confused and give up on it altogether. Testing would help answer these questions.

In summary, I convinced my team to give usability testing a try by catering my test plan to address the team’s concerns.

Summary of UX methods & goals

UX methods	Goals
Usability testing with external users using graduated prompting	To understand users’ perceptions of an “incomplete” design and the design’s impact on user mental models and workflows.
Interactive design review with internal stakeholders	To gather internal reactions toward the design by having each stakeholder work through scenarios from the usability tests.

Results and impact: Through watching real users and stakeholders interact with the design and listening to what they had to say in person, I helped my team understand the benefits of usability testing which served the basis for future engagement with users. For example, we discovered critical usability issues that the team hadn’t anticipated. The team addressed them before the design was release resulting in improved user experience. In addition, the feedback from usability testing helped to prioritize the team’s upcoming work plan. Instead of basing the work priority on instincts, my team chose to focus on what was important to users.