Usability Testing Case Study: Improving design based on user feedback & data

Anuj Raman
9 min readFeb 2


Usability testing was a part of full case study, please read the overview section which will take minimum 2 minutes to understand the context

What’s a usability study?

Usability testing is a technique used in user-centered interaction design to evaluate a product by testing it on users.

When performed correctly, it can help a business by:

  1. Reduce development costs: early testing helps reduce redesign costs in the later stages
  2. Reduce support costs: a product with better usability will minimize potential customer support issues
  3. Increased sales: better usability for customers will make them more likely to make a purchase or to increase their purchase size

Project Background

Passion project done over a period of 14-weeks. Goal was optimizing onboarding and activation for first time users of Yulu (e-bike sharing app)

Project objective

  • Identify the problems with the existing onboarding flows
  • Suggest areas of optimizations and limit them to 3 high-impactful areas which are likely to improve the user experience.

Project Recap

In case you didn’t read the case study overview

The following user problems were identified after conducting generative research

Pilot solutions

To solve the problems and achieve our project objective, onboarding and activation was broken down into four tasks:

  • Registering and sign up
  • Finding a ride or reserving a ride
  • Adding money before starting ride
  • Rental agreement acceptance before starting

And pilot solutions were wireframed

Rider safety training prototype flow
Add payment prototype flow

View the wireframed solutions that will be tested later by clicking here

01 — Planning usability study

All good usability studies start with a plan…

a) Research goal

Test the effectiveness of our new optimized onboarding flow with first time app user

Sub-research goals:

  • What can we learn from the user flow, or the steps that users take, to find a a ride? Are there parts of the user flow where participants get stuck?
  • Do users think new feature “reserve a ride” is helpful and/or useful? Are there reasons why participants won’t use this feature?
  • Are there design changes we can make to improve the user experience with this feature?

b) Methodology

Since this was a side-project, I opted for moderated usability study with both in-person tests and remote tests that were conducted online using Zoom & Google Meet. Aiming to test the design with minimum 4, maximum 5 participants

Users will be asked to think aloud and share what’s going through their mind as they progress with each screen

c) Key Performance Indicators

  • Completion rate
  • Time on task
  • Likert’s effectiveness scale survey (to measure solution effectiveness)

d) Participant criteria

Participants must be:

  • Age Group: 16–35
  • First time users that have never used the Yulu app before
  • Must have heard of e-bike sharing service or have used some other related apps before

Note: Since this was an unsponsored study, I asked my close friends and family that fit the above criteria to participate and help me out.

e) Defining tasks

As mentioned above, the onboarding and activation was broken down into 4 parts to make it easier to solve for the customer journey.

  • Registering and sign up
  • Finding a ride or reserving a ride
  • Adding money before starting ride
  • Rental agreement acceptance before starting

The tasks above were identified by looking at the user flow that I created before wireframing design solutions.

View full user flow here

02- Conducting Study

2 remote user tests

Before conducting studies with participants, I prepared an excel sheet with a custom dashboard (as you’ll see in images below) for KPIs that make it easy to quantify responses and made space for user comments since this was a “think aloud” test

Sub tasks were also identified and written down to make them easier to track and see where the deviations are happening

Before Starting — Task Framing

Imagine it’s a Saturday afternoon. You’re with couple of friends and want to try Yulu for the first time. The nearest Yulu zone is 15–20 minutes away. You’ve already installed the app on your phone, now open the app, sign up and book your first ride.

Task A) Register & Signing Up

Task A, T1 Usability stats

Task B) Reserve a ride

Task B) T1 Usability Stats

Task C) Add payment

Task C) T1 Usability Stats

Task D) Rider safety training

Task D) T1 Usability Stats

This same process was followed and tests were conducted to gather responses from 3 other test participants. Participants were later coded as T1, T2, T3, T4 to keep track.

Here’s the prototype I ran the initial usability tests with, you can also try it by clicking the link here

03- Analyzing Insights

The statistics and numbers collected from all four test participants was collated and combined in a usability test report.

  • Success threshold percentages were set based on the nature of the task
  • Averages were calculated by looking at user performances. Turning data into averages was easy since there were only 4 users and each user meant 25%.

Task 01 — Sign up

Solution objective: Segment the users based on their jobs to be done so that we can provide a more personalized experience for each segment.

Sign up report

Completion Rate: Each screen had a single call to action, therefore task completion rate was 100% for each step. No deviations were possible

Time on task: Average time 4 participants spent on the task was 25s

Key insight: The sign up flow uses the existing flows that the app currently uses and works perfectly, therefore no other changes needed outside of segmenting the users by use case in step 3

Task 02— Reserve a ride

Solution objective: Give users the ability to reserve a ride if they are in proximity of a Yulu zone and would like the ability to reserve a bikes when they want to explore the city with their group of friends.

Reserve a ride report

Completion Rate: Each screen had a single call to action, therefore task completion rate was 100% for each step. No deviations were possible by design.

Time on task: Average time 4 participants spent on the task was 34s

Key insight: This was a new feature introduction, therefore users had to be told what the feature does, but during testing users skipped over the screens where this feature was explained

Time on task average was achieved but it didn’t accomplish the objective, which was to introduce the new feature and make users aware of what it does and how it’s priced so that they will adapt to it and use the app likewise.

Task 03 — Add payment to wallet

Solution objective: Allow users to easily add money to the their Yulu wallet by suggesting smart top-ups based on the number of minutes/hours they’d like to ride the bike for.

Add payment report

This is where the designs weakness was exposed. The most critical flow failed to pass task completion rate and the time on task was high

Completion Rate: Users completed the task, but with multiple deviations in the beginning of the flow

Time on task: Average time 4 participants spent on the task was 42s. The reason was confusion in the selecting amount and tap suggessted money step.

Key insight: Instead of selecting payments by tapping on suggested top-up amount user opted to add their amount manually and were confused with the time and amount that was mentioned.

Task 04 — Rider safety training

Solution objective: Reduce driving misconducts and improve road safety.

Rider safety training report

Completion Rate: Since this was an educational quiz, there were a lot of deviations from the original path which led to increased time on task

Time on task: 52s time on task which is a lot when someone is trying to get activated and use the product in the customer journey

Likert Effectiveness Scale: Since we were also trying to measure the effectiveness of the solution, the scale was also included for this step.

Key insight: The design solution did not work as expected and failed to solved the original problem of helping drivers ride safely on an electric bike

04- Synthesis & Forming Recommendations

Task 1 — Sign up

It was observed that 4/4 (100%(users were able to select the use-case relevant to them

Participant quote: “What if I sign up as an explorer and later decide to become a commuter, how would I change that?”

Recommendation: Allow user to change their use-case in account section

Task 2— Reserve a ride

It was observed that 3/4 (75%) users would like an option to cancel bike reservation

Participant quotes:

  • “What if I don’t reach the zone on time, it should have an option to cancel”
  • “Am I reserving a bike of my choice or is it randomly allotted?


  • Allow users for an option to cancel their reservation within a certain time period
  • Although only one user pointed this out, this could be a real concern. Nobody would want to reserve a bike with low battery and have no option to change it. Therefore if it’s technologically feasible, we could give user the choice to select from a bike based on battery available

Task 3— Add payment

It was observed that 4/4 (100%) users found adding payment and ride fare complicated.

Participant quotes:

  • “This feels overwhelming and too cluttered, wish all of this was broken down into steps like how apps usually do”
  • “How many minutes am I riding the bike for? Do I need to calculate it before adding an amount?”

Recommendation: Make adding payment simpler and close to the original flow in the app

Task 4— Completing rider safety training

The solution was measured on Likert’s effectiveness scale and users did not find this to be an effective solution for decreasing driving misconducts.

  • Extremely Effective
  • Very Effective
  • Moderately Effective
  • Slightly Effective (1/4)
  • Not at all effective(3/4)

Recommendation: Run more explorations to find a solution that solves the problem of reducing driving misconducts and increasing road safety.

Design iterations and final solution

After making iterations to the design solution after considering all the recommendations, I was able to create the final prototype which when tested again helped achieve our project objective.

Final prototype flow

Test the final prototype yourself here

Lessons and takeaways

  • Feedback from users will make the product better — don’t take it personally no matter how harsh.
  • Design is about trade-offs and balancing conflicting requirements. Scoping problems, identifying which ones to prioritize and work on + why takes copious research.
  • Don’t get too attached to a particular solution or design instead look at everything as a prototype. With each round of feedback the prototype keeps evolving and making the designs better

Thank you for reading all the way through. If you’re intrigued you can read the whole process of how I went from identifying problem in onboarding to creating solutions and testing it with users here

If you’re interested in reading more case studies, visit



Anuj Raman