Usability Testing Case Study: Improving design based on user feedback & data
Usability testing was a part of full case study, please read the overview section which will take minimum 2 minutes to understand the context
What’s a usability study?
Usability testing is a technique used in user-centered interaction design to evaluate a product by testing it on users.
When performed correctly, it can help a business by:
- Reduce development costs: early testing helps reduce redesign costs in the later stages
- Reduce support costs: a product with better usability will minimize potential customer support issues
- Increased sales: better usability for customers will make them more likely to make a purchase or to increase their purchase size
Project Background
Passion project done over a period of 14-weeks. Goal was optimizing onboarding and activation for first time users of Yulu (e-bike sharing app)
Project objective
- Identify the problems with the existing onboarding flows
- Suggest areas of optimizations and limit them to 3 high-impactful areas which are likely to improve the user experience.
Project Recap
In case you didn’t read the case study overview
The following user problems were identified after conducting generative research
Pilot solutions
To solve the problems and achieve our project objective, onboarding and activation was broken down into four tasks:
- Registering and sign up
- Finding a ride or reserving a ride
- Adding money before starting ride
- Rental agreement acceptance before starting
And pilot solutions were wireframed
View the wireframed solutions that will be tested later by clicking here
01 — Planning usability study
All good usability studies start with a plan…
a) Research goal
Test the effectiveness of our new optimized onboarding flow with first time app user
Sub-research goals:
- What can we learn from the user flow, or the steps that users take, to find a a ride? Are there parts of the user flow where participants get stuck?
- Do users think new feature “reserve a ride” is helpful and/or useful? Are there reasons why participants won’t use this feature?
- Are there design changes we can make to improve the user experience with this feature?
b) Methodology
Since this was a side-project, I opted for moderated usability study with both in-person tests and remote tests that were conducted online using Zoom & Google Meet. Aiming to test the design with minimum 4, maximum 5 participants
Users will be asked to think aloud and share what’s going through their mind as they progress with each screen
c) Key Performance Indicators
- Completion rate
- Time on task
- Likert’s effectiveness scale survey (to measure solution effectiveness)
d) Participant criteria
Participants must be:
- Age Group: 16–35
- First time users that have never used the Yulu app before
- Must have heard of e-bike sharing service or have used some other related apps before
Note: Since this was an unsponsored study, I asked my close friends and family that fit the above criteria to participate and help me out.
e) Defining tasks
As mentioned above, the onboarding and activation was broken down into 4 parts to make it easier to solve for the customer journey.
- Registering and sign up
- Finding a ride or reserving a ride
- Adding money before starting ride
- Rental agreement acceptance before starting
The tasks above were identified by looking at the user flow that I created before wireframing design solutions.
02- Conducting Study
Before conducting studies with participants, I prepared an excel sheet with a custom dashboard (as you’ll see in images below) for KPIs that make it easy to quantify responses and made space for user comments since this was a “think aloud” test
Sub tasks were also identified and written down to make them easier to track and see where the deviations are happening
Before Starting — Task Framing
Imagine it’s a Saturday afternoon. You’re with couple of friends and want to try Yulu for the first time. The nearest Yulu zone is 15–20 minutes away. You’ve already installed the app on your phone, now open the app, sign up and book your first ride.
Task A) Register & Signing Up
Task B) Reserve a ride
Task C) Add payment
Task D) Rider safety training
This same process was followed and tests were conducted to gather responses from 3 other test participants. Participants were later coded as T1, T2, T3, T4 to keep track.
Here’s the prototype I ran the initial usability tests with, you can also try it by clicking the link here
03- Analyzing Insights
The statistics and numbers collected from all four test participants was collated and combined in a usability test report.
- Success threshold percentages were set based on the nature of the task
- Averages were calculated by looking at user performances. Turning data into averages was easy since there were only 4 users and each user meant 25%.
Task 01 — Sign up
Solution objective: Segment the users based on their jobs to be done so that we can provide a more personalized experience for each segment.
Completion Rate: Each screen had a single call to action, therefore task completion rate was 100% for each step. No deviations were possible
Time on task: Average time 4 participants spent on the task was 25s
Key insight: The sign up flow uses the existing flows that the app currently uses and works perfectly, therefore no other changes needed outside of segmenting the users by use case in step 3
Task 02— Reserve a ride
Solution objective: Give users the ability to reserve a ride if they are in proximity of a Yulu zone and would like the ability to reserve a bikes when they want to explore the city with their group of friends.
Completion Rate: Each screen had a single call to action, therefore task completion rate was 100% for each step. No deviations were possible by design.
Time on task: Average time 4 participants spent on the task was 34s
Key insight: This was a new feature introduction, therefore users had to be told what the feature does, but during testing users skipped over the screens where this feature was explained
Time on task average was achieved but it didn’t accomplish the objective, which was to introduce the new feature and make users aware of what it does and how it’s priced so that they will adapt to it and use the app likewise.
Task 03 — Add payment to wallet
Solution objective: Allow users to easily add money to the their Yulu wallet by suggesting smart top-ups based on the number of minutes/hours they’d like to ride the bike for.
This is where the designs weakness was exposed. The most critical flow failed to pass task completion rate and the time on task was high
Completion Rate: Users completed the task, but with multiple deviations in the beginning of the flow
Time on task: Average time 4 participants spent on the task was 42s. The reason was confusion in the selecting amount and tap suggessted money step.
Key insight: Instead of selecting payments by tapping on suggested top-up amount user opted to add their amount manually and were confused with the time and amount that was mentioned.
Task 04 — Rider safety training
Solution objective: Reduce driving misconducts and improve road safety.
Completion Rate: Since this was an educational quiz, there were a lot of deviations from the original path which led to increased time on task
Time on task: 52s time on task which is a lot when someone is trying to get activated and use the product in the customer journey
Likert Effectiveness Scale: Since we were also trying to measure the effectiveness of the solution, the scale was also included for this step.
Key insight: The design solution did not work as expected and failed to solved the original problem of helping drivers ride safely on an electric bike
04- Synthesis & Forming Recommendations
Task 1 — Sign up
It was observed that 4/4 (100%(users were able to select the use-case relevant to them
Participant quote: “What if I sign up as an explorer and later decide to become a commuter, how would I change that?”
Recommendation: Allow user to change their use-case in account section
Task 2— Reserve a ride
It was observed that 3/4 (75%) users would like an option to cancel bike reservation
Participant quotes:
- “What if I don’t reach the zone on time, it should have an option to cancel”
- “Am I reserving a bike of my choice or is it randomly allotted?
Recommendation:
- Allow users for an option to cancel their reservation within a certain time period
- Although only one user pointed this out, this could be a real concern. Nobody would want to reserve a bike with low battery and have no option to change it. Therefore if it’s technologically feasible, we could give user the choice to select from a bike based on battery available
Task 3— Add payment
It was observed that 4/4 (100%) users found adding payment and ride fare complicated.
Participant quotes:
- “This feels overwhelming and too cluttered, wish all of this was broken down into steps like how apps usually do”
- “How many minutes am I riding the bike for? Do I need to calculate it before adding an amount?”
Recommendation: Make adding payment simpler and close to the original flow in the app
Task 4— Completing rider safety training
The solution was measured on Likert’s effectiveness scale and users did not find this to be an effective solution for decreasing driving misconducts.
- Extremely Effective
- Very Effective
- Moderately Effective
- Slightly Effective (1/4)
- Not at all effective(3/4)
Recommendation: Run more explorations to find a solution that solves the problem of reducing driving misconducts and increasing road safety.
Design iterations and final solution
After making iterations to the design solution after considering all the recommendations, I was able to create the final prototype which when tested again helped achieve our project objective.
Test the final prototype yourself here
Lessons and takeaways
- Feedback from users will make the product better — don’t take it personally no matter how harsh.
- Design is about trade-offs and balancing conflicting requirements. Scoping problems, identifying which ones to prioritize and work on + why takes copious research.
- Don’t get too attached to a particular solution or design instead look at everything as a prototype. With each round of feedback the prototype keeps evolving and making the designs better
Thank you for reading all the way through. If you’re intrigued you can read the whole process of how I went from identifying problem in onboarding to creating solutions and testing it with users here
www.anujraman.com/yulu-case-study
If you’re interested in reading more case studies, visit https://www.anujraman.com/case-studies
