Chalmers V1.5 New Skills Evaluative Research (December 2019 - February 2020)

Project Length: 8 Weeks Scope: Generative Research, Card Sorting, Evaluative Research, Usability Testing Role and Responsibilities: Research Practice Lead, Discussion Guide Development, Participant Recruitment, Renumeration, Interview Facilitator, Synthesis and Analysis

Project Overview

Problem

On April 14, 2019 Ample Labs had just launched in Beta was able to sign the City of Barrie, Ontario for a 4 month pilot and and began to receive media attention after launching a successful crowdfunding campaign on IndieGoGo to fund the Development costs. Between April 2019 and October 2019 usage of Chalmers jumped by 553% to over 11,000 user sessions.

Following our launch our Pilot client requested that we to offer more services as they felt our offering was too limited for their needs as they needed more the aggregate data to better forecast service usage. Furthermore, we had high levels of user requests for additional services through our customer support channels.

Context

• We decided to design new features (known as skills) for the chatbot, specifically: legal aid, income support, health (medical), health (dental), housing, and food banks all based on the services suggested both by our Pilot customer and users.
• Before going into development it was determined that we needed to conduct user research with the folks who are experiencing homeless to gauge effectiveness.
• The purpose of the research was to assess the usability and comprehension of the Legal Aid, Income Support, Health (Medical), Health (Dental), Housing and Food Bank skills, specifically to address:
• The navigability of the new information architecture and ensure users able to comprehend the concepts associated with the new skills.

Methods

• A moderated usability test was created to assess the ability for our target users, homeless youth, to use the new features.
• The test had 8 tasks for users to do and task completion was assessed on a pass/fail basis.
• The tests were done at the Covenant House Toronto, a youth shelter in downtown Toronto over the course of a Saturday morning and afternoon.
• Covenant House assisted with participant recruitment to build credibility and reduce no-shows.
• Each test was 30 mins long and there was one facilitator, one note taker and one observer.
• 10 Youths ages 14-29 across the gender and racial spectrum.
• Mobile phones were used as the testing device.

What Did We Find?

For Chalmers

• Participant Mental Models were different than our assumptions
• People struggled to understand the purpose of income support and how it was tied to receiving money.
• Participants see Social Insurance and Ontario Works as equivalent.
• Most People have a lot of issues understanding how the Canadian health care system works.
• Newcomers struggled with understanding the difference between when to access ambulatory care (Walk-In clinics) and critical care (Emergency Room visits).
• Most viewed mental health care as distinct from physical health care but did know know where to properly access these resources.
• People struggle (especially newcomers) to understand the vocabulary and verbiage used by social support services.
• Most Participants did not understand key terms (Food Banks).
• Most Participants did not understand various skill headings (Income Support)

For the User Research Team

• Don’t make assumptions with the mental models of your users when adding complex features.
• Emojis are a great visual aid to communicate the purpose of a skill.
• The information architecture of the user flows needs to be re-engineered as the people got lost in the housing, income support and health user flows leading to high rates of task failure.
• Recognize cultural bias and how it can influence design leading to incorrect assumptions that do not meet user needs.

Outputs

Insight Presentation

Figma Prototype Comments

I turned the usability insights into Figma comments to help the designer quickly iterate on the revisions to produce an updated design.

What did we do next?

Findings were translated into a cardsort using Optimal Workshop to better understand user mental models around health care and financial resources.

We revised our prototype with the insights we gained from both the previous usability testing findings and the cardsort.

We went back and conducted another round of user interviews using Maze.Design using our revised prototype with the youth at Covenant House Toronto.

We began to develop user stories and had begun to prepare the updated services prototype for development.

Then the COVID-19 Pandemic hit. The pandemic had a widespread impact service availability and operating hours leading us to reprioritize the development of the additional new features to ensure that our existing service records were accurate provided the proper requirements so that users could effectively access services.

Based on the insights of our Hidden Homeless Research Project only the Food Bank skill was put into rapid development in March 2020. As a result, Food Bank inquiries on Chalmers between March 2020 to May 2020 increased by 1400% to over 10,400 requests.