NICO

TypeUX Design / Education

TopicAssistive Technology

YearSpring / 2018

Overview

NICO The Cleaning Assistant is a concept created by Matt Babb, Alysa Buchanan and myself as part of the Spring 2018 graduate graphic design studio at NC State University. This project was a culmination of exploratory research into machine learning, its affordances and application in assistive technology. NICO The Cleaning Assistant is a machine learning-powered, conversation-based mobile application and wearable that uses image recognition to help blind and visually-impaired (BVI) users locate spills and messes in the home.

fig. 1

fig. 2

fig. 3

Early Studies

While many current assistive technologies utilizing machine learning tend to deal with enhanced visibility (i.e. Seeing AI) or assisted mobility (i.e. Aira), the cooking experience remains fairly unexplored. Following an initial period of secondary research and benchmarking, our team conducted interviews with BVI users, which we then used to construct personas and user journey maps to better understand our area of focus. Once we compiled preliminary research, our team orchestrated written, verbal and visual brainstorming to encourage a rich design investigation. One method that we used to stimulate ideas was to perform "What If...?" exercises, where each member of the team contributed ideas in rapid succession. From these sessions, we developed multiple ideas, relating to different pain points in the user journey, from shopping, to cooking, to cleaning, to self-presentation (figs. 1 - 3). Ideas at this stage presumed the use of mobile devices, machine learning and conversational user interface (CUI).

fig. 4

fig. 5

Concept Development

The idea of secondary users, sighted users who had strong, coincidental relationships with the BVI users, came out of our research during this time. This led to the development of ideas that could integrate both BVI and non-BVI users. BVI users rely on systematic order of the home in order to locate items with relative ease. Friends or family members misplacing items in the home, like a prescription bottle or an important tax document, put BVI users at a major disadvantage. One of our teams ideas was to implement a conversation-based direction and sorting system, powered by a central hub that would monitor an area of the home, directing both primary (BVI) and secondary (non-BVI) users to an item's correct location (fig. 4).

We concurrently developed an alternative design concept, one that could locate items and detect hazards in the kitchen through triangulated cameras and a composite map of the space ( fig. 5). However, upon reflection of the implications of a multicamera system, two of our preestablished guidelines, affordability and autonomy had been violated. The premise of a technology that monitored your every move felt overly protective and patronizing. Returning to previous interviews and studies, our team pivoted to a mobile-based system that the BVI user had active control over, dropping the premise of "protection" altogether. Instead, we combined our idea with an older premise of a mobile application that could identify anomalies in the environment, stains, spills and messes that the BVI user would otherwise miss, using the camera on their mobile device. The user's mobile application or smart speaker would link to smart sensors around the kitchen that could prompt secondary users to return items to their proper storage location.

fig. 6

Concept Refinement

In order to understand our design interventions better, our team conducted bodystorming exercises with outside participants and lo-fi user testing with some of our interviewees. These studies revealed a number of important details including the awkwardness of holding a mobile device to identify spills and stains while simultaneously trying to clean. This prompted the idea of a tiered product system, where BVI users could pay to receive wearable eyeglasses with an embedded camera, infrared laser, microphone and bone-conduction headset (fig. 6). The glasses would link to the free mobile application and allow the BVI user to operate the system hands-free, while receiving personal auditory cues through the bone-conduction audio drivers.

Jackie has age-related far-sightedness.
Carl is Jackie’s husband.

Carl has just arrived home with groceries. Jackie is out of the home currently, but he would like to help by putting the purchases away.

Carl selects an item, baking soda, and is unsure of where it should be placed. Understanding the importance of her organizational system, he does not wish to disrupt it.

Carl asks Alexa, which is linked to his wife’s Kitchen Glasses, where she usually stores the baking soda.

The system replies with a chime. He turns in the direction of the noise. It is coming from a cabinet. He sees the cabinet light also turned on.

Carl opens the cabinet and sees a light directing him to the specific shelf. The system chimes triumphantly, indicating that he has placed the item in the correct location.

User Scenarios

Our user personas for this project, Hillary, Jackie and Carl, represent multiple ends of the visual impairment spectrum. Jackie, who lives with her husband Carl, has age-related far-sightedness. As the secondary user of the system, Carl must adhere to the sorting system Jackie has established to maintain order and help her locate items quickly. Hillary has moderate to severe Stargardt's disease, which means that she has partial peripheral vision and no central vision whatsoever. These two ends of the spectrum target different elements of the system. For Carl and Jackie, identifying proper locations of commonly used items allows Carl to adhere to Jackie's own sorting system. For Hillary, the smart glasses allow her to scan the room using her intuitive sense of body orientation, while the smart glasses use embedded sensors linked to the mobile application to process anomalies in her kitchen.

Hillary has moderate to severe Stargardt's disease. She has no central and limited peripheral vision. She has hosted a dinner with friends at her house.

After her guests have left, Hillary conducts a primary round of cleaning. Hillary decides to utilize her Kitchen Glasses to help her locate and address spots she may have missed at first.

She initiates the cleaning program by pressing the POWER button on the arms of the glasses. Hillary converses with the CUI to engage the GENERAL cleaning mode.

The glasses' camera has detected an irregularity on her counter surfaces. A small chime and gentle voice notifies the user of the area which needs attention.

The assistant guides the user during the cleaning process to ensure the entire area is cleaned efficiently. Micro-directions and progress updates are given to help Hillary clean.

The assistant detects another anomaly on the kitchen counter. After Hillary realizes the anomaly is a permanent scratch on the counter, she converses with the CUI and prompts the system to ignore the scratch during future cleanings.

CUI Design

Our system utilizes the affordances of conversational user interfaces (CUI's). Our studio had previously spent time at the beginning of the semester exploring different ways of displaying information transmition between the CUI and user. Of these interactions, visualizations and auralizations (audio-based representations) of the user activating the system through speech, system processing of user input and system response are the most important elements of a CUI. For our design and target user, graphic visualization was secondary to audio representation due to our users' reliance on sounds to navigate a given system. Our graphic user interface (GUI) is fairly simple, allowing BVI users to navigate the app more easily via their "speech selection" accessibility settings. While our primary user would rely mostly on auditory cues to operate the app, our team felt it necessary to include a limited set of visual cues as well, to account for BVI users on the partially sighted end of the vision spectrum. Although not represented here, some of the settings for our system include speech rate and verbosity (or speech detail).

CUI: Scan Start / Stop / Pause

CUI: Alert

CUI: Processing

CUI: Incorrect

CUI: Success

CUI: Completed

User: Speaking

CUI: Thinking

CUI: Replying

NICO Concept Video

Our team's final concept is NICO, The Cleaning Assistant, a machine learning-powered, conversation-based mobile application and wearable that uses image recognition to help blind and visually-impaired (BVI) users locate spills and messes in the home. The concept video explores basic usage of the free app and enhanced usage with the smart glasses option. For our team's final scenario, we chose to frame the narrative around Hillary, our BVI user with a more severe visual impairment, to showcase an array of interactions and assistive features.

NICO Concept Video