Yuhang Zhao joined the UW-Madison Computer Sciences faculty a mere three years ago and is already making significant contributions to the Human-Computer Interaction world. In particular, Zhao’s work is impacting people with low vision by focusing on using AI to empower them in real-life activities.
In Zhao’s research, she seeks to understand the challenges and needs of people with diverse abilities and then to design and build intelligent interactive systems to enhance human abilities.
A recent National Eye Institute (NEI) and National Institutes of Health (NIH) R01 award for her project entitled “AI-Assisted Vision: Scene-Aware Augmented Reality Systems to Support Low Vision People in Activities of Daily Living” is a huge step toward her research goals and toward helping people with low vision in their daily activities.
With the NIH grant, Zhao and her team tackle the challenges faced by people with low vision in complex activities of daily living, for example, cooking in the kitchen and safely navigating a crowded street. All of these activities involve constant motion and object interactions,
and current low vision aids like magnification, which distort users’ natural vision or diminish important details for these activities, can’t effectively support them. “As opposed to generic enhancements that apply to a low vision user’s full field of view, our research will leverage state-of-the-art AI technology to semantically understand the users’ environments and provide tailored, unobtrusive feedback only at necessary locations and time,” says Zhao.
The grant also allows Zhao to establish an annual workshop with the co-PIs Yapeng Tian from the University of Texas at Dallas and Jon E. Froehlich from the University of Washington, the local blind and low-vision communities (e.g., the Wisconsin Council of the Blind & Visually Impaired, the Wisconsin Office for the Blind and Visually Impaired), and rehabilitation organizations (e.g., UW-Madison Vision Rehabilitation Services) to present the work the team is doing and discuss opportunities and collaboration plans. This workshop will not only help the team collaborate but will also increase exposure of the low-vision community to state-of-the-art technology.
Another step toward Zhao’s research goals is a Best Paper Award (Belonging & Inclusion Track) at the ACM Symposium on User Interface Software and Technology (UIST) 2024, a top tier conference on technical human-computer interaction. The paper, entitled “CookAR: Affordance Augmentations in Wearable AR to Support Kitchen Tool Interactions for People with Low Vision,” introduces a tool for people with low vision to help them with the challenges of cooking.
“Cooking is a central activity of daily living, supporting independence as well as mental and physical health,” says Zhao. CookAR is a head-mounted AR system that recognizes and augments object affordances in real time. With CookAR, Zhao and her team are helping people with low vision to easily distinguish the different interactive components of kitchen tools, such as a knife blade vs. a knife handle, thus interacting with these tools more safely and efficiently.
With both the grant and the best paper award, Zhao and her team are working diligently to equip people with low vision to meet the unique challenges that they face. “Low vision is a complex and pervasive condition that affects millions of people in the world,” says Zhao. “However, it receives little attention in the technology literature. It is important for us to deeply understand the unique experiences and needs of low vision people and create intelligent assistive technologies that are adaptive to their contexts and preferences.”