CS assistant professor Yuhang Zhao developing AI and AR technology to support day-to-day activities of people with low vision
Yuhang Zhao joined the UW-Madison Computer Sciences faculty a mere three years ago and is already making significant contributions to the Human-Computer Interaction world. In particular, Zhao’s work is impacting people with low vision by focusing on using AI to empower them in real-life activities.
In Zhao’s research, she seeks to understand the challenges and needs of people with diverse abilities and then to design and build intelligent interactive systems to enhance human abilities.
A recent National Eye Institute (NEI) and National Institutes of Health (NIH) R01 award for her project entitled “AI-Assisted Vision: Scene-Aware Augmented Reality Systems to Support Low Vision People in Activities of Daily Living” is a huge step toward her research goals and toward helping people with low vision in their daily activities.
With the NIH grant, Zhao and her team tackle the challenges faced by people with low vision in complex activities of daily living, for example, cooking in the kitchen and safely navigating a crowded street. All of these activities involve constant motion and object interactions,
and current low vision aids like magnification, which distort users’ natural vision or diminish important details for these activities, can’t effectively support them. “As opposed to generic enhancements that apply to a low vision user’s full field of view, our research will leverage state-of-the-art AI technology to semantically understand the users’ environments and provide tailored, unobtrusive feedback only at necessary locations and time,” says Zhao.
The grant also allows Zhao to establish an annual workshop with the co-PIs Yapeng Tian from the University of Texas at Dallas and Jon E. Froehlich from the University of Washington, the local blind and low-vision communities (e.g., the Wisconsin Council of the Blind & Visually Impaired, the Wisconsin Office for the Blind and Visually Impaired), and rehabilitation organizations (e.g., UW-Madison Vision Rehabilitation Services) to present the work the team is doing and discuss opportunities and collaboration plans. This workshop will not only help the team collaborate but will also increase exposure of the low-vision community to state-of-the-art technology.
Another step toward Zhao’s research goals is a Best Paper Award (Belonging & Inclusion Track) at the ACM Symposium on User Interface Software and Technology (UIST) 2024, a top tier conference on technical human-computer interaction. The paper, entitled “CookAR: Affordance Augmentations in Wearable AR to Support Kitchen Tool Interactions for People with Low Vision,” introduces a tool for people with low vision to help them with the challenges of cooking.
CookAR: a wearable AR system that distinguishes and augments the different interaction areas of kitchen tools (e.g., knife blade vs. knife handle) to facilitate safe and efficient tool interaction for low vision users.