UW Madison researchers have developed a method that tracks and models low vision users’ gazes via eye tracking data, recognizes their intent, and generates multi-modal augmentations (e.g., visual, audio) based on the current visual tasks to support low vision people in daily activities. For example, in a reading task, a system can detect a user's reading behaviors via eye tracking (e.g., switch line, difficult word for low vision people to recognize, revisit previous chapters) and generate augmentations (e.g., highlighting the next line so that they won't struggle identifying the next line due to their vision loss). A recurrent neural network is trained using sighted user data that was labeled and segmented and is fine-tuned using low-vision user data. The model can use the context of the user’s action to infer the next action and provide an augmentation.