Visual assist chatbot for improved accessibility of content by a visually impaired user

A method for providing assistance to a visually impaired user navigating a website, the method comprising:

• observing user interaction with a webpage in real time as a user navigates the webpage via a user interface (UI) device;
• capturing user interaction data describing the user interaction with the webpage, the user interaction data being generated without applying a visual impairment test or evaluation, the user interaction data comprising a number of navigation errors of the user with the webpage occurring within a period of time, the number of navigation errors being a count of user misses during attempted clicking of components of the webpage;
• based on capturing the user interaction data, predicting a visual assist (VA) level of the user by a trained machine learning (ML) model using the user interaction data and one or more visual impairment criteria, the visual assist level indicating a degree of visual impairment of the user, the one or more visual impairment criteria comprising a minimum threshold number of navigation errors, wherein the number of navigation errors exceeding the minimum threshold number of navigation errors indicates a presence of the visual impairment, and wherein the trained ML model predicts an absence of the visual impairment in response to the number of navigation errors falling below the minimum threshold number of navigation errors;
• modifying a layout of the webpage in response to the VA level exceeding a threshold VA level, modifying the layout comprising increasing a font size of text on the webpage, increasing a size of text boxes on the webpage, and removing content of the webpage to decrease a difficulty of webpage navigation;
• generating an audible VA activation prompt via the UI device in response to the visual assist level exceeding a threshold minimum value, the audible VA activation prompt requesting whether the user wants to activate a VA chatbot;
• and
• activating the VA chatbot, wherein the VA chatbot is configured to perform text-to-speech and to assist the user in performing a task during navigation of the webpage in response to VA activation by the user.

A non-transitory computer storage device having computer-executable instructions stored thereon, which, upon execution by a computer, cause the computer to perform operations comprising:

• generating user interaction data describing user interaction with a webpage in real time as a user navigates the webpage via a user interface (UI) device, the user interaction data being generated without applying a visual impairment test or evaluation, the user interaction data comprising a number of navigation errors of the user with the webpage occurring within a period of time, the number of navigation errors being a count of user misses during attempted clicking of components of the webpage;
• based on generating the user interaction data, predicting a presence of a visual impairment associated with the user, by a trained machine learning (ML) model using the user interaction data and one or more visual impairment criteria, the visual impairment criteria comprising a minimum threshold number of navigation errors;
• generating a predicted visual assist (VA) level using the number of navigation errors, wherein the number of navigation errors exceeding the minimum threshold number of navigation errors indicates the presence of the visual impairment, and wherein the trained ML model predicts an absence of the visual impairment in response to the number of navigation errors falling below the minimum threshold number of navigation errors;
• modify a layout of the webpage in response to the VA level exceeding a threshold VA level, modifying the layout comprising increasing a font size of text on the webpage, increasing a size of text boxes on the webpage, and removing content of the webpage to decrease a difficulty of webpage navigation;
• generating an audible VA activation prompt via the UI device in response to the trained ML model predicting the presence of the visual impairment, the audible VA activation prompt requesting whether the user wants to activate a VA chatbot;
• and
• activating the VA chatbot, wherein the VA chatbot configured to perform text-to-speech and to assist the user in performing a task during navigation of the webpage in response to VA activation by the user.