Extended reality system including AI-assisted image capture optical character recognition for use with visual aid correction for low vision

A method of operating an extended reality (XR) system including a headset adapted to be worn by a user, a camera coupled to the headset, a display system mounted to the headset for displaying a display screen including computer-generated images thereon, and a controller including one or more processors coupled to the camera and the display system, the method including the one or more processors performing an algorithm for operating in an optical character recognition (OCR) mode and a scrolling text mode to display computer-generated images using the display system including the steps of:

• operating in the OCR mode to display a text selection screen by:
• receiving video images from the camera;
• accessing a retinal map associated with the user identifying a non-viewable boundary defining an area of the user's field of vision associated with low vision;
• generating an image distortion zone within the text selection screen associated with the identified non-viewable boundary;
• modifying the received video images to adjust a viewable image attribute of a portion of the received video images displayed within the image distortion zone;
• displaying the modified video images on the text selection screen;
• and
• selecting text images displayed within the modified video images;
• and
• operating in the scrolling text mode to display a text display screen by:
• generating machine-readable text of the selected text images using optical character recognition (OCR);
• modifying the generating machine-readable text using a lightweight language model for text correction;
• and
• displaying the modified generated machine-readable text as an animated line of text scrolling across the text display screen.

A non-transitory computer-readable storage media having computer-executable instructions embodied thereon to operate an extended reality (XR) system including a headset adapted to be worn by a user, a camera coupled to the headset, a display system mounted to the headset for displaying a display screen including computer-generated images thereon, and a controller including one or more processors coupled to the camera and the display system, when executed by the one or more processors the computer-executable instructions cause the one or more processors to perform an algorithm for operating in an optical character recognition (OCR) mode and a scrolling text mode to display computer-generated images using the display system including the steps of:

• operating in the OCR mode to display a text selection screen by:
• receiving video images from the camera;
• accessing a retinal map associated with the user identifying a non-viewable boundary defining an area of the user's field of vision associated with low vision;
• generating an image distortion zone within the text selection screen associated with the identified non-viewable boundary;
• modifying the received video images to adjust a viewable image attribute of a portion of the received video images displayed within the image distortion zone;
• displaying the modified video images on the text selection screen;
• and
• selecting text images displayed within the modified video images;
• and
• operating in the scrolling text mode to display a text display screen by:
• generating machine-readable text of the selected text images using optical character recognition (OCR);
• modifying the generating machine-readable text using a lightweight language model for text correction;
• and
• displaying the modified generated machine-readable text as an animated line of text scrolling across the text display screen.