# US 2019/0379869 A1 · Vision-based alerting based on physical contact prediction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0379869 A1 - Document ID: US20190379869A1 - Status: Published application - Kind: A1 - Application number: 16/538,057 - Filed: 2019-08-12 - Published: 2019-12-12 - Earliest claimed: 2017-09-27 - Assignees: University of Miami - Applicants: University of Miami - Inventors: Mohamed Abou Shousha - Web page: https://patentgrep.com/patent/US20190379869A1 ## CPC - A61B 3/032: Devices for presenting test symbols or characters, e.g. test chart projectors (A61B3/036 takes precedence) - A61B 3/0025: {characterised by electronic signal processing, e.g. eye models} - A61B 3/113: for determining or recording eye movement - A61B 3/14: Arrangements specially adapted for eye photography - G02B 27/0172: {characterised by optical features} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06T 5/60: using machine learning, e.g. neural networks - G06V 20/20: in augmented reality scenes - G06V 40/193: {Preprocessing; Feature extraction} - H04N 9/646: {for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters} - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract In certain embodiments, vision-related alerting may be facilitated based on a physical contact prediction. In some embodiments, a prediction indicating defective visual field portions of a user's visual field may be generated based on feedback related to a set of stimuli displayed to the user. The defective visual field portions may be visual field portions of the visual field that fail to satisfy one or more vision criteria. Based on the prediction of the defective visual field portions, a modified video stream derived from a live video stream to be presented via a wearable device. Activation of an alert may be caused via the wearable device based on (i) a prediction of physical contact (of an object with the user) and (ii) the object not being sufficiently in any image portion of the modified video stream that corresponds to at least one visual field portion satisfying the vision criteria. ## Claims ### Claim 1 A system for facilitating vision-related alerting based on a physical contact prediction, the system comprising: a computer system that comprises one or more processors executing computer program instructions that, when executed, cause the computer system to: obtain, via a wearable device, feedback related to a set of stimuli displayed to a user during a visual test presentation, the feedback related to the set of stimuli indicating whether or how the user sees one or more stimuli of the set, the feedback comprising: (i) an eye image captured during the visual test presentation, the eye image being an image of an eye of the user, (ii) an eye response of an eye of the user to one or more stimuli, or (iii) a user modification of stimuli; generate a prediction indicating defective visual field portions of a visual field of the user based on the eye image, the eye response, or the user modification of stimuli, wherein the visual field of the user comprises visual field portions, the defective visual field portions being ones of the visual field portions that fail to satisfy one or more vision criteria; cause, based on the prediction of the defective visual field portions, a modified video stream derived from a live video stream to be displayed on one or more other display portions of the wearable device, the live video stream representing an environment of the user; detect an object in the defective visual field portions or predicted to be in the defective visual field portions; generating a prediction indicating that the object will come in physical contact with the user; and causing an alert to be presented via the wearable device based on (i) the prediction of physical contact and (ii) the object not being sufficiently in any image portion of the modified video stream that corresponds to at least one of the visual field portions satisfying the one or more vision criteria. ### Claim 2 (depends on 1) The system of claim 1, wherein the feedback related to the set of stimuli comprises the eye image captured during the visual test presentation, and the eye image is an image of a retina of the eye or an image of a cornea of the eye, and wherein generating the prediction indicating the defective visual field portions comprises generating, based on the eye image, the prediction indicating the defective visual field portions. ### Claim 3 (depends on 1) The system of claim 1, wherein the feedback related to the set of stimuli comprises the eye response to one or more stimuli, and the eye response comprises an eye movement, a gaze direction, or a pupil size change, and wherein generating the prediction indicating the defective visual field portions comprises generating, based on the eye response to one or more stimuli, the prediction indicating the defective visual field portions. ### Claim 4 (depends on 1) The system of claim 1, wherein the feedback related to the set of stimuli comprises the user modification of one or more stimuli via user input of the user, and wherein generating the prediction indicating the defective visual field portions comprises generating, based on the user modification of one or more stimuli, the prediction indicating the defective visual field portions. ### Claim 5 A method being implemented by one or more processors executing computer program instructions that, when executed, perform the method, the method comprising: obtaining feedback related to a set of stimuli displayed to a user, the feedback related to the set of stimuli comprising: (i) an eye image captured during a visual test presentation, the eye image being an image of an eye of the user, (ii) an eye response of an eye of the user to one or more stimuli, or (iii) a user modification of stimuli; generate a prediction indicating defective visual field portions of a visual field of the user based on the eye image, the eye response, or the user modification of stimuli, wherein the visual field of the user comprises visual field portions, the defective visual field portions being ones of the visual field portions that fail to satisfy one or more vision criteria; cause, based on the prediction of the defective visual field portions, a modified video stream derived from a live video stream to be displayed on one or more other display portions of a wearable device, the live video stream representing an environment of the user; detect an object in the defective visual field portions or predicted to be in the defective visual field portions; generating a prediction indicating that the object will come in physical contact with the user; and causing an alert to be presented via the wearable device based on (i) the prediction of physical contact and (ii) the object not being sufficiently in any image portion of the modified video stream that corresponds to at least one of the visual field portions satisfying the one or more vision criteria. ### Claim 6 (depends on 5) The method of claim 5, wherein the feedback related to the set of stimuli comprises the eye image captured during the visual test presentation, and wherein generating the prediction indicating the defective visual field portions comprises generating, based on the eye image, the prediction indicating the defective visual field portions. ### Claim 7 (depends on 6) The method of claim 6, wherein the eye image is an image of a cornea of the eye, and wherein generating the prediction indicating the defective visual field portions comprises generating, based on the image of the cornea, the prediction indicating the defective visual field portions. ### Claim 8 (depends on 5) The method of claim 5, wherein the feedback related to the set of stimuli comprises the eye response to one or more stimuli, and the eye response comprises an eye movement of an eye of the user, and wherein generating the prediction indicating the defective visual field portions comprises generating, based on the eye movement, the prediction indicating the defective visual field portions. ### Claim 9 (depends on 5) The method of claim 5, wherein the feedback related to the set of stimuli comprises the eye response to one or more stimuli, and the eye response comprises a gaze direction of an eye of the user, and wherein generating the prediction indicating the defective visual field portions comprises generating, based on the gaze direction, the prediction indicating the defective visual field portions. ### Claim 10 (depends on 5) The method of claim 5, wherein the feedback related to the set of stimuli comprises the eye response to one or more stimuli, and the eye response comprises a pupil size change of an eye of the user, and wherein generating the prediction indicating the defective visual field portions comprises generating, based on the pupil size change, the prediction indicating the defective visual field portions. ### Claim 11 (depends on 5) The method of claim 5, wherein the feedback related to the set of stimuli comprises the user modification of one or more stimuli via user input of the user, and wherein generating the prediction indicating the defective visual field portions comprises generating, based on the user modification of one or more stimuli, the prediction indicating the defective visual field portions. ### Claim 12 (depends on 11) The method of claim 11, wherein the user modification of one or more stimuli comprises a user manipulation of a graphical representation of one or more stimuli, and wherein generating the prediction indicating the defective visual field portions comprises generating, based on the user manipulation of the graphical representation, the prediction indicating the defective visual field portions. ### Claim 13 One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors, cause operations comprising: obtaining feedback related to a set of stimuli displayed to a user, the feedback related to the set of stimuli indicating whether or how the user sees one or more stimuli of the set; generate a prediction indicating defective visual field portions of a visual field of the user based on the feedback related to the set of stimuli, wherein the visual field of the user comprises visual field portions, the defective visual field portions being ones of the visual field portions that fail to satisfy one or more vision criteria; cause, based on the prediction of the defective visual field portions, a modified video stream derived from a live video stream to be presented via a wearable device, the live video stream representing an environment of the user; detect an object in the defective visual field portions or predicted to be in the defective visual field portions; generating a prediction indicating that the object will come in physical contact with the user; and causing activation of an alert via the wearable device based on (i) the prediction of physical contact and (ii) the object not being sufficiently in any image portion of the modified video stream that corresponds to at least one of the visual field portions satisfying the one or more vision criteria. ### Claim 14 (depends on 13) The one or more non-transitory computer-readable media of claim 13, wherein the feedback related to the set of stimuli comprises an eye image captured during a visual test presentation, and wherein generating the prediction indicating the defective visual field portions comprises generating, based on the eye image, the prediction indicating the defective visual field portions. ### Claim 15 (depends on 14) The one or more non-transitory computer-readable media of claim 14, wherein the eye image is an image of a cornea of the eye, and wherein generating the prediction indicating the defective visual field portions comprises generating, based on the image of the cornea, the prediction indicating the defective visual field portions. ### Claim 16 (depends on 13) The one or more non-transitory computer-readable media of claim 13, wherein the feedback related to the set of stimuli comprises an eye response of an eye of the user to one or more stimuli, and the eye response comprises an eye movement of an eye of the user, and wherein generating the prediction indicating the defective visual field portions comprises generating, based on the eye movement, the prediction indicating the defective visual field portions. ### Claim 17 (depends on 13) The one or more non-transitory computer-readable media of claim 13, wherein the feedback related to the set of stimuli comprises an eye response of an eye of the user to one or more stimuli, and the eye response comprises a gaze direction of an eye of the user, and wherein generating the prediction indicating the defective visual field portions comprises generating, based on the gaze direction, the prediction indicating the defective visual field portions. ### Claim 18 (depends on 13) The one or more non-transitory computer-readable media of claim 13, wherein the feedback related to the set of stimuli comprises an eye response of an eye of the user to one or more stimuli, and the eye response comprises a pupil size change of an eye of the user, and wherein generating the prediction indicating the defective visual field portions comprises generating, based on the pupil size change, the prediction indicating the defective visual field portions. ### Claim 19 (depends on 13) The one or more non-transitory computer-readable media of claim 13, wherein the feedback related to the set of stimuli comprises a user modification of one or more stimuli via user input of the user, and wherein generating the prediction indicating the defective visual field portions comprises generating, based on the user modification of one or more stimuli, the prediction indicating the defective visual field portions. ### Claim 20 (depends on 19) The one or more non-transitory computer-readable media of claim 19, wherein the user modification of one or more stimuli comprises a user manipulation of a graphical representation of one or more stimuli, and wherein generating the prediction indicating the defective visual field portions comprises generating, based on the user manipulation of the graphical representation, the prediction indicating the defective visual field portions.