# US 12,118,655 B2 · Minimal latency apparatus and method for supporting foveated rendering > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,655 B2 - Document ID: US12118655B2 - Status: Granted - Kind: B2 - Application number: 18/211,156 - Filed: 2023-06-16 - Published: 2023-12-21 - Granted: 2024-10-15 - Earliest claimed: 2022-06-17 - Assignees: Inseye Inc. - Applicants: Inseye Inc. - Inventors: Michal Meina; Piotr Krukowski; Piotr Redmerski - Web page: https://patentgrep.com/patent/US12118655B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06T 15/005: {General purpose rendering architectures} - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 27/0172: {characterised by optical features} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G02B 2027/0147: {comprising a device modifying the resolution of the displayed image} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract A virtual reality/augmented reality (VR/AR) wearable assembly is described herein. The VR/AR wearable assembly includes a display device, a photosensor oculography (PSOG) assembly including an eye tracking assembly, and a processor coupled to the PSOG assembly and the display device. The processor including an eye-tracking module configured to execute an algorithm to render computer-generated images on the display device including the steps of detecting a saccade of a corresponding eye of the patient via the eye tracking assembly and determining an initial saccade gaze position location of the corresponding eye associated with the detected saccade, determining a peak velocity of the saccade, determining a final saccade end gaze position based on the determined peak velocity of the saccade, and rendering a foveated image on the display device at an image location corresponding to the determined final saccade end gaze position. ## Claims ### Claim 1 A virtual reality/augmented reality (VR/AR) wearable assembly comprising: a wearable frame adapted to be worn over a patient's eyes; a pair of photosensor oculography (PSOG) assemblies mounted to the wearable frame, each PSOG assembly including: a micro-OLED display; an eye tracking assembly including a plurality of light emitters configured to illuminate an area of a corresponding eye and a plurality of photosensors configured to receive reflected light from different portions of the illuminated corresponding eye; and a substantially disk-shaped pancake optics positioned between the eye tracking assembly and the micro-OLED display; and a processor coupled to each PSOG assembly, the processor including an eye-tracking module configured to execute an algorithm to render computer-generated images on the display device including the steps of: determining eye movement tracking based on processing signals from the plurality of photosensors positioned to gather eye movement information; detecting a movement of the eye position of the corresponding eye of the patient via a corresponding eye tracking assembly and determining whether the detected eye movement is the beginning of a saccade; upon determining the eye movement is the beginning of a saccade, determining an initial saccade gaze position location of the corresponding eye associated with the saccade; determining a peak velocity of the saccade; determining an amplitude of the saccade based on the determined peak velocity; determining a final saccade end gaze position based on the determined peak velocity and the determined amplitude of the saccade; and rendering a foveated image on a corresponding micro-OLED display at an image location corresponding to the determined final saccade end gaze position. ### Claim 2 (depends on 1) The VR/AR wearable assembly of claim 1, wherein each PSOG assembly includes: a ring support frame coupled to the wearable frame; a first group of photosensors and light emitters mounted along a first side of the ring support frame; and a second group of photosensors and light emitters mounted along a second side of the ring support frame opposite the first group of photosensors and light emitters. ### Claim 3 (depends on 1) The VR/AR wearable assembly of claim 1, wherein the processor is programmed to execute the algorithm including the steps of: determining an intermediate gaze position location of the corresponding eye associated with the detected saccade; and determining the final saccade end gaze position based on the amplitude of the saccade, the initial saccade gaze position location, and the intermediate gaze position location of the corresponding eye. ### Claim 4 (depends on 1) The VR/AR wearable assembly of claim 1, wherein the processor is programmed to execute the algorithm including the steps of: estimating current eye parameters and predicting eye position before the end of the saccade. ### Claim 5 (depends on 1) The VR/AR wearable assembly of claim 1, wherein the processor is programmed to execute the algorithm including the steps of: detecting an other type of eye-movement of the corresponding eye via the eye tracking assembly; determining a first gaze position location associated with the detected other type of eye-movement; estimating corresponding eye parameters associated with the detected other type of eye-movement; predicting a second eye position based on the estimated corresponding eye parameters associated with the detected other type of eye-movement; and rendering a second foveated image on the display device corresponding to the determined second eye position. ### Claim 6 (depends on 5) The VR/AR wearable assembly of claim 5, wherein the processor is programmed to execute the algorithm including the steps of: detecting the other type of eye-movement including at least one of a smooth pursuit and an intersaccadic drift. ### Claim 7 (depends on 2) The VR/AR wearable assembly of claim 2, wherein each PSOG assembly includes: a thermistor mounted to the ring support frame between the first and second group of photosensors and light emitters for sensing an ambient temperature adjacent the ring support frame. ### Claim 8 A method of operating a VR/AR wearable assembly including a wearable frame adapted to be worn over a patient's eyes, a pair of PSOG assembles mounted to the wearable frame with each PSOG assembly including a micro-OLED display, an eye tracking assembly including a plurality of light emitters configured to illuminate an area of a corresponding eye and a plurality of photosensors configured to receive reflected light from different portions of the illuminated corresponding eye, and a substantially disk-shaped pancake optics positioned between the eye tracking assembly and the micro-OLED display, and a processor coupled to each PSOG assembly, the method including the processor performing an algorithm to render computer-generated images on the display device including the steps of: determining eye movement tracking based on processing signals from the plurality of photosensors positioned to gather eye movement information; detecting a movement of the eye position of the corresponding eye of the patient via a corresponding eye tracking assembly and determining whether the detected eye movement is the beginning of a saccade; upon determining the eye movement is the beginning of a saccade, determining an initial saccade gaze position location of the corresponding eye associated with the saccade; determining a peak velocity of the saccade; determining an amplitude of the saccade based on the determined peak velocity; determining a final saccade end gaze position based on the determined peak velocity and the determined amplitude of the saccade; and rendering a foveated image on a corresponding micro-OLED display at an image location corresponding to the determined final saccade end gaze position. ### Claim 9 (depends on 8) The method of claim 8, wherein each PSOG assembly includes: a ring support frame coupled to the wearable frame; a first group of photosensors and light emitters mounted along a first side of the ring support frame; and a second group of photosensors and light emitters mounted along a second side of the ring support frame opposite the first group of photosensors and light emitters. ### Claim 10 (depends on 8) The method of claim 8, including the processor performing the algorithm including the steps of: determining an intermediate gaze position location of the corresponding eye associated with the detected saccade; and determining the final saccade end gaze position based on the amplitude of the saccade, the initial saccade gaze position location, and the intermediate gaze position location of the corresponding eye. ### Claim 11 (depends on 8) The method of claim 8, including the processor performing the algorithm including the steps of: estimating current eye parameters and predicting eye position before the end of the saccade. ### Claim 12 (depends on 8) The method of claim 8, including the processor performing the algorithm including the steps of: detecting an other type of eye-movement of the corresponding eye via the eye tracking assembly; determining a first gaze position location associated with the detected other type of eye-movement; estimating corresponding eye parameters associated with the detected other type of eye-movement; predicting a second eye position based on the estimated corresponding eye parameters associated with the detected other type of eye-movement; and rendering a second foveated image on the display device corresponding to the determined second eye position. ### Claim 13 (depends on 12) The method of claim 12, including the processor performing the algorithm including the steps of: detecting the other type of eye-movement including at least one of a smooth pursuit and an intersaccadic drift. ### Claim 14 (depends on 9) The method of claim 9, wherein each PSOG assembly includes a thermistor mounted to the ring support frame between the first and second group of photosensors and light emitters for sensing an ambient temperature adjacent the ring support frame. ### Claim 15 A non-transitory computer-readable storage media having computer-executable instructions embodied thereon for operating a VR/AR wearable assembly including a wearable frame adapted to be worn over a patient's eyes, a pair of PSOG assemblies mounted to the wearable frame with each PSOG assembly including a micro-OLED display, an eye tracking assembly including a plurality of light emitters configured to illuminate an area of a corresponding eye and a plurality of photosensors configured to receive reflected light from different portions of the illuminated corresponding eye, and a substantially disk-shaped pancake optics positioned between the eye tracking assembly and the micro-OLED display, and a processor coupled to each PSOG assembly, when executed by the processor the computer-executable instructions cause the processor to perform an algorithm to render computer-generated images on the display device including the steps of: determining eye movement tracking based on processing signals from the plurality of photosensors positioned to gather eye movement information; detecting a movement of the eye position of the corresponding eye of the patient via a corresponding eye tracking assembly and determining whether the detected eye movement is the beginning of a saccade; upon determining the eye movement is the beginning of a saccade, determining an initial saccade gaze position location of the corresponding eye associated with the saccade; determining a peak velocity of the saccade; determining an amplitude of the saccade based on the determined peak velocity; determining a final saccade end gaze position based on the determined peak velocity and the determined amplitude of the saccade; and rendering a foveated image on a corresponding micro-OLED display at an image location corresponding to the determined final saccade end gaze position. ### Claim 16 (depends on 15) The non-transitory computer-readable storage media of claim 15, wherein each PSOG assembly includes: a ring support frame coupled to the wearable frame; a first group of photosensors and light emitters mounted along a first side of the ring support frame; and a second group of photosensors and light emitters mounted along a second side of the ring support frame opposite the first group of photosensors and light emitters. ### Claim 17 (depends on 15) The non-transitory computer-readable storage media of claim 15, wherein the computer-executable instructions cause the processor to perform the algorithm including the steps of: determining an intermediate gaze position location of the corresponding eye associated with the detected saccade; and determining the final saccade end gaze position based on the amplitude of the saccade, the initial saccade gaze position location, and the intermediate gaze position location of the corresponding eye. ### Claim 18 (depends on 15) The non-transitory computer-readable storage media of claim 15, wherein the computer-executable instructions cause the processor to perform the algorithm including the steps of: estimating current eye parameters and predicting eye position before the end of the saccade. ### Claim 19 (depends on 15) The non-transitory computer-readable storage media of claim 15, wherein the computer-executable instructions cause the processor to perform the algorithm including the steps of: detecting an other type of eye-movement of the corresponding eye via the eye tracking assembly; determining a first gaze position location associated with the detected other type of eye-movement; estimating corresponding eye parameters associated with the detected other type of eye-movement; predicting a second eye position based on the estimated corresponding eye parameters associated with the detected other type of eye-movement; and rendering a second foveated image on the display device corresponding to the determined second eye position. ### Claim 20 (depends on 19) The non-transitory computer-readable storage media of claim 19, wherein the computer-executable instructions cause the processor to perform the algorithm including the steps of: detecting the other type of eye-movement including at least one of a smooth pursuit and an intersaccadic drift.