# US 2024/0004466 A1 · Generating and rendering primitives in a frame period with late update of eye tracking information for fast foveated rendering

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004466 A1
- Document ID: US20240004466A1
- Status: Published application
- Kind: A1
- Application number: 18/470,381
- Filed: 2023-09-19
- Published: 2024-01-04
- Earliest claimed: 2018-05-17
- Assignees: Sony Interactive Entertainment Inc.
- Applicants: Sony Interactive Entertainment Inc.
- Inventors: Andrew Young; Javier Fernandez Rico
- Web page: https://patentgrep.com/patent/US20240004466A1

## CPC

- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G09G 5/37: Details of the operation on graphic patterns (G09G5/38 takes precedence)
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 5/046: Forward inferencing; Production systems
- G06T 15/005: {General purpose rendering architectures}
- G09G 5/391: Resolution modifying circuits, e.g. variable screen formats
- G09G 5/393: Arrangements for updating the contents of the bit-mapped memory
- G06N 3/045: Combinations of networks
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G09G 2320/0261: in the context of movement of objects on the screen or movement of the observer relative to the screen
- G09G 2320/106: Determination of movement vectors or equivalent parameters within the image
- G09G 2340/0407: Resolution change, inclusive of the use of different resolutions for different screen areas
- G09G 2354/00: Aspects of interface with display user
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method for updating information for a graphics pipeline including executing in the first frame period an application on a CPU to generate primitives of a scene for a first video frame. Gaze tracking information is received in a second frame period for an eye of a user. In the second frame period a landing point on an HMD display is predicted at the CPU based at least on the gaze tracking information. A late update of the predicted landing point to a buffer accessible by the GPU is performed in the second frame period. Shader operations are performed in the GPU in the second frame period to generate pixel data based on the primitives and based on the predicted landing point, wherein the pixel data is stored into a frame buffer. The pixel data is scanned out in the third frame period from the frame buffer to the HMD.

## Claims

### Claim 1

A method, comprising:
executing in a frame period an application to generate a plurality of primitives for a video frame;
receiving in the frame period gaze tracking information for at least one eye of a user;
predicting in the frame period a landing point on a display based on the gaze tracking information, wherein the landing point corresponds to a gaze direction of the at least one eye; and
rendering in the frame period the video frame based on the primitives and the landing point that is predicted.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining a foveal region in the video frame centered around the landing point that is predicted; and
rendering one or more pixel sized fragments of the plurality of primitives located in the foveal region at high resolution.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
rendering one or more pixel sized fragments of the plurality of primitives located outside of the foveal region at a low resolution.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
wherein a plurality of gaze tracking information is generated at a frequency,
wherein a plurality of video frames is generated through a rendering pipeline at the frequency.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
transferring in the frame period the landing point to a buffer accessible by a rendering pipeline configured for rendering the video frame.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
predicting in the frame period a point in time for the gaze direction of the at least one eye to be pointed at the landing point; and
terminating rendering of a second video frame to be scanned out for display before the point in time.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
predicting in the frame period a point in time for the gaze direction of the at least one eye to be pointed at the landing point; and
rendering each of a plurality of pixels of a second video frame at low resolution, wherein the second video frame is to be scanned out for display before the point in time.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
detecting that the at least one eye of the user is experiencing a saccade; and
performing the predicting the landing point based on the saccade.

### Claim 9

A computer system comprising:
a processor;
memory coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to execute a method, comprising:
executing in a frame period an application to generate a plurality of primitives for a video frame;
receiving in the frame period gaze tracking information for at least one eye of a user;
predicting in the frame period a landing point on a display based on the gaze tracking information, wherein the landing point corresponds to a gaze direction of the at least one eye; and
rendering in the frame period the video frame based on the primitives and the landing point that is predicted.

### Claim 10 (depends on 9)

The computer system of claim 9, the method further comprising:
determining a foveal region in the video frame centered around the landing point that is predicted; and
rendering one or more pixel sized fragments of the plurality of primitives located in the foveal region at high resolution.

### Claim 11 (depends on 10)

The computer system of claim 10, the method further comprising:
rendering one or more pixel sized fragments of the plurality of primitives located outside of the foveal region at a low resolution.

### Claim 12 (depends on 9)

The computer system of claim 9, the method further comprising:
wherein a plurality of gaze tracking information is generated at a frequency,
wherein a plurality of video frames is generated through a rendering pipeline at the frequency.

### Claim 13 (depends on 9)

The computer system of claim 9, the method further comprising:
transferring in the frame period the landing point to a buffer accessible by a rendering pipeline configured for rendering the video frame.

### Claim 14 (depends on 9)

The computer system of claim 9, the method further comprising:
predicting in the frame period a point in time for the gaze direction of the at least one eye to be pointed at the landing point; and
terminating rendering of a second video frame to be scanned out for display before the point in time.

### Claim 15 (depends on 9)

The computer system of claim 9, the method further comprising:
predicting in the frame period a point in time for the gaze direction of the at least one eye to be pointed at the landing point; and
rendering each of a plurality of pixels of a second video frame at low resolution, wherein the second video frame is to be scanned out for display before the point in time.

### Claim 16 (depends on 9)

The computer system of claim 9, the method further comprising:
detecting that the at least one eye of the user is experiencing a saccade; and
performing the predicting the landing point based on the saccade.

### Claim 17

A non-transitory computer-readable medium storing a computer program for performing a method, the computer-readable medium comprising:
program instructions for executing in a frame period an application to generate a plurality of primitives for a video frame;
program instructions for receiving in the frame period gaze tracking information for at least one eye of a user;
program instructions for predicting in the frame period a landing point on a display based on the gaze tracking information, wherein the landing point corresponds to a gaze direction of the at least one eye; and
program instructions for rendering in the frame period the video frame based on the primitives and the landing point that is predicted.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, further comprising:
program instructions for determining a foveal region in the video frame centered around the landing point that is predicted;
program instructions for rendering one or more pixel sized fragments of the plurality of primitives located in the foveal region at high resolution; and
program instructions for rendering one or more pixel sized fragments of the plurality of primitives located outside of the foveal region at a low resolution.

### Claim 19 (depends on 17)

The non-transitory computer-readable medium of claim 17, further comprising:
program instructions for predicting in the frame period a point in time for the gaze direction of the at least one eye to be pointed at the landing point; and
program instructions for terminating rendering of a second video frame to be scanned out for display before the point in time.

### Claim 20 (depends on 17)

The non-transitory computer-readable medium of claim 17, further comprising:
program instructions for detecting that the at least one eye of the user is experiencing a saccade; and
program instructions for performing the predicting the landing point based on the saccade.
