# US 2017/0287446 A1 · Real-time user adaptive foveated rendering

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0287446 A1
- Document ID: US20170287446A1
- Status: Published application
- Kind: A1
- Application number: 15/086,645
- Filed: 2016-03-31
- Published: 2017-10-05
- Earliest claimed: 2016-03-31
- Assignees: Sony Computer Entertainment Inc.
- Applicants: Sony Computer Entertainment Inc.
- Inventors: Andrew Young; Jeffrey Roger Stafford
- Web page: https://patentgrep.com/patent/US20170287446A1

## CPC

- G09G 5/391: Resolution modifying circuits, e.g. variable screen formats
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/0101: {characterised by optical features (G02B27/0172 takes precedence)}
- G06F 1/3265: {Power saving in display device}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G02B 27/017: {Head mounted}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G09G 2340/0407: Resolution change, inclusive of the use of different resolutions for different screen areas
- Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE

## Abstract

Foveated rendering based on user gaze tracking may be adjusted to account for the realities of human vision. Gaze tracking error and state parameters may be determined from gaze tracking data representing a user's gaze with respect to one or more images presented to a user. Adjusted foveation data representing an adjusted size and/or shape of one or more regions of interest in one or more images to be subsequently presented to a user may be generated based on the one or more gaze tracking error or state parameters. Foveated image data representing one or more foveated images may be generated with the adjusted foveation data. The foveated images are characterized by level of detail within the one or more regions of interest and lower level of detail outside the one or more regions of interest. The foveated images may then be presented to the user.

## Claims

### Claim 1

A graphics processing method comprising:
obtaining gaze tracking data representing a user's gaze with respect to one or more images presented to the user;
determining one or more gaze tracking parameters from gaze tracking data, wherein the gaze tracking parameters include one or more gaze tracking error parameters;
generating adjusted foveation data representing an adjusted size and/or shape of one or more regions of interest in one or more images to be subsequently presented to the user based on the one or more gaze tracking parameters including the one or more gaze tracking error parameters;
generating foveated image data representing one or more foveated images using the adjusted foveation data, wherein the one or more foveated images are characterized by level of detail within the one or more regions of interest and lower level of detail outside the one or more regions of interest; and
presenting the one or more foveated images to the user.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining a rate of rotation of a user's eye with respect to one or more axes.

### Claim 3 (depends on 1)

The method of claim 1, wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining a rate of rotation of a user's eye with respect to one or more axes and determining an error in a fixation of the user's eye on a region of interest of the one or more regions of interest.

### Claim 4 (depends on 1)

The method of claim 1, wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining a rate of rotation of a user's eye with respect to one or more axes and determining whether the user's eye is moving in smooth pursuit.

### Claim 5 (depends on 1)

The method of claim 1, wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining a confidence interval regarding the current and/or predicted future gaze position.

### Claim 6 (depends on 1)

The method of claim 1, further comprising wherein the gaze tracking parameters further include gaze tracking state parameters.

### Claim 7 (depends on 6)

The method of claim 6, wherein determining one or more gaze tracking state parameters from the gaze tracking data includes determining the metrics of a blink of a user.

### Claim 8 (depends on 6)

The method of claim 6, wherein determining one or more gaze tracking state parameters from the gaze tracking data includes determining the metrics of a saccade of a user.

### Claim 9 (depends on 6)

The method of claim 6, wherein determining one or more gaze tracking state parameters from the gaze tracking data includes determining a transition in the gaze direction of a user as a result of a change in the depth of field of the presented image.

### Claim 10 (depends on 6)

The method of claim 6, wherein determining one or more gaze tracking state parameters from the gaze tracking data includes determining whether a user is color blind.

### Claim 11 (depends on 6)

The method of claim 6, wherein determining one or more gaze tracking state parameters from the gaze tracking data includes determining a rate of rotation of a user's eye with respect to one or more axes and determining the level of gaze stability of the user's eye.

### Claim 12 (depends on 6)

The method of claim 6, wherein determining one or more gaze tracking state parameters from the gaze tracking data includes determining a rate of rotation of a user's eye with respect to one or more axes and determining whether the user's eye movement is a precursor to the user's head movement.

### Claim 13 (depends on 1)

The method of claim 1, wherein the adjusted foveation data includes data characterizing a particular region of interest of the one or more regions of interest, wherein the data characterizing the particular region of interest includes a foveation region and a transition region, wherein the foveation region is characterized by a higher level of detail than the transition region and the transition region is characterized by a higher level of detail than regions of a corresponding image outside the particular region of interest.

### Claim 14 (depends on 12)

The method of claim 12, wherein generating adjusted foveation data representing an adjusted size and/or shape of one or more regions of interest includes adjusting a size of the transition region.

### Claim 15 (depends on 1)

The method of claim 1, wherein the adjusted foveation data includes data used to determine geometric density in the one or more regions of interest.

### Claim 16 (depends on 1)

The method of claim 1, wherein the adjusted foveation data includes data used to determine pixel resolution in the one or more regions of interest.

### Claim 17 (depends on 1)

The method of claim 1, wherein the adjusted foveation data includes data used to determine pixel shading quality in the one or more regions of interest.

### Claim 18 (depends on 1)

The method of claim 1, further comprising creating a standard 2D compliant image from the foveated image data for presentation on additional displays.

### Claim 19

A system, comprising:
a processor;
a memory; and
computer-readable instructions embodied in the memory, the computer-readable instructions being configured to implement a graphics processing method when executed, the graphics processing method comprising:
obtaining gaze tracking data representing a user's gaze with respect to one or more images presented to the user;
determining one or more gaze tracking parameters from gaze tracking data, wherein the gaze tracking parameters include at least one gaze tracking error parameter or at least one gaze tracking state parameter;
generating adjusted foveation data representing an adjusted size and/or shape of one or more regions of interest in one or more images to be subsequently presented to the user based on the one or more gaze tracking parameters;
generating foveated image data representing one or more foveated images using the adjusted foveation data, wherein the one or more foveated images are characterized by level of detail within the one or more regions of interest and lower level of detail outside the one or more regions of interest; and
presenting the one or more foveated images to the user.

### Claim 20

A non-transitory computer-readable medium having computer-readable instructions embodied therein, the computer-readable instructions being configured to implement a graphics processing method when executed, the graphics processing method comprising:
obtaining gaze tracking data representing a user's gaze with respect to one or more images presented to the user;
determining one or more gaze tracking parameters from gaze tracking data, wherein the gaze tracking parameters include at least one gaze tracking error parameter or at least one gaze tracking state parameter;
generating adjusted foveation data representing an adjusted size and/or shape of one or more regions of interest in one or more images to be subsequently presented to the user based on the one or more gaze tracking error parameters;
generating foveated image data representing one or more foveated images using the adjusted foveation data, wherein the one or more foveated images are characterized by level of detail within the one or more regions of interest and lower level of detail outside the one or more regions of interest; and
presenting the one or more foveated images to the user.
