# US 2024/0404009 A1 · Foveated display image enhancement systems and methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0404009 A1
- Document ID: US20240404009A1
- Status: Published application
- Kind: A1
- Application number: 18/326,630
- Filed: 2023-05-31
- Published: 2024-12-05
- Earliest claimed: 2023-05-31
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Yun Gong; Jim C. Chou; Guy Cote; Haiyan He; Michael J. Rockwell
- Web page: https://patentgrep.com/patent/US20240404009A1

## CPC

- G06T 5/20: using local operators
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/13: Edge detection
- G06T 11/00: Two-dimensional [2D] image generation
- G09G 3/2096: {Details of the interface to the display terminal specific for a flat panel (suitable for both CRT and flat panel G09G5/006; specific for a CRT G09G1/167)}
- G06T 2207/10024: Color image
- G06T 2207/20192: Edge enhancement; Edge preservation
- G06T 2207/20221: Image fusion; Image merging
- G09G 2340/0407: Resolution change, inclusive of the use of different resolutions for different screen areas
- G09G 2340/10: Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
- G09G 2354/00: Aspects of interface with display user
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

A system may include a display for displaying an image frame that is divided into regions having respective resolutions based on display image data. The system may also include image processing circuitry to generate the display image data based on multi-resolution image data of the image frame. Generating the display image data may include determining an enhancement to be applied to a portion of the multi-resolution image data and adjusting the determined enhancement to be applied to the portion of the multi-resolution image data based on boundary data associated with locations of boundaries between the regions.

## Claims

### Claim 1

A system comprising:
an electronic display comprising a plurality of pixels and configured to display an image frame, the image frame having a plurality of resolutions based on display image data, wherein the image frame is divided into a plurality of regions having respective resolutions of the plurality of resolutions; and
image processing circuitry configured to generate the display image data based on multi-resolution image data of the image frame, wherein generating the display image data comprises:
determining an enhancement to be applied to a portion of the multi-resolution image data; and
adjusting the enhancement to be applied to the portion of the multi-resolution image data based on boundary data associated with locations of boundaries between the plurality of regions.

### Claim 2 (depends on 1)

The system of claim 1, wherein generating the display image data comprises:
applying the adjusted enhancement to the portion of the multi-resolution image data to generate enhanced image data; and
blending the enhanced image data and additional image data of the image frame.

### Claim 3 (depends on 2)

The system of claim 2, wherein the additional image data comprises captured image data and the multi-resolution image data comprises rendered graphics image data.

### Claim 4 (depends on 1)

The system of claim 1, wherein determining the enhancement to be applied to the portion of the multi-resolution image data comprises performing an example-based enhancement, wherein performing the example-based enhancement for a pixel value of the portion of the multi-resolution image data comprises performing a comparison of a first pixel group comprising a first plurality of pixel values including the pixel value and a second pixel group comprising a second plurality of pixel values corresponding to same pixel position of the first plurality of pixel values, wherein the second plurality of pixel values are generated based at least in part on filtering of the first plurality of pixel values.

### Claim 5 (depends on 4)

The system of claim 4, wherein the comparison comprises a difference between a first weighted average of a first subset of the first plurality of pixel values of the first pixel group and a second weighted average of a second subset of the second plurality of pixel values of the second pixel group.

### Claim 6 (depends on 1)

The system of claim 1, wherein the enhancement to be applied to a pixel value of a channel of the portion of the multi-resolution image data comprises an increase or a decrease to the pixel value.

### Claim 7 (depends on 1)

The system of claim 1, wherein adjusting the enhancement to be applied to a pixel value of the portion of the multi-resolution image data comprises interpolating an enhancement factor for the pixel value from a plurality of enhancement scaling factors and applying the enhancement factor to the enhancement to be applied to the pixel value.

### Claim 8 (depends on 7)

The system of claim 7, wherein the plurality of enhancement scaling factors corresponds to a plurality of enhancement grid points disposed at corresponding grid point positions relative to the image frame, the corresponding grid point positions based on the boundary data, wherein interpolation of the enhancement factor is based on a pixel location of the pixel value in relation to the corresponding grid point positions.

### Claim 9 (depends on 1)

The system of claim 1, wherein the multi-resolution image data comprises a chromatic color space format comprising a luma channel, and wherein determining the enhancement to be applied to the portion of the multi-resolution image data comprises determining a value change for a pixel value of the luma channel of the portion of the multi-resolution image data.

### Claim 10 (depends on 1)

The system of claim 1, wherein determining the enhancement to be applied to the portion of the multi-resolution image data comprises applying one or more peaking filters to a luma channel of the portion of the multi-resolution image data.

### Claim 11 (depends on 1)

The system of claim 1, wherein a center region of the plurality of regions is centered about a focal point of a user eye gaze on the electronic display, wherein the locations of the boundaries between the plurality of regions are adjustable and based on the focal point.

### Claim 12

Image processing circuitry comprising:
enhancement circuitry configured to generate enhancement data indicative of gains to be applied to corresponding pixel values of an image frame of multi-resolution image data, wherein the image frame is divided into a plurality of regions having respective resolutions of a plurality of resolutions; and
blend circuitry configured to adjust the enhancement data based on boundary data associated with locations of boundaries between the plurality of regions.

### Claim 13 (depends on 12)

The image processing circuitry of claim 12, wherein the blend circuitry is configured to adjust the enhancement data for a pixel of interest based on a pixel location of the pixel relative to an enhancement grid, wherein grid point positions of the enhancement grid are based on the boundary data.

### Claim 14 (depends on 13)

The image processing circuitry of claim 13, wherein adjusting the enhancement data for the pixel of interest comprises:
interpolating an enhancement factor for the pixel of interest from a first subset of a plurality of enhancement scaling factors, wherein the plurality of enhancement scaling factors corresponds to a second subset of a plurality of enhancement grid points of the enhancement grid; and
applying the enhancement factor to the enhancement data for the pixel of interest.

### Claim 15 (depends on 14)

The image processing circuitry of claim 14, wherein enhancement grid points of the plurality of enhancement grid points further from a reference point correspond to lower enhancement scaling factors of the plurality of enhancement scaling factors.

### Claim 16 (depends on 15)

The image processing circuitry of claim 15, wherein the boundary data is based on the reference point, and wherein the reference point corresponds to a user eye gaze relative to an electronic display.

### Claim 17

A non-transitory machine-readable medium comprising instructions, wherein, when executed by one or more processors, the instructions cause the one or more processors to perform operations or to control image processing circuitry to perform the operations, wherein the operations comprise:
determining boundary data associated with locations of boundaries between a plurality of regions that define areas of different content resolutions of an image frame of multi-resolution image data;
determining an enhancement to be applied to a portion of the multi-resolution image data; and
adjusting the enhancement to be applied to the portion of the multi-resolution image data based on the boundary data.

### Claim 18 (depends on 17)

The non-transitory machine-readable medium of claim 17, wherein determining the enhancement to be applied comprises performing a tone-based enhancement, wherein the tone-based enhancement comprises correlating a color tone of the portion of the multi-resolution image data with a type of image content of a plurality of types of image content.

### Claim 19 (depends on 18)

The non-transitory machine-readable medium of claim 18, wherein different types of image content of the plurality of types of image content are associated with different enhancements.

### Claim 20 (depends on 17)

The non-transitory machine-readable medium of claim 17, wherein the operations comprise adjusting the enhancement to be applied based on values of alpha data indicative of how different portions of the multi-resolution image data are to be combined with other image data.
