# US 2024/0403999 A1 · Mixed reality recording of foveated display content systems and methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0403999 A1
- Document ID: US20240403999A1
- Status: Published application
- Kind: A1
- Application number: 18/325,434
- Filed: 2023-05-30
- Published: 2024-12-05
- Earliest claimed: 2023-05-30
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Jim C. Chou; Yung-Chin Chen
- Web page: https://patentgrep.com/patent/US20240403999A1

## CPC

- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/147: using display panels
- G06T 7/11: Region-based segmentation
- G06T 2207/20132: Image cropping
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system may include a display for displaying an image frame that is divided into regions having respective resolutions based on display image data. The system may also include image processing circuitry to generate the display image data based on multi-resolution image data and generate record image data based on the multi-resolution image data. Generating the record image data may include obtaining boundary data indicative of locations of boundaries between the regions resampling the multi-resolution image data based on the boundary data. Resampling the multi-resolution image data may include performing a first resampling on a first portion of the multi-resolution image data corresponding to a first region and performing a second resampling, different from the first resampling, on a second portion of the multi-resolution image data corresponding to a second region.

## Claims

### Claim 1

A system comprising:
an electronic display comprising a plurality of pixels and configured to display an image frame, the image frame having a plurality of resolutions based on display image data, wherein the image frame is divided into a plurality of regions having respective resolutions of the plurality of resolutions; and
image processing circuitry configured to generate the display image data based on multi-resolution image data of the image frame and generate record image data based on the multi-resolution image data of the image frame, wherein generating the record image data comprises:
determining boundary data indicative of locations of boundaries between the plurality of regions; and
resampling the multi-resolution image data based on the boundary data, wherein resampling the multi-resolution image data comprises performing a first resampling on a first portion of the multi-resolution image data corresponding to a first region of the plurality of regions and performing a second resampling, different from the first resampling, on a second portion of the multi-resolution image data corresponding to a second region of the plurality of regions.

### Claim 2 (depends on 1)

The system of claim 1, wherein generating the record image data comprises cropping the multi-resolution image data to a crop window, the crop window comprises less than an entirety of the image frame.

### Claim 3 (depends on 2)

The system of claim 2, wherein the crop window is centered about a focal point of a user eye gaze on the electronic display, wherein the locations of the boundaries between the plurality of regions are adjustable and based on the focal point.

### Claim 4 (depends on 2)

The system of claim 2, wherein the crop window excludes a third portion of the multi-resolution image data corresponding to a third region of the plurality of regions having a respective resolution, of the plurality of resolutions, less than a threshold resolution.

### Claim 5 (depends on 1)

The system of claim 1, wherein performing the first resampling comprises resampling the first portion of the multi-resolution image data by a first factor in a first direction and downsampling the first portion of the multi-resolution image data by a second factor in a second direction, and wherein performing the second resampling comprises resampling the second portion of the multi-resolution image data by the first factor in the first direction and not downsampling the second portion of the multi-resolution image data by the second factor in the second direction.

### Claim 6 (depends on 5)

The system of claim 5, wherein performing the first resampling comprises upsampling the first portion of the multi-resolution image data by the first factor in the first direction.

### Claim 7 (depends on 1)

The system of claim 1, wherein the record image data comprises a consistent resolution for the image frame.

### Claim 8 (depends on 7)

The system of claim 7, wherein the consistent resolution is the same as a respective resolution of the plurality of resolutions of a third region of the plurality of regions.

### Claim 9 (depends on 1)

The system of claim 1, wherein generating the record image data comprises transforming a color space of the multi-resolution image data.

### Claim 10 (depends on 9)

The system of claim 9, wherein generating the record image data comprises converting the color space of the multi-resolution image data to a chromatic color space.

### Claim 11 (depends on 1)

The system of claim 1, wherein generating the record image data comprises gamma encoding the multi-resolution image data.

### Claim 12

Image processing circuitry comprising:
processing block circuitry configured to generate display image data based on an image frame of multi-resolution image data, wherein the image frame is divided into a plurality of regions having respective resolutions of a plurality of resolutions; and
record block circuitry configured to generate record image data having a consistent resolution based on the image frame of the multi-resolution image data by:
determining a cropped region of the image frame of the multi-resolution image data corresponding to the record image data, wherein the cropped region excludes a portion of the multi-resolution image data corresponding to a region, of the plurality of regions, having less than a threshold resolution; and
resampling the multi-resolution image data to the consistent resolution.

### Claim 13 (depends on 12)

The image processing circuitry of claim 12, wherein resampling the multi-resolution image data comprises resampling the multi-resolution image data by performing a first resampling on a second portion of the multi-resolution image data corresponding to a second region of the plurality of regions and performing a second resampling, different from the first resampling, on a third portion of the multi-resolution image data corresponding to a third region of the plurality of regions.

### Claim 14 (depends on 12)

The image processing circuitry of claim 12, wherein the consistent resolution comprises a lowest resolution of the cropped region.

### Claim 15 (depends on 12)

The image processing circuitry of claim 12, wherein the record block circuitry is configured to generate the record image data by reducing a precision of the multi-resolution image data.

### Claim 16 (depends on 15)

The image processing circuitry of claim 15, wherein the record block circuitry is configured to generate the record image data by downsampling and interleaving a first chromatic channel of the multi-resolution image data and a second chromatic channel of the multi-resolution image data and maintaining a luma channel of the multi-resolution image data.

### Claim 17

A non-transitory machine-readable medium comprising instructions, wherein, when executed by one or more processors, the instructions cause the one or more processors to perform operations or to control image processing circuitry to perform the operations, wherein the operations comprise:
determining boundary data indicative of locations of boundaries between a plurality of regions that define areas of different content resolutions of an image frame of multi-resolution image data;
generating record image data consisting of a portion of the image frame by resampling the multi-resolution image data to a consistent resolution across the portion of the image frame based on the boundary data; and
storing the record image data in a non-transitory memory, transmitting the record image data to an electronic display, or both.

### Claim 18 (depends on 17)

The non-transitory machine-readable medium of claim 17, wherein resampling the multi-resolution image data comprises performing a first resampling on a first portion of the multi-resolution image data corresponding to a first region of the plurality of regions and performing a second resampling, different from the first resampling, on a second portion of the multi-resolution image data corresponding to a second region of the plurality of regions.

### Claim 19 (depends on 18)

The non-transitory machine-readable medium of claim 18, wherein the first resampling comprises a downsampling in a first direction and no resampling in a second direction.

### Claim 20 (depends on 17)

The non-transitory machine-readable medium of claim 17, wherein the operations comprise generating display image data, based on the multi-resolution image data, to be displayed on a second electronic display, wherein the second electronic display comprises a foveated display, and wherein the locations of the boundaries of the plurality of regions are set based on a focal point of a viewer's gaze on the electronic display.
