# US 2023/0067584 A1 · Adaptive Quantization Matrix for Extended Reality Video Encoding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0067584 A1
- Document ID: US20230067584A1
- Status: Published application
- Kind: A1
- Application number: 17/821,981
- Filed: 2022-08-24
- Published: 2023-03-02
- Earliest claimed: 2021-08-27
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Yi Zhou
- Web page: https://patentgrep.com/patent/US20230067584A1

## CPC

- H04N 19/167: Position within a video image, e.g. region of interest [ROI]
- H04N 19/119: Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
- H04N 19/124: Quantisation
- H04N 19/14: Coding unit complexity, e.g. amount of activity or edge presence estimation (H04N19/146 takes precedence)
- H04N 19/17: the unit being an image region, e.g. an object
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Encoding an extended-reality (XR) video frame may include obtaining an XR video frame comprising a background image and a virtual object; obtaining, from an image renderer, a first region of the background image over which the virtual object is overlaid; dividing the XR video frame into a virtual region and a real region, wherein the virtual region comprises the first region of the background image and the virtual object and the real region comprises a second region of the background image; determining, for the virtual region, a corresponding first quantization parameter based on an initial quantization parameter associated with virtual regions; determining, for the real region, a corresponding second quantization parameter based on an initial quantization parameter associated with real regions; and encoding the virtual region based on the corresponding first quantization parameter and the real region based on the corresponding second quantization parameter.

## Claims

### Claim 1

A method for encoding an extended-reality (XR) video frame, comprising:
obtaining an XR video frame comprising a background image and a virtual object overlaying at least a portion of the background image;
dividing the XR video frame into a virtual region and a real region, wherein the virtual region comprises at least a portion of the virtual object, and wherein the real region comprises a region of the background image separate from the virtual region;
determining, for the virtual region, a corresponding first quantization parameter based on an initial quantization parameter associated with virtual regions;
determining, for the real region, a corresponding second quantization parameter based on an initial quantization parameter associated with real regions; and
encoding the virtual region based on the corresponding first quantization parameter and the real region based on the corresponding second quantization parameter.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining, for the virtual region, the corresponding first quantization parameter is further based on a threshold upper limit associated with virtual regions and a threshold lower limit associated with virtual regions.

### Claim 3 (depends on 1)

The method of claim 1, wherein determining, the real region, the corresponding second quantization parameter is further based on a threshold upper limit associated with real regions and a threshold lower limit associated with real regions.

### Claim 4 (depends on 1)

The method of claim 1, wherein:
dividing the XR video frame further comprises dividing the XR video frame into a medial region;
the virtual region comprises at least a portion of the virtual object that satisfies a first complexity criterion;
the real region comprises a first portion of the region of the background image separate from the virtual region, wherein the first portion of the region fails to satisfy a second complexity criterion;
the at least one medial region comprises at least one of: (i) a portion of the at least one virtual object that fails to satisfy the first complexity criterion, and (ii) a second portion of the region of the background image separate from the virtual region, wherein the second portion of the region satisfies the second complexity criterion; and
the method further comprising:
determining, for the medial region, a corresponding third quantization parameter based on an initial quantization parameter associated with medial regions; and
encoding the medial region based on the corresponding third quantization parameter.

### Claim 5 (depends on 4)

The method of claim 4, wherein determining, for the medial region, the corresponding third quantization parameter is further based on a threshold upper limit associated with medial regions and a threshold lower limit associated with medial regions.

### Claim 6 (depends on 1)

The method of claim 1, further comprising obtaining an input indicative of an area of focus via a gaze-tracking user interface, wherein dividing the XR video frame is based at least in part on the area of focus.

### Claim 7

A non-transitory computer readable medium, comprising computer code executable by at least one processor to:
obtain an extended reality (XR) video frame comprising a background image and a virtual object overlaying at least a portion of the background image;
divide the XR video frame into a virtual region and a real region, wherein the virtual region comprises at least a portion of the virtual object, and wherein the real region comprises a region of the background image separate from the virtual region;
determine, for the virtual region, a corresponding first quantization parameter based on an initial quantization parameter associated with virtual regions;
determine, for the real region, a corresponding second quantization parameter based on an initial quantization parameter associated with real regions; and
encode the virtual region based on the corresponding first quantization parameter and the real region based on the corresponding second quantization parameter.

### Claim 8 (depends on 7)

The non-transitory computer readable medium of claim 7, wherein the computer readable code to determine, for the virtual region, the corresponding first quantization parameter further comprises computer readable code to determine the corresponding first quantization parameter further based on a threshold upper limit associated with virtual regions and a threshold lower limit associated with virtual regions.

### Claim 9 (depends on 7)

The non-transitory computer readable medium of claim 7, wherein the computer readable code to determine, for the real region, the corresponding second quantization parameter further comprises computer readable code to determine the corresponding second quantization parameter further based on a threshold upper limit associated with real regions and a threshold lower limit associated with real regions.

### Claim 10 (depends on 7)

The non-transitory computer readable medium of claim 7, wherein the initial quantization parameter associated with virtual regions is smaller than the initial quantization parameter associated with real regions.

### Claim 11 (depends on 7)

The non-transitory computer readable medium of claim 7, wherein the computer readable media further comprises computer readable code executable by the at least one processor to:
for the virtual region:
determine a corresponding first region size based on an initial region size associated with virtual regions; and
divide the virtual region into one or more additional virtual regions based on the corresponding first region size;
for the real regions:
determine a corresponding second region size based on an initial region size associated with real regions; and
divide the real region into one or more additional real regions based on the corresponding second region size.

### Claim 12 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein the initial region size associated with virtual regions is smaller than the initial region size associated with real regions.

### Claim 13 (depends on 7)

The non-transitory computer readable medium of claim 7, wherein:
the computer readable code to divide the XR video frame further comprises computer readable code to divide the XR video frame into a medial region;
the virtual region comprises at least a portion of the virtual object that satisfies a first complexity criterion;
the real region comprises a first portion of the region of the background image separate from the virtual region, wherein the first portion of the region fails to satisfy a second complexity criterion; and
the medial region comprises at least one of: (i) a portion of the virtual object that fails to satisfy the first complexity criterion, and (ii) a second portion of the region of the background image separate from the virtual region, wherein the second portion of the region satisfies the second complexity criterion; and
the non-transitory computer readable medium further comprises computer readable code executable by the at least one processor to:
determine, for the medial region, a corresponding third quantization parameter based on an initial quantization parameter associated with medial regions; and
encode the medial region based on the corresponding third quantization parameters.

### Claim 14 (depends on 13)

The non-transitory computer readable medium of claim 13, wherein the computer readable code to determine, for the medial region, the corresponding third quantization parameter further comprises computer readable code to determine the corresponding third quantization parameter further based on a threshold upper limit associated with medial regions and a threshold lower limit associated with medial regions.

### Claim 15 (depends on 13)

The non-transitory computer readable medium of claim 13, wherein the initial quantization parameter associated with medial regions is smaller than the initial quantization parameter associated with real regions and larger than the initial quantization parameter associated with virtual regions.

### Claim 16 (depends on 13)

The non-transitory computer readable medium of claim 13, wherein the computer readable media further comprises computer readable code executable by the at least one processor to:
for the virtual region:
determine a corresponding first region size based on an initial region size associated with virtual regions; and
divide the virtual region into one or more additional virtual regions based on the corresponding first region size;
for the real region:
determine a corresponding second region size based on an initial region size associated with real regions; and
divide the real region into one or more additional real regions based on the corresponding second region size; and
for the medial regions:
determine a corresponding third region size based on an initial region size associated with medial regions; and
divide the medial region into one or more additional medial regions based on the corresponding third region size.

### Claim 17 (depends on 16)

The non-transitory computer readable medium of claim 16, wherein the initial region size associated with medial regions is larger than the initial region size associated with virtual regions and smaller than the initial region size associated with real regions.

### Claim 18 (depends on 7)

The non-transitory computer readable medium of claim 7, wherein the computer readable media further comprises computer readable code executable by the at least one processor to obtain an input indicative of an area of focus via a gaze-tracking user interface, wherein the computer readable code to divide the XR video frame further comprises computer readable code to divide the XR video frame based at least in part on the area of focus.

### Claim 19

A device, comprising:
an image capturing device configured to capture a background image;
at least one processor; and
at least one computer readable media comprising computer readable code executable by the at least one processor to:
obtain an extended reality (XR) video frame comprising the background image and a virtual object overlaying at least a portion of the background image;
divide the XR video frame into a virtual region and a real region, wherein the virtual region comprises at least a portion of the virtual object, and wherein the real region comprises a region of the background image separate from the virtual region;
determine, for the virtual region, a corresponding first quantization parameter based on an initial quantization parameter associated with virtual regions;
determine, for the real region, a corresponding second quantization parameter based on an initial quantization parameter associated with real regions; and
encode the virtual region based on the corresponding first quantization parameter and the real region based on the corresponding second quantization parameter.

### Claim 20 (depends on 19)

The system of claim 19, wherein the computer readable code to determine, for the virtual region, the corresponding first quantization parameter further comprises computer readable code to determine the corresponding first quantization parameter further based on a threshold upper limit associated with virtual regions and a threshold lower limit associated with virtual regions.