# US 12,015,801 B2 · Systems and methods for streaming extensions for video encoding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,015,801 B2
- Document ID: US12015801B2
- Status: Granted
- Kind: B2
- Application number: 17/589,641
- Filed: 2022-01-31
- Published: 2023-03-16
- Granted: 2024-06-18
- Earliest claimed: 2021-09-13
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Sorin C Cismas; Ganesh G Yadav
- Web page: https://patentgrep.com/patent/US12015801B2

## CPC

- H04N 19/119: Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
- H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards
- H04N 19/164: Feedback from the receiver or from the transmission channel
- H04N 19/174: the region being a slice, e.g. a line of blocks or a group of blocks
- H04N 19/184: the unit being bits, e.g. of the compressed video stream
- H04N 19/186: the unit being a colour or a chrominance component
- H04N 19/88: involving rearrangement of data among different coding units, e.g. shuffling, interleaving, scrambling or permutation of pixel data or permutation of transform coefficient data among different blocks
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

This disclosure is directed to systems and methods of streaming extensions for video encoding. The streaming extensions may enable the bitstream syntax for layered video data to be modified to reduce overhead for encoding. The bitstream syntax may be modified to enable variable length luma and chroma components, and enable the alignment between the layers and slice to be bit aligned to enable increased granularity in image encoding, and to minimize overhead between different elements within the layers.

## Claims

### Claim 1

An electronic device comprising:
a network interface configured to receive a stream of image data comprising multiple layers, wherein the stream of image data comprises streaming extensions that define a bitstream syntax for the stream of image data; and
video encoder hardware, wherein the video encoder hardware is operable to be reconfigured to use the bitstream syntax for the stream of image data according to the streaming extensions and to bit-align the multiple layers of the stream of image data.

### Claim 2 (depends on 1)

The electronic device of claim 1, wherein the bitstream syntax comprises variable length luma bit sizes and chroma bit sizes.

### Claim 3 (depends on 1)

The electronic device of claim 1, wherein the multiple layers comprise scanned discrete cosine transform (DCT) coefficients coded within the multiple layers.

### Claim 4 (depends on 3)

The electronic device of claim 3, wherein the scanned DCT coefficients comprise separate DC quantization and AC quantization.

### Claim 5 (depends on 1)

The electronic device of claim 1, wherein the video encoder hardware is configured to encode one or more slices comprising the bitstream syntax in a vertical slice order.

### Claim 6 (depends on 1)

The electronic device of claim 1, wherein the stream of image data comprises one or more image frames that are split into multiple horizontal strips.

### Claim 7 (depends on 6)

The electronic device of claim 6, wherein a last horizontal strip within an image frame of the one or more image frames comprises a variable height.

### Claim 8 (depends on 6)

The electronic device of claim 6, wherein the video encoder hardware is configured to add one or more macroblocks to a right edge of a horizontal strip of the multiple horizontal strips, in response to determining a macroblock width of the horizontal strip of the multiple horizontal strips is below a threshold value.

### Claim 9

An electronic device comprising:
a network interface configured to receive a stream of image data, wherein the stream of image data comprises one or more layers; and
video encoder circuitry configured to:
determine the streaming image data comprises the one or more layers; and
reconfigure the video encoder circuitry to utilize bitstream syntax corresponding to one or more streaming extensions to bit-align the one or more layers of the stream of image data and to encode one or more horizontal strips of an image frame in a vertical order, at least in part by concatenating one or more bitstream sections of the stream of image data without byte boundary padding.

### Claim 10 (depends on 9)

The electronic device of claim 9, wherein the one or more horizontal strips comprise one or more tiles that correspond to horizontal and vertical sizes of the image frame that are multiples of sixteen pixels.

### Claim 11 (depends on 10)

The electronic device of claim 10, wherein each of the one or more tiles comprise coded slices that include coded layers within the slices.

### Claim 12 (depends on 9)

The electronic device of claim 9, wherein the video encoder circuitry is configured to add zero-bits to an end of an image slice to byte align the image slice.

### Claim 13 (depends on 9)

The electronic device of claim 9, wherein a plurality of luma and chroma data within an image slice is a variable length field in a range of five to sixteen bits.

### Claim 14

A tangible, non-transitory, machine-readable medium comprising instructions that, when executed by image processing circuitry, causes the image processing circuitry to perform operations comprising:
receiving a stream of image data comprising multiple layers, wherein the stream of image data comprises streaming extensions that define a bitstream syntax for the stream of image data; and
sending the stream of image data comprising the streaming extensions to encoder hardware that is operable to bit-align the multiple layers of the stream of image data.

### Claim 15 (depends on 14)

The tangible, non-transitory, machine-readable medium of claim 14, wherein the multiple layers comprise scanned discrete cosine transform (DCT) coefficients distributed within the multiple layers.

### Claim 16 (depends on 15)

The tangible, non-transitory, machine-readable medium of claim 15, wherein the scanned DCT coefficients comprise separate DC quantization and AC quantization.

### Claim 17 (depends on 15)

The tangible, non-transitory, machine-readable medium of claim 15, wherein one or more DCT coefficients comprise a base layer of the multiple layers.

### Claim 18 (depends on 14)

The tangible, non-transitory, machine-readable medium of claim 14, wherein the operations comprise determining one or more quantization matrices to use for encoding, based on tile header syntax of the bitstream syntax corresponding to the image data.

### Claim 19 (depends on 14)

The tangible, non-transitory, machine-readable medium of claim 14, wherein the operations comprise identifying a layer order of the multiple layers based on a layer ID syntax of the bitstream syntax corresponding to the image data.

### Claim 20 (depends on 14)

The tangible, non-transitory, machine-readable medium of claim 14, wherein the bitstream syntax comprises variable length luma bit sizes and chroma bit sizes.