# US 2023/0065862 A1 · Scalable coding of video and associated features

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0065862 A1
- Document ID: US20230065862A1
- Status: Published application
- Kind: A1
- Application number: 17/981,163
- Filed: 2022-11-04
- Published: 2023-03-02
- Earliest claimed: 2021-01-13
- Assignees: Huawei Technologies Co., Ltd.
- Applicants: Huawei Technologies Co., Ltd.
- Inventors: Alexander Alexandrovich Karabutov; Hyomin Choi; Ivan Bajic; Robert A. Cohen; Saeed Ranjbar Alvar; Sergey Yurievich Ikonin; Elena Alexandrovna Alshina; Yin Zhao
- Web page: https://patentgrep.com/patent/US20230065862A1

## CPC

- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- H04N 19/30: using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence)
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- H04N 19/172: the region being a picture, frame or field
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The present disclosure relates to scalable encoding and decoding of pictures. In particular, a picture is processed by one or more network layers of a trained module to obtain base layer features. Then, enhancement layer features are obtained, e.g. by a trained network processing in sample domain. The base layer features are for use in computer vision processing. The base layer features together with enhancement layer features are for use in picture reconstruction, e.g. for human vision. The base layer features and the enhancement layer features are coded in a respective base layer bitstream and an enhancement layer bitstream. Accordingly, a scalable coding is provided which supports computer vision processing and/or picture reconstruction.

## Claims

### Claim 1

An apparatus for encoding an input picture, the apparatus comprising:
a memory comprising instructions and processing circuitry configured to execute the instructions to cause the apparatus to:
generate base layer features of a latent space, wherein the generating of the base layer features includes processing the input picture with one or more base layer network layers of a trained network;
generate, based on the input picture, enhancement layer features for reconstructing the input picture; and
encode the base layer features into a base layer bitstream and the enhancement layer features into an enhancement layer bitstream.

### Claim 2 (depends on 1)

The apparatus according to claim 1, wherein the processing circuitry is further configured to execute the instructions to further cause the apparatus to:
generate the enhancement layer features of the latent space by processing the input picture with one or more enhancement layer network layers of the trained network; and
subdivide the features of the latent space into the base layer features and the enhancement layer features.

### Claim 3 (depends on 1)

The apparatus according to claim 1, wherein the processing circuitry is further configured to execute the instructions to further cause the apparatus to generate the enhancement layer features by:
reconstructing a base layer picture based on the base layer features; and
determining the enhancement layer features based on the input picture and the base layer picture.

### Claim 4 (depends on 3)

The apparatus according to claim 3, wherein the determining of the enhancement layer features is based on differences between the input picture and the base picture.

### Claim 5 (depends on 1)

The apparatus according to claim 1, wherein the input picture is a frame of a video, and the processing circuitry is configured to generate the base layer features and the enhancement layer features for a plurality of frames of the video.

### Claim 6 (depends on 1)

The apparatus according to claim 1, wherein the processing circuitry is further configured to execute the instructions to further cause the apparatus to multiplex the base layer features and the enhancement layer features into a bitstream per frame.

### Claim 7 (depends on 1)

The apparatus according to claim 1, wherein the processing circuitry is further configured to execute the instructions to further cause the apparatus to encrypt a portion of a bitstream including the enhancement layer features.

### Claim 8

An apparatus for processing a bitstream, the apparatus comprising:
a memory comprising instructions and a processing circuitry configured to execute the instructions to cause the apparatus to:
obtain a base layer bitstream including base layer features of a latent space and an enhancement layer bitstream including enhancement layer features;
extract from the base layer bitstream the base layer features; and
perform at least one out of:
computer-vision processing based on the base layer features; and
extracting the enhancement layer features from the enhancement layer bitstream and reconstructing a picture based on the base layer features and the enhancement layer features.

### Claim 9 (depends on 8)

The apparatus according to claim 8, wherein the processing circuitry is configured to execute the instructions to cause the apparatus to perform the extracting the enhancement layer features from the enhancement layer bitstream and reconstructing the picture based on the base layer features and the enhancement layer features, and the reconstructing of the picture includes:
combining the base layer features and the enhancement layer features; and
reconstructing the picture based on the combined features.

### Claim 10 (depends on 8)

The apparatus according to claim 8, wherein the processing circuitry is configured to execute the instructions to cause the apparatus to perform computer-vision processing based on the base layer features, and the computer-vision processing includes processing of the base layer features by one or more network layers of a first trained subnetwork.

### Claim 11 (depends on 8)

The apparatus according to claim 8, wherein the processing circuitry is configured to execute the instructions to cause the apparatus to perform the extracting the enhancement layer features from the enhancement layer bitstream and reconstructing the picture based on the base layer features and the enhancement layer features, and the reconstructing of the picture includes processing the combined features by one or more network layers of a second trained subnetwork different from the first trained subnetwork.

### Claim 12 (depends on 8)

The apparatus according to claim 8, wherein the processing circuitry is configured to execute the instructions to cause the apparatus to perform the extracting the enhancement layer features from the enhancement layer bitstream and reconstructing the picture based on the base layer features and the enhancement layer features, and the reconstructing of the picture includes:
reconstructing a base layer picture based on the base layer features; and
adding the enhancement layer features to the base layer picture.

### Claim 13 (depends on 12)

The apparatus according to claim 12, wherein the enhancement layer features are based on differences between an encoder-side input picture and the base layer picture.

### Claim 14 (depends on 8)

The apparatus according to claim 8, wherein the processing circuitry is configured to execute the instructions to cause the apparatus to perform the extracting the enhancement layer features from the enhancement layer bitstream and reconstructing the picture based on the base layer features and the enhancement layer features, and the reconstructed picture is a frame of a video, and the base layer features and the enhancement layer features are for a plurality of frames of the video.

### Claim 15 (depends on 8)

The apparatus according to claim 8, wherein the processing circuitry is further configured to execute the instructions to further cause the apparatus to de-multiplex the base layer features and the enhancement layer features from a bitstream per frame.

### Claim 16 (depends on 8)

The apparatus according to claim 8, wherein the processing circuitry is further configured to execute the instructions to further cause the apparatus to decrypt a portion of a bitstream including the enhancement layer features.

### Claim 17

A method for encoding an input picture, the method comprising:
generating base layer features of a latent space, wherein the generating of the base layer features includes processing the input picture with one or more base layer network layers of a trained network;
generating, based on the input picture, enhancement layer features for reconstructing the input picture; and
encoding the base layer features into a base layer bitstream and the enhancement layer features into an enhancement layer bitstream.

### Claim 18 (depends on 17)

The method according to claim 17, the method further comprising:
generating the enhancement layer features of the latent space by processing the input picture with one or more enhancement layer network layers of the trained network; and
subdividing the features of the latent space into the base layer features and the enhancement layer features.

### Claim 19 (depends on 17)

The method according to claim 17, wherein the generating the enhancement layer features comprises:
reconstructing a base layer picture based on the base layer features; and
determining the enhancement layer features based on the input picture and the base layer picture.

### Claim 20 (depends on 17)

The method according to claim 17, wherein the determining of the enhancement layer features is based on differences between the input picture and the base picture.
