# US 2020/0288149 A1 · Scene Classification and Learning for Video Compression

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0288149 A1
- Document ID: US20200288149A1
- Status: Published application
- Kind: A1
- Application number: 16/291,076
- Filed: 2019-03-04
- Published: 2020-09-10
- Earliest claimed: 2019-03-04
- Assignees: Comcast Cable Communications, LLC
- Applicants: Comcast Cable Communications, LLC
- Inventors: Weidong Mao; Alexander Giladi
- Web page: https://patentgrep.com/patent/US20200288149A1

## CPC

- H04N 19/176: the region being a block, e.g. a macroblock
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- H04N 19/137: Motion inside a coding unit, e.g. average field, frame or block difference
- H04N 19/166: concerning the amount of transmission errors, e.g. bit error rate [BER]
- H04N 19/517: by encoding
- H04N 21/64738: {Monitoring network characteristics, e.g. bandwidth, congestion level (data switched network analysis H04L41/14; monitoring functioning in data switched networks H04L43/0817; flow control in packet networks H04L47/10)}
- H04N 19/115: Selection of the code volume for a coding unit prior to coding
- H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635)
- H04N 19/154: Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion (use of rate-distortion criteria H04N19/147)
- H04N 19/162: User input
- H04N 19/17: the unit being an image region, e.g. an object
- H04N 19/177: the unit being a group of pictures [GOP]
- H04N 19/179: the unit being a scene or a shot
- H04N 19/192: the adaptation method, adaptation tool or adaptation type being iterative or recursive
- H04N 19/196: being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters (processing of motion vectors H04N19/513)
- H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence)
- H04N 19/61: in combination with predictive coding
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems, apparatuses, and methods are described for encoding a scene of media content based on visual elements of the scene. A scene of media content may comprise one or more visual elements, such as individual objects in the scene. Each visual element may be classified based on, for example, the motion and/or identity of the visual element. Based on the visual element classifications, scene encoder parameters and/or visual element encoder parameters for different visual elements may be determined. The scene may be encoded using the scene encoder parameters and/or the visual element encoder parameters.

## Claims

### Claim 1

A method comprising:
determining, by a computing device, a plurality of visual elements of a frame of media content by recognizing one or more image objects in the frame of media content;
selecting, based on the one or more image objects in the frame of media content, different visual element encoder parameters corresponding to at least some different visual elements of the plurality of visual elements;
determining, based on the different visual element encoder parameters, scene encoder parameters for encoding a plurality of frames of the media content; and
causing the different visual elements to be encoded using the different visual element encoder parameters and the scene encoder parameters.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining, using perceptual video quality metrics, a quality of the encoded different visual elements;
determining, based on the quality, new visual element encoder parameters; and
causing the different visual elements to be encoded using the new visual element encoder parameters.

### Claim 3 (depends on 1)

The method of claim 1, wherein selecting the different visual element encoder parameters comprises:
determining that a first visual element of the plurality of visual elements corresponds to a first image object of the one or more image objects;
determining that a second visual element of the plurality of visual elements corresponds to a second image object of the one or more image objects;
determining, based on the first object, that the first visual element should be encoded at a first bitrate; and
determining, based on the second object, that the second element should be encoded at a second bitrate different from the first bitrate.

### Claim 4 (depends on 1)

The method of claim 1, wherein causing the different visual elements to be encoded based on the different visual element encoder parameters and the scene encoder parameters comprises:
determining a first portion, of the media content corresponding to a first visual element of the plurality of visual elements;
determining a second portion, of the media content not corresponding to the first visual element;
encoding the first portion using the different visual element encoder parameters; and
encoding the second portion using the scene encoder parameters.

### Claim 5 (depends on 1)

The method of claim 1, wherein the different visual elements of the plurality of visual elements are associated with different levels of fidelity.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
determining a predicted motion of a first visual element of the plurality of visual elements, wherein the selecting the different visual element encoder parameters is based on the predicted motion.

### Claim 7 (depends on 1)

The method of claim 1, wherein selecting the different visual element encoder parameters is based on respective locations of the different visual elements.

### Claim 8 (depends on 1)

The method of claim 1, wherein causing the different visual elements to be encoded based on the different visual element encoder parameters and the scene encoder parameters comprises:
determining one or more portions of the media content;
determining, for each of the one or more portions, portion encoder parameters based on the different visual element encoder parameters and the scene encoder parameters; and
causing the different visual elements to be encoded based on the portion encoder parameters.

### Claim 9 (depends on 1)

The method of claim 1, wherein recognizing one or more image objects in the frame of media content comprises:
comparing a shape of a first visual element of the plurality of visual elements with a plurality of shapes corresponding to the one or more image objects.

### Claim 10 (depends on 1)

The method of claim 1, wherein selecting the different visual element encoder parameters is based on a metadata corresponding to the media content.

### Claim 11

A method comprising:
determining, by a computing device and for media content, a plurality of visual elements of a frame of media content by performing object recognition on one or more portions of the frame of media content, wherein each visual element of the plurality of visual elements comprises a plurality of pixels of the frame;
selecting different visual element encoder parameters corresponding to different visual elements of the plurality of visual elements;
determining different encoder regions of the media content;
selecting, for each of the different encoder regions of the media content and based on the different visual element encoder parameters, encoder region encoding parameters; and
causing the different visual elements to be encoded based on the encoder region encoding parameters.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
determining, using perceptual video quality metrics, a quality of the encoded different visual elements;
determining, based on the quality, new visual element encoder parameters; and
causing the different visual elements to be encoded using the new visual element encoder parameters.

### Claim 13 (depends on 11)

The method of claim 11, wherein performing object recognition on one or more portions of the frame of media content comprises:
identifying one or more objects in a first portion of the one or more portions of the frame of media content based on a shape of the one or more objects.

### Claim 14 (depends on 11)

The method of claim 11, wherein each of the encoder region encoding parameters specifies a different bitrate used to encode one or more macroblocks of each of the different encoder regions.

### Claim 15 (depends on 11)

The method of claim 11, further comprising:
determining a predicted motion of a first visual element of the plurality of visual elements, wherein the selecting the different visual element encoder parameters is based on the predicted motion.

### Claim 16

A method comprising:
storing, prior to encoding media content, information indicating different encoding bitrates corresponding to different image objects;
receiving, by a computing device, the media content;
recognizing one or more image objects in the media content;
allocating, for each object of the one or more image objects and based on the information, different encoding bitrates to different image objects of the one or more image objects in the media content; and
causing the media content to be encoded based on the allocated different encoding bitrates.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
determining, using perceptual video quality metrics, a quality of a second portion of the encoded media content;
allocating, based on the quality, second encoding bitrates to different image objects of the one or more image objects in the media content; and
causing the media content to be encoded based on the second encoding bitrates.

### Claim 18 (depends on 16)

The method of claim 16, wherein a first image object of the one or more image objects corresponds to a plurality of macroblocks.

### Claim 19 (depends on 16)

The method of claim 16, wherein recognizing one or more image objects in the media content comprises:
identifying a first object of the one or more image objects based on a shape of the first object.

### Claim 20 (depends on 16)

The method of claim 16, wherein a sum of the allocated different encoding bitrates is higher than an average bit rate of the media content.