# US 2019/0246111 A1 · Techniques for modeling temporal distortions when predicting perceptual video quality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0246111 A1
- Document ID: US20190246111A1
- Status: Published application
- Kind: A1
- Application number: 15/890,709
- Filed: 2018-02-07
- Published: 2019-08-08
- Earliest claimed: 2018-02-07
- Assignees: Netflix, Inc.
- Applicants: Netflix, Inc.
- Inventors: Zhi LI; Christos BAMPIS
- Web page: https://patentgrep.com/patent/US20190246111A1

## CPC

- H04N 19/154: Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion (use of rate-distortion criteria H04N19/147)
- G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines
- G06N 3/045: Combinations of networks
- G06N 3/09: Supervised learning
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N 20/20: Ensemble learning
- H04N 17/004: {for digital television systems}
- H04N 19/91: Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
- H04N 21/235: Processing of additional data, e.g. scrambling of additional data or processing content descriptors
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

In various embodiments, a prediction application computes a quality score for re-constructed visual content that is derived from visual content. The prediction application generates a frame difference matrix based on two frames included in the re-constructed video content. The prediction application then generates a first entropy matrix based on the frame difference matrix and a first scale. Subsequently, the prediction application computes a first value for a first temporal feature based on the first entropy matrix and a second entropy matrix associated with both the visual content and the first scale. The prediction application computes a quality score for the re-constructed video content based on the first value, a second value for a second temporal feature associated with a second scale, and a machine learning model that is trained using subjective quality scores. The quality score indicates a level of visual quality associated with streamed video content.

## Claims

### Claim 1

A computer-implemented method, comprising:
generating a frame difference matrix based on a first frame and a second frame that are included in re-constructed video content;
computing a first entropy matrix based on the frame difference matrix and a first scale;
computing a first value for a first temporal feature based on the first entropy matrix and a second entropy matrix associated with both video content from which the re-constructed video content is derived and the first scale; and
computing a quality score for the re-constructed video content based on the first value, a second value for a second temporal feature associated with a second scale, and a machine learning model trained using a plurality of subjective quality scores, wherein the quality score indicates a level of visual quality associated with streamed video content.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein computing the first entropy matrix comprises:
performing one or more down-sampling operations on the frame difference matrix based on the first scale to generate a scaled frame difference matrix;
performing one or more spatial filtering operations on the scaled frame difference matrix to generate a local mean-subtracted matrix; and
performing one or more conditioning operations on the local mean-subtracted matrix.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein computing the first value for the first temporal feature comprises performing a subtraction operation between a first entropy value included in the first entropy matrix and a second entropy value included in the second entropy matrix.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein computing the quality score comprises:
determining a first frame quality score associated with the first frame based on a plurality of values for a plurality of temporal features, one or more values for one or more spatial features, and the machine learning model, wherein the plurality of values includes the first value and the second value; and
performing one or more temporal pooling operations between the first frame quality score and a second frame quality score associated with the second frame.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein performing the one or more temporal pooling operations comprises performing at least one of a linear low pass operation and a non-linear rank-order weighting operation on the first frame quality score and the second frame quality score.

### Claim 6 (depends on 4)

The computer implemented method of claim 4, further comprising computing the one or more values for the one or more spatial features based on the re-constructed video content.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein generating the frame difference matrix comprises, for each of a plurality of pixels, computing a luminance difference between a first luminance of the pixel in the first frame and a second luminance of the pixel in the second frame.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising training the machine learning model based on one or more machine learning algorithms and the plurality of subjective quality scores, wherein each subjective quality score included in the plurality of subjective quality scores is associated with different re-constructed test video content.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein the one or more machine learning algorithms comprises at least one of a support vector machine algorithm and an artificial neural network algorithm.

### Claim 10

A computer-readable storage medium including instructions that, when executed by a processor, cause the processor to perform the steps of:
generating a first scaled frame difference matrix based on a first frame, a second frame, and a first scale, wherein the first frame and the second frame are included in re-constructed video content;
generating a second scaled frame difference matrix based on a third frame, a fourth frame, and the first scale, wherein the third frame and the fourth frame are included in video content from which the re-constructed video content is derived;
computing a first entropy matrix based on the first scaled frame difference matrix;
computing a first value for a first temporal feature based on the first entropy matrix and a second entropy matrix associated with the second scaled frame difference matrix; and
computing a quality score for the re-constructed video content based on the first value, a second value for a second temporal feature associated with a second scale, and a machine learning model trained using a plurality of subjective quality scores, wherein the quality score indicates a level of visual quality associated with streamed video content.

### Claim 11 (depends on 10)

The computer-readable storage medium of claim 10, wherein computing the first entropy matrix comprises:
performing one or more spatial filtering operations on the first scaled frame difference matrix to generate a local mean-subtracted matrix; and
performing one or more conditioning operations on the local mean-subtracted matrix.

### Claim 12 (depends on 11)

The computer-readable storage medium of claim 11, wherein performing the one or more spatial filtering operations comprises applying a spatial isotropic Gaussian filter to the first scaled frame difference matrix.

### Claim 13 (depends on 10)

The computer-readable storage medium of claim 10, wherein computing the quality score comprises:
determining a first frame quality score associated with the first frame based on a plurality of values for a plurality of temporal features, one or more values for one or more spatial features, and the machine learning model, wherein the plurality of values includes the first value and the second value; and
performing one or more temporal pooling operations between the first frame quality score and a second frame quality score associated with the second frame.

### Claim 14 (depends on 13)

The computer-readable storage medium of claim 13, wherein performing the one or more temporal pooling operations comprises performing at least one of a linear low pass operation and a non-linear rank-order weighting operation on the first frame quality score and the second frame quality score.

### Claim 15 (depends on 13)

The computer-readable storage medium of claim 13, wherein the one or more spatial features comprise at least one of an additive impairment measure feature, a blind or referenceless image spatial quality evaluator feature, and a visual information fidelity feature.

### Claim 16 (depends on 10)

The computer-readable storage medium of claim 10, wherein generating the first scaled frame difference matrix comprises:
for each of a plurality of pixels, computing a luminance difference between a first luminance of the pixel in the first frame and a second luminance of the pixel in the second frame to generate a frame difference matrix; and
performing one or more down-sampling operations on the frame difference matrix based on the first scale.

### Claim 17 (depends on 10)

The computer-readable storage medium of claim 10, wherein a first subjective quality score included in the plurality of subjective quality scores is associated with one or more human-observed visual quality scores for re-constructed test video content.

### Claim 18 (depends on 10)

The computer-readable storage medium of claim 10, further comprising training the machine learning model based on the plurality of subjective quality scores and at least one of a support vector machine algorithm and an artificial neural network algorithm.

### Claim 19

A system, comprising:
a memory storing instructions; and
a processor that is coupled to the memory and, when executing the instructions, is configured to:
generate a frame difference matrix based on a first frame and a second frame that are included in re-constructed video content;
perform one or more down-sampling operations on the frame difference matrix based on a first scale to generate a first scaled frame difference matrix;
compute a first entropy matrix based on the first scaled frame difference matrix;
compute a first value for a first temporal feature based on the first entropy matrix and a second entropy matrix associated with both video content from which the re-constructed video content is derived and the first scale;
perform one or more down-sampling operations on the frame difference matrix based on a second scale to generate a second scaled frame difference matrix; and
compute a quality score for the re-constructed video content based on the first value, a second value for a second temporal feature associated with the second scaled frame difference matrix, and a machine learning model trained using a plurality of subjective quality scores, wherein the quality score indicates a level of visual quality associated with streamed video content.

### Claim 20 (depends on 19)

The system of claim 19, wherein the processor is configured to compute the first entropy matrix by:
performing one or more spatial filtering operations on the first scaled frame difference matrix to generate a local mean-subtracted matrix; and
performing one or more conditioning operations on the local mean-subtracted matrix.
