# US 8,279,259 B2 · Mimicking human visual system in detecting blockiness artifacts in compressed video streams

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 8,279,259 B2
- Document ID: US8279259B2
- Status: Granted
- Kind: B2
- Application number: 12/566,590
- Filed: 2009-09-24
- Published: 2011-03-24
- Granted: 2012-10-02
- Earliest claimed: 2009-09-24
- Assignees: Microsoft Corporation
- Inventors: Walid S. I. Ali; Rony Ferzli
- Web page: https://patentgrep.com/patent/US8279259B2

## CPC

- H04N 7/152: {Multipoint control units therefor}
- H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635)
- H04N 19/154: Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion (use of rate-distortion criteria H04N19/147)
- H04N 19/164: Feedback from the receiver or from the transmission channel
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/865: {with detection of the former encoding block subdivision in decompressed video}
- H04N 19/186: the unit being a colour or a chrominance component
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A low complexity block-based, no-reference objective blockiness metric is provided that may be combined with other artifact metrics to measure overall quality of received video stream in a video conferencing application such that measures can be taken at the transmitter or in post-processing to enhance video quality. Prior knowledge of the blockiness boundaries may be used to reduce number of computations in determining the blockiness of a particular video frame.

## Claims

### Claim 1

A method to be executed at least in part in a computing device for providing a blockiness metric associated with received video quality in a video conference system, the method comprising:
receiving a decoded video frame at a post-processing module;
computing a blockiness metric for columns of the decoded video frame based on an autocorrelation value and a cross-correlation value of each column;
computing a blockiness metric for rows of the decoded video frame based on an autocorrelation and a cross-correlation of each row; and
combining the blockiness metric for the columns and the blockiness metric for the rows to obtain an overall blockiness metric for the decoded video frame.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
scaling the overall blockiness metric by applying a non-linear logistic function to replicate a behavior of Human Visual System (HSV).

### Claim 3 (depends on 2)

The method of claim 2, wherein the logistic function is based on an exponential model.

### Claim 4 (depends on 2)

The method of claim 2, wherein applying the logistic function comprises:
determining a maximum overall blockiness metric among a predefined number of decoded video frames;
determining an exponential model based on the determined maximum overall blockiness metric; and
applying the exponential model to the overall blockiness metrics of the predefined number of decoded video frames such that the overall blockiness metrics have defined lower and upper limits.

### Claim 5 (depends on 2)

The method of claim 2, further comprising:
employing the overall blockiness metric to adjust at least one operational parameter of the video conference system for automatically reducing blockiness artifacts in received video.

### Claim 6 (depends on 2)

The method of claim 2, further comprising:
combining the overall blockiness metric with at least one from a set of: a sharpness metric, a jerkiness metric, and a dropped frame number to adjust at least one operational parameter of the video conference system for automatically enhancing received video quality.

### Claim 7 (depends on 6)

The method of claim 6, wherein the at least one operational parameter is adjusted at one of: a pre-processing module, an encoder, a decoder, and the post-processing module of the video conference system.

### Claim 8 (depends on 1)

The method of claim 1, wherein the blockiness metric for the columns and the blockiness metric for the rows are averaged prior to being combined to obtain the overall blockiness metric for the decoded video frame.

### Claim 9 (depends on 1)

The method of claim 1, wherein the blockiness metric for the columns and the blockiness metric for the rows are computed using a ratio of the autocorrelation and cross-correlation of the corresponding columns and rows such that the overall blockiness metric for the decoded video frame is inverse proportional to a perceivable appearance of blockiness artifacts.

### Claim 10 (depends on 1)

The method of claim 1, wherein the overall blockiness metric for the decoded video frame is further computed by determining and combining blockiness metrics for Y, U, and V planes of the decoded video frame.

### Claim 11 (depends on 10)

The method of claim 10, wherein the blockiness metrics for Y, U, and V planes of the decoded video frame are combined through averaging employing a weighting factor of ½ for the Y plane, a weighting factor of ¼ for the U plane, and a weighting factor of ¼ for the V plane.

### Claim 12

A computing device capable of providing video conferencing services, the computing device comprising:
a video capture device;
a pre-processing module;
an encoder;
a decoder;
a post-processing module configured to:
receive a decoded video frame;
compute a blockiness metric for columns of the decoded video frame based on averaging autocorrelation and cross-correlation values of each column;
compute a blockiness metric for rows of the decoded video frame based on averaging autocorrelation and cross-correlation values of each row;
combine the blockiness metrics for the columns and the rows to obtain an overall blockiness metric for the decoded video frame; and
scale the overall blockiness metric by applying a non-linear logistic function to replicate a behavior of Human Visual System (HSV), wherein the scaled overall blockiness metric is used to adjust at least one operational parameter of the video conference system for automatically enhance received video; and
a display device for displaying the enhanced video.

### Claim 13 (depends on 12)

The computing device of claim 12, wherein the post-processing module is further configured to:
compute the blockiness metrics for the columns and the rows by determining a ratio of the autocorrelation and the cross-correlation values of each column and row, respectively;
compute the overall blockiness metric for Y, U, and V planes of the decoded video frame; and
combine the overall blockiness metrics of the Y, U, and V planes by weighting them with ½, ¼, and ¼, respectively.

### Claim 14 (depends on 12)

The computing device of claim 12, wherein the overall blockiness metric is employed to perform one of:
instruct the encoder to increase a transmit bit rate in response to the metric being below a predefined threshold; and
instruct the post-processing module to increase a filtering strength in response to the metric being below the predefined threshold.

### Claim 15 (depends on 12)

The computing device of claim 12, wherein a Multipoint Control Unit (MCU) of the video conference system is employed to receive the overall blockiness metric from a receiving participant and instruct a transmitting participant to adjust the at least one operational parameter for enhancing the received video.

### Claim 16 (depends on 12)

The computing device of claim 12, wherein at least one from a set of: the pre-processing module, the encoder, the decoder, and the post-processing module are implemented as part of a processor configured to compute the overall blockiness metric.

### Claim 17

A computer-readable memory device having instructions stored thereon for providing video conferencing services, the instructions comprising:
receiving a decoded video frame at a post-processing module;
computing a blockiness metric for columns of the decoded video frame based on averaging a ratio of autocorrelation and cross-correlation values of each column;
computing a blockiness metric for rows of the decoded video frame based on averaging a ratio of autocorrelation and cross-correlation values of each row;
combining the blockiness metric for the columns and the blockiness metric for the rows to obtain overall blockiness metrics for Y, U, and V planes of the decoded video frame;
combining the overall blockiness metrics for Y, U, and V planes of the decoded video frame in a weighted manner;
scaling the overall blockiness metric by:
determining a maximum overall blockiness metric among a predefined number of decoded video frames;
determining an exponential model based on the determined maximum overall blockiness metric; and
applying the exponential model to the overall blockiness metrics of the predefined number of decoded video frames such that the overall blockiness metrics have defined lower and upper limits.

### Claim 18 (depends on 17)

The computer-readable memory device of claim 17, wherein the scaled overall blockiness metric is computed as:
B
scaled
⁡
(
i
)
=
c
⁢
E
⁡
(
i
)
E
max
,
where “i” is a frame number, E(i) is the exponential model defined as:
E
⁡
(
i
)
=
exp
⁡
(
8
*
B
⁡
(
i
)
B
max
)
,
and Emax=maxi[E1, E2, E3, . . . EN] with “c” determining the upper limit for the scaled overall blockiness metric.

### Claim 19 (depends on 17)

The computer-readable memory device of claim 17, wherein the blockiness metrics for each of the columns and the rows are determined by B=1−A/C with A representing the autocorrelation values and C representing the cross-correlation values for respective columns and rows.

### Claim 20 (depends on 17)

The computer-readable memory device of claim 17, wherein the instructions further comprise:
combining the scaled overall blockiness metric with at least one from a set of a sharpness metric, a jerkiness metric, and a dropped frame number to obtain an overall video quality metric; and
employing the overall video quality metric in automatically adjusting at least one operational parameter of a video conference system providing the video conferencing services for enhancing video quality.
