# US 2016/0240224 A1 · Reference and non-reference video quality evaluation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0240224 A1
- Document ID: US20160240224A1
- Status: Published application
- Kind: A1
- Application number: 15/137,998
- Filed: 2016-04-25
- Published: 2016-08-18
- Earliest claimed: 2013-05-28
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Krasimir D. Kolarov; Steven E Saunders
- Web page: https://patentgrep.com/patent/US20160240224A1

## CPC

- H04N 17/002: {for television cameras}
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- H04N 17/004: {for digital television systems}
- H04N 19/426: using memory downsizing methods
- G06T 2207/30168: Image quality inspection
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)

## Abstract

Devices and methods for determining image quality using full-reference and non-reference techniques. Full reference image quality may be determined prior to output of an image or video frame from an image sensor processor by temporarily retaining image data from the image sensor and comparing processed image data of the image to the retained, non-processed image data of the same image. Full reference image quality determination may be assisted by a heuristic-based fault indicator. Image quality may also be determined by a non-reference technique of matching the image to one of various scenarios that are associated with sets of heuristics and applying the heuristics of the particular scenario to the image. Instead of relying on a nominal frame rate, video timing quality may be determined by comparing the capture time interval between successive video frames to the presentation time interval of the same video frames.

## Claims

### Claim at position 1

1-9. (canceled)

### Claim 10

A method, comprising:
performing, by one or more computing devices having one or more processors:
obtaining an image;
evaluating the image to determine an image content scenario for the image;
selecting a heuristic set from among a plurality of scenario-specific heuristic sets, wherein each scenario-specific heuristic set corresponds to a different image content scenario, wherein said selecting is based on the determined image content scenario;
analyzing the image with the selected scenario-specific heuristic set to determine one or more image quality metric measurements for the image; and
outputting the one or more image quality metric measurements.

### Claim 11 (depends on 10)

The method of claim 10, wherein each scenario-specific heuristic set comprises quality indicators for one or more characteristic features of the corresponding image content scenario, and wherein said analyzing the image with the selected scenario-specific heuristic set comprises applying the quality indicators to corresponding features of the image to determine the one or more quality metrics.

### Claim 12 (depends on 10)

The method of claim 10, wherein the different image content scenarios comprise a human face scenario, a multiple human faces scenario, a landscape scenario, or a cityscape scenario.

### Claim 13 (depends on 10)

The method of claim 10, wherein the one or more characteristic features of the different image content scenarios comprise a facial feature, landscape feature, or a cityscape feature.

### Claim 14 (depends on 10)

The method of claim 10, further comprising
repeating the obtaining, evaluating, selecting and analyzing for a plurality of video images, wherein the repeating creates a plurality of image quality metrics from a respective plurality of video images;
comparing at least one of the plurality of image quality metrics to a threshold, wherein comparing at least one of the plurality of image quality metrics comprises determining whether the image quality metric indicates poor image quality; and
in response to determining that the image quality metric exceeds the threshold, outputting an image quality indicator for use in navigating to the particular image associated with the image quality metric.

### Claim at position 7

15-20. (canceled)

### Claim 21

A system, comprising:
one or more computer processors; and
memory storing program instructions that are executed by the one or more computer processors to implement a heuristics-based analyzer configured to:
obtain an image;
evaluate the image to determine an image content scenario for the image;
select, based at least in part on the determined image content scenario, a heuristic set from among a plurality of scenario-specific heuristic sets, wherein each scenario-specific heuristic set corresponds to a different image content scenario;
determine, based at least in part on analysis of the image with the selected scenario-specific heuristic set, one or more image quality metric measurements for the image; and
output the one or more image quality metric measurements.

### Claim 22 (depends on 21)

The system of claim 21, wherein the different image content scenarios comprise a human face scenario, a multiple human faces scenario, a landscape scenario, or a cityscape scenario.

### Claim 23 (depends on 21)

The system of claim 21, wherein each scenario-specific heuristic set comprises quality indicators for one or more characteristic features of the corresponding image content scenario, and wherein to analyze the image with the selected scenario-specific heuristic set the instructions are further executed to apply the quality indicators to corresponding features of the image to determine the one or more quality metrics.

### Claim 24 (depends on 23)

The system of claim 23, wherein the characteristic features of the different image content scenarios comprise a facial feature, landscape feature, or a cityscape feature.

### Claim 25 (depends on 21)

The system of claim 21, wherein the program instructions are further executed to:
repeat said obtain, evaluate, select and analyze for a plurality of video images to create a plurality of image quality metrics from a respective plurality of video images;
determine, based at least in part on a comparison of at least one of the plurality of image quality metrics to a threshold, whether the image quality metric indicates poor image quality; and
in response to a determination that the image quality metric exceeds the threshold, output an image quality indicator for use in navigation to the particular image associated with the image quality metric.

### Claim 26 (depends on 21)

The system of claim 21, wherein the different image content scenarios include a human face scenario that is associated with a heuristic that identifies high contrasts and sharp detail in the facial area.

### Claim 27 (depends on 21)

The system of claim 21, wherein the different image content scenarios include an outdoor landscape scenario that is associated with a heuristic that emphasizes detail sharpness in the foliage areas, or a city landscape scenario that is associated with a heuristic that emphasizes sharp edges of rectangular features.

### Claim 28 (depends on 21)

The system of claim 21, further comprising:
an image sensor; and
an image sensor data processor configured to process images captured from the image sensor into processed versions of the captured image and evaluate the quality of the processed version of the captured image;
wherein the heuristics-based analyzer is configured to work in coordination with the image sensor data processor.

### Claim 29 (depends on 21)

The system of claim 21, further comprising an image data pre-processor configured to correct for one or more distortions in image data from an image sensor of a device, wherein the selected scenario-specific heuristic set are applied to determine that the image data pre-processor is functioning correctly.

### Claim 30

A non-transitory computer-readable medium storing program instructions executable to perform:
obtain an image;
evaluate the image to determine an image content scenario for the image;
select, based on the determined image content scenario, a heuristic set from among a plurality of scenario-specific heuristic sets, wherein each scenario-specific heuristic set corresponds to a different image content scenario;
determine, based at least in part on analysis of the image with the selected scenario-specific heuristic set, one or more image quality metric measurements for the image; and
output the one or more image quality metric measurements.

### Claim 31 (depends on 30)

The non-transitory computer-readable medium of claim 30, wherein the different image content scenarios comprise a human face scenario, a multiple human faces scenario, a landscape scenario, or a cityscape scenario.

### Claim 32 (depends on 30)

The non-transitory computer-readable medium of claim 30, wherein each scenario-specific heuristic set comprises quality indicators for one or more characteristic features of the corresponding image content scenario, and wherein to analyze the image with the selected scenario-specific heuristic set the instructions are further executed to apply the quality indicators to corresponding features of the image to determine the one or more quality metrics.

### Claim 33 (depends on 32)

The non-transitory computer-readable medium of claim 32, wherein the characteristic features of the different image content scenarios comprise a facial feature, landscape feature, or a cityscape feature.

### Claim 34

The non-transitory computer-readable medium of 30, wherein the program instructions are further executed to:
repeat said obtain, evaluate, select and analyze for a plurality of video images to create a plurality of image quality metrics from a respective plurality of video images;
determine, based at least in part on a comparison of at least one of the plurality of image quality metrics to a threshold, whether the image quality metric indicates poor image quality; and
in response to a determination that the image quality metric exceeds the threshold, output an image quality indicator for use in navigation to the particular image associated with the image quality metric.

### Claim 35 (depends on 30)

The non-transitory computer-readable medium of claim 30, wherein the program instructions are further executable to determine, based at least in part on application of the selected scenario-specific heuristic set, that an image data pre-processor of a device is functioning correctly.
