# US 11,308,345 B2 · Saliency of an object for image processing operations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,345 B2
- Document ID: US11308345B2
- Status: Granted
- Kind: B2
- Application number: 16/848,315
- Filed: 2020-04-14
- Published: 2020-12-03
- Granted: 2022-04-19
- Earliest claimed: 2019-05-31
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Vignesh Jagadeesh; Yingjun Bai; Guillaume Tartavel; Gregory Guyomarc'h
- Web page: https://patentgrep.com/patent/US11308345B2

## CPC

- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- H04N 23/67: Focus control based on electronic image sensor signals
- G06V 20/64: Three-dimensional [3D] objects
- H04N 23/88: for colour balance, e.g. white-balance circuits or colour temperature control
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Various methods for utilizing a saliency heatmaps are described. The methods include obtaining image data corresponding to an image of a scene, obtaining a saliency heatmap for the image of the scene based on a saliency network, wherein the saliency heatmap indicates a likelihood of saliency for a corresponding portion of the scene, and manipulating the image data based on the saliency heatmap. In embodiments, the saliency heatmap may be produced using a trained machine learning model. The saliency heatmap may be used for various image processing tasks, such as determining which portion(s) of a scene to base an image capture device's autofocus, auto exposure, and/or white balance operations upon. According to some embodiments, one or more bounding boxes may be generated based on the saliency heatmap, e.g., using an optimization operation, which bounding box(es) may be used to assist or enhance the performance of various image processing tasks.

## Claims

### Claim 1

A non-transitory computer readable medium comprising computer readable code executable by one or more processors to:
obtain image data corresponding to an image of a scene;
generate a saliency heatmap for the image of the scene based on a saliency network, wherein the saliency heatmap indicates a likelihood that a corresponding portion of the scene includes a class-agnostic salient object; and
manipulate properties of the image data in accordance with the saliency heatmap.

### Claim 2 (depends on 1)

The non-transitory computer readable medium of claim 1, wherein the saliency network is trained on training image data to identify a likelihood of saliency in a class-agnostic manner.

### Claim 3 (depends on 1)

The non-transitory computer readable medium of claim 1, wherein the saliency heatmap provides a saliency value for each pixel in the image.

### Claim 4 (depends on 3)

The non-transitory computer readable medium of claim 3, further comprising computer readable code to: obtain a set of autofocus statistics for one or more pixels in the image; and weight the set of autofocus statistics based on the saliency value for the one or more pixels in the image.

### Claim 5 (depends on 3)

The non-transitory computer readable medium of claim 3, further comprising computer readable code to:
generate a binary mask of the image by applying a threshold value to the saliency value for each pixel; and
generate a bounding box based on the saliency heatmap and a bounding box network.

### Claim 6 (depends on 5)

The non-transitory computer readable medium of claim 5, wherein the bounding box network has been trained to estimate a location of the bounding box and a movement of the bounding box over a series of frames comprising the image.

### Claim 7 (depends on 1)

The non-transitory computer readable medium of claim 1, wherein the saliency network comprises optical flow data.

### Claim 8

A system for obtaining a saliency heatmap, comprising:
one or more processors; and
one or more computer readable media comprising computer readable code executable by the one or more processors to:
obtain image data corresponding to an image of a scene;
generate a saliency heatmap for the image of the scene based on a saliency network, wherein the saliency heatmap indicates a likelihood that a corresponding portion of the scene includes a class-agnostic salient object; and
manipulate properties of the image data in accordance with the saliency heatmap.

### Claim 9 (depends on 8)

The system of claim 8, wherein the saliency network is trained on training image data to identify a likelihood of saliency in a class-agnostic manner.

### Claim 10 (depends on 8)

The system of claim 8, wherein the saliency heatmap provides a saliency value for each pixel in the image.

### Claim 11 (depends on 10)

The system of claim 10, further comprising computer readable code to: obtain a set of autofocus statistics for one or more pixels in the image; and weight the set of autofocus statistics based on the saliency value for the one or more pixels in the image.

### Claim 12 (depends on 10)

The system of claim 10, further comprising computer readable code to:
generate a binary mask of the image by applying a threshold value to the saliency value for each pixel; and
generate a bounding box based on the saliency heatmap and a bounding box network.

### Claim 13 (depends on 12)

The system of claim 12, wherein the bounding box network has been trained to estimate a location of the bounding box and a movement of the bounding box over a series of frames comprising the image.

### Claim 14 (depends on 8)

The system of claim 8, wherein the saliency network comprises optical flow data.

### Claim 15

A method for utilizing a saliency heatmap, comprising:
obtaining image data corresponding to an image of a scene;
generating a saliency heatmap for the image of the scene based on a saliency network, wherein the saliency heatmap indicates a likelihood that a corresponding portion of the scene includes a class-agnostic salient object; and
manipulating properties of the image data in accordance with the saliency heatmap.

### Claim 16 (depends on 15)

The method of claim 15, wherein the saliency network is trained on training image data to identify a likelihood of saliency in a class-agnostic manner.

### Claim 17 (depends on 15)

The method of claim 15, wherein the saliency heatmap provides a saliency value for each pixel in the image.

### Claim 18 (depends on 17)

The method of claim 17, further comprising:
obtaining a set of autofocus statistics for one or more pixels in the image; and
weighting the set of autofocus parameters based on the saliency value for the one or more pixels in the image.

### Claim 19 (depends on 17)

The method of claim 17, further comprising:
generating a binary mask of the image by applying a threshold value to the saliency value for each pixel; and
generating a bounding box based on the saliency heatmap and a bounding box network.

### Claim 20 (depends on 19)

The method of claim 19, wherein the bounding box network has been trained to estimate a location of the bounding box and a movement of the bounding box over a series of frames comprising the image.