# US 2019/0244327 A1 · Image Cropping Suggestion Using Multiple Saliency Maps

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244327 A1
- Document ID: US20190244327A1
- Status: Published application
- Kind: A1
- Application number: 16/384,593
- Filed: 2019-04-15
- Published: 2019-08-08
- Earliest claimed: 2014-10-09
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Zhe Lin; Radomir Mech; Xiaohui Shen; Brian L. Price; Jianming Zhang; Anant Gilra; Jen-Chan Jeff Chien
- Web page: https://patentgrep.com/patent/US20190244327A1

## CPC

- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06T 3/04: Context-preserving transformations, e.g. by using an importance map (panospheric to cylindrical image transformations G06T3/12)
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 11/60: Creating or editing images; Combining images with text
- G06T 2210/22: Cropping
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Image cropping suggestion using multiple saliency maps is described. In one or more implementations, component scores, indicative of visual characteristics established for visually-pleasing croppings, are computed for candidate image croppings using multiple different saliency maps. The visual characteristics on which a candidate image cropping is scored may be indicative of its composition quality, an extent to which it preserves content appearing in the scene, and a simplicity of its boundary. Based on the component scores, the croppings may be ranked with regard to each of the visual characteristics. The rankings may be used to cluster the candidate croppings into groups of similar croppings, such that croppings in a group are different by less than a threshold amount and croppings in different groups are different by at least the threshold amount. Based on the clustering, croppings may then be chosen, e.g., to present them to a user for selection.

## Claims

### Claim 1

A method implemented by a computing device, the method comprising:
clustering items of image content based on at least two different types of saliency maps generated from the image content; and
generating a suggestion of an image content item based on the clustering.

### Claim 2 (depends on 1)

A method as described in claim 1, wherein clustering the items of image content further includes:
scoring the items of image content based on visual characteristics determined from the at least two different types of saliency maps; and
clustering the items of image content according to the scores.

### Claim 3 (depends on 2)

A method as described in claim 2, further comprising ranking the items of image content within a given cluster based on the scores.

### Claim 4 (depends on 3)

A method as described in claim 3, wherein generating the suggestion further comprises:
selecting a highest-ranked image content item from different clusters of the items of image content; and
identifying a selected image content item for the suggestion.

### Claim 5 (depends on 1)

A method as described in claim 1, wherein the items of image content in one cluster are different from the items of image content in other clusters by at least a threshold amount.

### Claim 6 (depends on 1)

A method as described in claim 1, wherein the items of image content in a given cluster are different, one item of image content from another, by less than a threshold amount.

### Claim 7 (depends on 1)

A method as described in claim 1, wherein the at least two different types of saliency maps include:
a dense saliency map that combines a saliency map computed from red, green, blue (RGB) information of the image content and single-channel saliency maps computed for each color channel of the RGB information; and
a saliency edge map computed using the dense saliency map and a gradient magnitude map that indicates, for each pixel of the image content, an amount the image is changing at the pixel relative to other pixels.

### Claim 8 (depends on 1)

A method as described in claim 1, wherein the at least two different types of saliency maps include two or more of:
a base saliency map that is computed from RGB information of the image content;
a dense saliency map that combines the base saliency map and single-channel saliency maps computed for each color channel of the RGB information; and
a row-normalized gradient map that is computed by applying a threshold to pixel values of a gradient magnitude map and then normalizing the pixel values of the gradient magnitude map by row, the gradient magnitude map configured to indicate, for each pixel of the image content, an amount the image content is changing at the pixel relative to other pixels.

### Claim 9 (depends on 1)

A method as described in claim 1, further comprising presenting the suggestion of the image content item for selection by a user.

### Claim 10 (depends on 9)

A method as described in claim 9, wherein the clustering is performed to implement an image search and the suggestion of the image content item is presented for selection by the user based on the image search.

### Claim 11

A system comprising:
an image saliency module implemented at least partially in hardware of at least one computing device to generate at least two different types of saliency maps from image content of items of the image content; and
a selection module implemented at least partially in the hardware of the at least one computing device to cluster the items of image content based on the at least two different types of saliency maps and generate a suggestion of an image content item based on clusters of the items of image content.

### Claim 12 (depends on 11)

A system as described in claim 11, wherein the selection module is further configured to:
score the items of image content based on visual characteristics determined from the at least two different types of saliency maps; and
cluster the items of image content according to the scores.

### Claim 13 (depends on 12)

A system as described in claim 12, wherein the items of image content are scored with multiple component scores that are computed for each item of the image content and are indicative of a different visual characteristic established for visually pleasing image content items.

### Claim 14 (depends on 13)

A system as described in claim 13, wherein the multiple component scores include at least one of:
a composition quality score; and
a boundary simplicity score.

### Claim 15 (depends on 12)

A system as described in claim 12, wherein the selection module is further configured to rank the items of image content within a given cluster based on the scores.

### Claim 16 (depends on 15)

A system as described in claim 15, wherein the selection module is further configured to:
select a highest-ranked image content item from different clusters of the items of image content; and
identify a selected image content item for the suggestion.

### Claim 17

A method implemented by a computing device, the method comprising:
computing multiple different saliency maps from image content of items of the image content; and
generating a suggestion of an image content item based on rankings assigned to multiple candidate image content item suggestions, wherein the rankings reflect an assessment of:
composition quality of the multiple candidate image content item suggestions, the assessment of the composition quality performed using a first combination of the multiple different saliency maps; and
boundary simplicity of the multiple candidate image content item suggestions, the assessment of the boundary simplicity performed using a second combination of the multiple different saliency maps.

### Claim 18 (depends on 17)

A method as described in claim 17, wherein the first combination of saliency maps includes:
a dense saliency map that combines a saliency map computed from red, green, blue (RGB) information of the image content and single-channel saliency maps computed for each color channel of the RGB information; and
a saliency edge map computed using the dense saliency map and a gradient magnitude map that indicates, for each pixel of the image content, an amount the image content is changing at the pixel relative to other pixels.

### Claim 19 (depends on 17)

A method as described in claim 17, wherein the second combination of saliency maps includes edge saliency maps computed for each of:
a left edge boundary region of a particular candidate image content item suggestion;
a right edge boundary region of the particular candidate image content item suggestion;
a top edge boundary region of the particular candidate image content item suggestion; and
a bottom edge boundary region of the particular candidate image content item suggestion.

### Claim 20 (depends on 17)

A method as described in claim 17, wherein generating the suggestion of the image content item includes clustering the multiple candidate image content item suggestions into clusters of similar candidates based on the assessment.
