# US 2020/0401831 A1 · Content-aware selection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401831 A1
- Document ID: US20200401831A1
- Status: Published application
- Kind: A1
- Application number: 17/013,171
- Filed: 2020-09-04
- Published: 2020-12-24
- Earliest claimed: 2019-01-31
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Subham Gupta; Ajay Bedi; Poonam Bhalla; Krishna Singh Karki
- Web page: https://patentgrep.com/patent/US20200401831A1

## CPC

- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06V 10/235: {based on user input or interaction}
- G06V 20/20: in augmented reality scenes
- G06V 30/268: {Lexical context}
- G06V 2201/10: Recognition assisted with metadata
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

An image editing program can include a content-aware selection system. The content-aware selection system can enable a user to select an area of an image using a label or a tag that identifies object in the image, rather than having to make a selection area based on coordinates and/or pixel values. The program can receive a digital image and metadata that describes an object in the image. The program can further receive a label, and can determine from the metadata that the label is associated with the object. The program can then select a bounding box for the object, and identify in the bounding box, pixels that represent the object. The program can then output a selection area that surrounds the pixels.

## Claims

### Claim 1

A computer-implemented method, comprising:
receiving metadata information for a plurality of objects depicted in a digital image, wherein, for each object in the plurality of objects, the metadata information comprises a label associated with that object;
generating a set of hierarchical relationships among the labels identified in the metadata information for the plurality of objects;
receiving a user input directed to a location within the digital image;
based upon the location, identifying a first object from the plurality of objects and a first label associated with the first object;
determining, based upon the set of hierarchical relationships, a second label that is included in the set of hierarchical relationships;
generating a menu comprising a plurality of user-selectable options for selecting one or more of the plurality of objects, the plurality of user-selectable options comprising a first option based upon the first label and a second option based upon the second label; and
outputting the menu including the plurality of user-selectable options via an output device.

### Claim 2

The computer-implemented method of claim Error! Reference source not found., further comprising:
determining that the location is within a region of the digital image that includes a set of pixels representing the first object;
identifying a first bounding box for the first object;
determining that the region of the digital image is included in a second bounding box for an additional object from the plurality of objects, wherein an additional label is associated with the additional object; and
responsive to determining that the region of the digital image is included in the second bounding box, including an additional option in the menu based upon the additional label.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining that, in the set of hierarchical relationships, the second label and the first label are hierarchically related via a super-category that includes the first label and the second label,
wherein the second option is one of:
a pluralized version of the first label, or
a pluralized version of the super-category.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining that, in the set of hierarchical relationships, the second label and the first label are hierarchically related via a super-category that includes the first label and the second label; and
responsive to determining that the second label and the first label are hierarchically related via the super-category, including an additional option in the menu based upon the super-category.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising:
generating the metadata information for the plurality of objects by performing object recognition on the digital image.

### Claim 6

The computer-implemented method of claim Error! Reference source not found., further comprising:
generating a relationship tree data structure that indicates the set of hierarchical relationships, the relationship tree data structure including:
a sub-node corresponding to the first label associated with the first object, and
a super-node corresponding to a super-category of the first label, wherein the super-node encompasses the sub-node.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the super-node indicates a count of labels corresponding to sub-nodes encompassed by the super-node.

### Claim 8

The computer-implemented method of claim Error! Reference source not found., further comprising:
filtering the labels identified in the metadata information, wherein filtering the labels identified in the metadata information produces a subset of labels,
wherein generating the set of hierarchical relationships is performed using the subset of labels.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein filtering the labels identified in the metadata information is performed using a lexical dictionary.

### Claim 10 (depends on 8)

The computer-implemented method of claim 8, wherein:
the metadata information for the plurality of objects further comprises, for each object in the plurality of objects, a bounding box that includes a set of pixels representing that object,
filtering the labels identified in the metadata information includes:
determining an overlap between a first bounding box associated with the first object and a second bounding box associated with a second object from the plurality of objects; and
omitting, from the subset of labels, an additional label associated with the second object, responsive to determining a conceptual similarity between the first label and the additional label.

### Claim 11

A computing device executing a content-aware selection system, the computing device comprising:
one or more processors; and
a memory component configured to implement instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:
receiving metadata information for a plurality of objects depicted in a digital image, wherein, for each object in the plurality of objects, the metadata information comprises a label associated with that object;
generating a set of hierarchical relationships among the labels identified in the metadata information for the plurality of objects;
receiving a user input directed to a location within the digital image;
based upon the location, identifying a first object from the plurality of objects and a first label associated with the first object;
determining, based upon the set of hierarchical relationships, a second label that is included in the set of hierarchical relationships;
generating a menu comprising a plurality of user-selectable options for selecting one or more of the plurality of objects, the plurality of user-selectable options comprising a first option based upon the first label and a second option based upon the second label; and
outputting the menu including the plurality of user-selectable options via an output device.

### Claim 12 (depends on 11)

The computing device of claim 11, wherein the instructions further cause the one or more processors to perform operations including:
determining that the location is within a region of the digital image that includes a set of pixels representing the first object;
identifying a first bounding box for the first object;
determining that the region of the digital image is included in an additional bounding box for an additional object from the plurality of objects, wherein an additional label is associated with the additional object; and
responsive to determining that the region of the digital image is included in the additional bounding box, including an additional option in the menu based upon the additional label.

### Claim 13 (depends on 11)

The computing device of claim 11, wherein the instructions further cause the one or more processors to perform operations including:
determining that, in the set of hierarchical relationships, the second label and the first label are hierarchically related via a super-category that includes the first label and the second label,
wherein the second option is one of:
a pluralized version of the first label, or
a pluralized version of the super-category.

### Claim 14 (depends on 11)

The computing device of claim 11, wherein the instructions further cause the one or more processors to perform operations including:
determining that, in the set of hierarchical relationships, the second label and the first label are hierarchically related via a super-category that includes the first label and the second label; and
responsive to determining that the second label and the first label are hierarchically related via the super-category, including an additional option in the menu based upon the super-category.

### Claim 15 (depends on 11)

The computing device of claim 11, wherein the instructions further cause the one or more processors to perform operations including:
generating a relationship tree data structure that indicates the set of hierarchical relationships, the relationship tree data structure including:
a sub-node corresponding to the first label associated with the first object, and
a super-node corresponding to a super-category of the first label, wherein the super-node encompasses the sub-node.

### Claim 16 (depends on 11)

The computing device of claim 11, wherein the instructions further cause the one or more processors to perform operations including:
filtering the labels identified in the metadata information, wherein filtering the labels identified in the metadata information produces a subset of labels,
wherein generating the set of hierarchical relationships is performed using the subset of labels.

### Claim 17 (depends on 16)

The computing device of claim 16, wherein:
the metadata information for the plurality of objects further comprises, for each object in the plurality of objects, a bounding box that includes a set of pixels representing that object,
filtering the labels identified in the metadata information includes:
determining an overlap between a first bounding box associated with the first object and a second bounding box associated with a second object from the plurality of objects; and
omitting, from the subset of labels, an additional label associated with the second object, responsive to determining a conceptual similarity between the first label and the additional label.

### Claim 18

A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:
receiving metadata information for a plurality of objects depicted in a digital image, wherein, for each object in the plurality of objects, the metadata information comprises a label associated with that object;
generating a set of hierarchical relationships among the labels identified in the metadata information for the plurality of objects;
receiving a user input directed to a location within the digital image;
based upon the location, identifying a first object from the plurality of objects and a first label associated with the first object;
determining, based upon the set of hierarchical relationships, a second label that is included in the set of hierarchical relationships;
generating a menu comprising a plurality of user-selectable options for selecting one or more of the plurality of objects, the plurality of user-selectable options comprising a first option based upon the first label and a second option based upon the second label; and
outputting the menu including the plurality of user-selectable options via an output device.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium of claim 18, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:
determining that the location is within a region of the digital image that includes a set of pixels representing the first object;
identifying a first bounding box for the first object;
determining that the region of the digital image is included in an additional bounding box for an additional object from the plurality of objects, wherein an additional label is associated with the additional object; and
responsive to determining that the region of the digital image is included in the additional bounding box, including an additional option in the menu based upon the additional label.

### Claim 20 (depends on 18)

The non-transitory computer-readable medium of claim 18, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:
determining that, in the set of hierarchical relationships, the second label and the first label are hierarchically related via a super-category that includes the first label and the second label; and
responsive to determining that the second label and the first label are hierarchically related via the super-category, including an additional option in the menu based upon the super-category.