# US 2015/0178596 A1 · Label Consistency for Image Analysis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0178596 A1
- Document ID: US20150178596A1
- Status: Published application
- Kind: A1
- Application number: 14/135,816
- Filed: 2013-12-20
- Published: 2015-06-25
- Earliest claimed: 2013-12-20
- Assignees: Google Inc.
- Applicants: Google Inc.
- Inventors: Samy Bengio; Jeffrey Adgate Dean; Quoc Le; Jonathon Shlens; Yoram Singer
- Web page: https://patentgrep.com/patent/US20150178596A1

## CPC

- G06N 20/00: Machine learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06V 10/758: {Involving statistics of pixels or of feature values, e.g. histogram matching}
- G06V 10/768: {using context analysis, e.g. recognition aided by known co-occurring patterns}
- G06V 20/30: in albums, collections or shared content, e.g. social network photos or video
- G06V 30/268: {Lexical context}
- G06V 10/759: {Region-based matching}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and techniques are disclosed for labeling objects within an image. The objects may be labeled by selecting an option from a plurality of options such that each option is a potential label for the object. An option may have an option score associated with. Additionally, a relation score may be calculated for a first option and a second option corresponding to a second object in an image. The relation score may be based on a frequency, probability, or observance corresponding to the co-occurrence of text associated with the first option and the second option in a text corpus such as the World Wide Web. An option may be selected as a label for an object based on a global score calculated based at least on an option score and relation score associated with the option.

## Claims

### Claim 1

A computer implemented method comprising:
receiving, by the computer, a first option for a first object in an image from a first plurality of options;
receiving, by the computer, a second option for a second object in the image from a second plurality of options;
generating, by the computer, a relation score between the first option and the second option based on a co-occurrence model; and
designating, by the computer, the first option as a label for the first object in the image based on solving a global optimization problem utilizing at least the relation score.

### Claim 2 (depends on 1)

The method of claim 1, further comprising designating the second option as a label for the second object in the image.

### Claim 3 (depends on 1)

The method of claim 1, wherein the co-occurrence model is a machine learning model trained using a text corpus, and the relation score is based upon a co-occurrence of the first and second options in the text corpus.

### Claim 4 (depends on 1)

The method of claim 1, further comprising designating the first option as a label for the first object based on a regularization factor.

### Claim 5 (depends on 1)

The method of claim 1, wherein generating the relation score further comprises, determining the frequency at which text associated with the first option and the second option co-occur within a text corpus.

### Claim 6 (depends on 1)

The method of claim 1, wherein generating the relation score further comprises applying the co-occurrence model to the first option and the second option wherein the co-occurrence model is generated based at least on co-occurrence within a limit selected from the group consisting of: adjacent words or terms, a number of words, an area, a same sentence, a same paragraph, a same page, and a same document.

### Claim 7 (depends on 3)

The method of claim 3, wherein the text corpus is a portion of the World Wide Web.

### Claim 8 (depends on 1)

The method of claim 1, wherein the first option is generated using an image patch labeler.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
applying a machine learned model to data extracted from the first object in the image; and
generating the first option based on the applied model.

### Claim 10 (depends on 1)

The method of claim 1, wherein the first object is an area of an image.

### Claim 11 (depends on 10)

The method of claim 10, wherein the second object is an area of an image different than the area of the image corresponding to the first object.

### Claim 12 (depends on 1)

The method of claim 1, wherein receiving a first option for a first object further comprising:
determining a first potential option with a first potential option score, for the first object;
determining a second potential option with a second potential option score, for the first object;
determining that the first potential option score is higher than the second potential option score; and
providing the first potential option as the first option for the first object based on determining that the first potential option score is higher than the second potential option.

### Claim 13 (depends on 12)

The method of claim 12, wherein the first potential option and the second potential option are assigned based on a likelihood that that the first potential label and the second potential label identify the first object.

### Claim 14 (depends on 1)

The method of claim 1, further comprising:
receiving a search query;
identifying the first image based on the label for the first object as a result for the query; and
providing the first image based on the identification.

### Claim 15

A computer implemented method comprising:
receiving, by the computer, a first option for a first object in an image from a first plurality of options;
receiving, by the computer, a second option for the first object in the image from a first plurality of options;
receiving, by the computer, a control label for a second object in the image;
generating a first relation score between the first option and the control label based on a co-occurrence model;
generating, by the computer, a second relation score between the second option and the control label based on the co-occurrence model;
determining that the first relation score exceeds the second relation score; and
designating the first option as a label for the first object in the image.

### Claim 16 (depends on 15)

The method of claim 15, wherein the co-occurrence model is a machine learning model trained using a text corpus.

### Claim 17 (depends on 15)

The method of claim 15, wherein generating the first relation score further comprises, determining the probability that text associated with the first option and the control option co-occur within a text corpus.

### Claim 18 (depends on 15)

The method of claim 15, wherein generating the first relation score further comprises applying the co-occurrence model to the first option and the control option wherein the co-occurrence model is generated based at least on co-occurrence within a limit selected from the group consisting of: adjacent words or terms, a same sentence, a same paragraph, a same page, and a same document.

### Claim 19 (depends on 16)

The method of claim 16, wherein the text corpus is a portion of the World Wide Web.

### Claim 20 (depends on 15)

The method of claim 15, wherein the first option is generated using a label generator.

### Claim 21 (depends on 15)

The method of claim 15, further comprising:
receiving a search query;
identifying the first image based on the label for the first object as a result for the query; and
providing the first image based on the identification.

### Claim 22 (depends on 15)

The method of claim 15, further comprising designating the first option as a label for the first object based on a regularization factor.

### Claim 23

A computer implemented method comprising:
receiving a first option for a first patch within an image;
receiving a second option for a second patch within the image;
generating a first option score for a first patch;
generating a first relation score based on the consistency between the first option and the second option; and
generating a first global score for the first patch based on the first option score and the first relation score.

### Claim 24 (depends on 23)

The method of claim 23, further comprising designating the first option as a label for the first object in the image if the global score exceeds a global score threshold.

### Claim 25 (depends on 23)

The method of claim 23 further comprising:
receiving a third option for the first patch within an image;
generating a second option score for a first patch;
generating a second relation score based on the consistency between the third option and the second option;
generating a second global score for the first patch based on the third option score and the second relation score;
comparing the first global score to the second global score; and
designating the first option as a label for the first object in the image based on the comparison.

### Claim 26

A system comprising:
a processor configured to:
receive a first option for a first object in an image from a first plurality of options;
receive a second option for a second object in the image from a second plurality of options;
generate a relation score between the first option and the second option based on a co-occurrence model; and
designate the first option as a label for the first object in the image based on solving a global optimization problem utilizing at least the relation score.

### Claim 27 (depends on 26)

The system of claim 26, wherein the co-occurrence model is a machine learning model trained using a text corpus, and the relation score is based upon a co-occurrence of the first and second options in the text corpus.

### Claim 28 (depends on 27)

The system of claim 27, wherein the text corpus is a portion of the World Wide Web.

### Claim 29 (depends on 26)

The system of claim 26, further configured to designate the first option as a label for the first object based on a regularization factor.

### Claim 30 (depends on 26)

The system of claim 26, wherein generating the relation score further comprises, determining the frequency at which text associated with the first option and the second option co-occur within a text corpus.

### Claim 31 (depends on 26)

The system of claim 26, wherein generating the relation score further comprises applying the co-occurrence model to the first option and the second option wherein the co-occurrence model is generated based at least on co-occurrence within a limit selected from the group consisting of: adjacent words or terms, a number of words, an area, a same sentence, a same paragraph, a same page, and a same document.

### Claim 32

A system comprising:
a processor configured to:
receive a first option for a first object in an image from a first plurality of options;
receive a second option for the first object in the image from a first plurality of options;
receive a control label for a second object in the image;
generate a first relation score between the first option and the control label based on a co-occurrence model;
generate, by the computer, a second relation score between the second option and the control label based on the co-occurrence model;
determine that the first relation score exceeds the second relation score; and
designate the first option as a label for the first object in the image.

### Claim 33 (depends on 32)

The system of claim 32, wherein the co-occurrence model is a machine learning model trained using a text corpus.

### Claim 34 (depends on 32)

The system of claim 32, wherein generating the first relation score further comprises, determining the probability that text associated with the first option and the control option co-occur within a text corpus.

### Claim 35 (depends on 32)

The system of claim 32, further configured to:
receive a search query;
identify the first image based on the label for the first object as a result for the query; and
provide the first image based on the identification.

### Claim 36

A system comprising:
a processor configured to:
receive a first option for a first patch within an image;
receive a second option for a second patch within the image;
generate a first option score for a first patch;
generate a first relation score based on the consistency between the first option and the second option; and
generate a first global score for the first patch based on the first option score and the first relation score.

### Claim 37 (depends on 36)

The system of claim 36, further configured to designate the first option as a label for the first object in the image if the global score exceeds a global score threshold.

### Claim 38 (depends on 36)

The system of claim 36 further configured to:
receive a third option for the first patch within an image;
generate a second option score for a first patch;
generate a second relation score based on the consistency between the third option and the second option;
generate a second global score for the first patch based on the third option score and the second relation score;
compare the first global score to the second global score; and
designate the first option as a label for the first object in the image based on the comparison.

### Claim 39

A device comprising:
a storage; and
a processor configured to:
receive a first option for a first object in an image from a first plurality of options;
receive a second option for a second object in the image from a second plurality of options;
generate a relation score between the first option and the second option based on a co-occurrence model; and
designate the first option as a label for the first object in the image based on solving a global optimization problem utilizing at least the relation score.

### Claim 40 (depends on 39)

The device of claim 39, wherein the co-occurrence model is a machine learning model trained using a text corpus, and the relation score is based upon a co-occurrence of the first and second options in the text corpus.

### Claim 41 (depends on 39)

The device of claim 39, further configured to designate the first option as a label for the first object based on a regularization factor.

### Claim 42 (depends on 39)

The device of claim 39, wherein generating the relation score further comprises, determining the frequency at which text associated with the first option and the second option co-occur within a text corpus.

### Claim 43 (depends on 39)

The device of claim 39, wherein generating the relation score further comprises applying the co-occurrence model to the first option and the second option wherein the co-occurrence model is generated based at least on co-occurrence within a limit selected from the group consisting of: adjacent words or terms, a number of words, an area, a same sentence, a same paragraph, a same page, and a same document.

### Claim 44

A device comprising:
a storage; and
a processor configured to:
receive a first option for a first object in an image from a first plurality of options;
receive a second option for the first object in the image from a first plurality of options;
receive a control label for a second object in the image;
generate a first relation score between the first option and the control label based on a co-occurrence model;
generate, by the computer, a second relation score between the second option and the control label based on the co-occurrence model;
determine that the first relation score exceeds the second relation score; and
designate the first option as a label for the first object in the image.

### Claim 45 (depends on 44)

The device of claim 44, wherein the co-occurrence model is a machine learning model trained using a text corpus.

### Claim 46 (depends on 44)

The device of claim 44, wherein generating the first relation score further comprises, determining the probability that text associated with the first option and the control option co-occur within a text corpus.

### Claim 47 (depends on 44)

The device of claim 44, further configured to:
receive a search query;
identify the first image based on the label for the first object as a result for the query; and
provide the first image based on the identification.

### Claim 48

A device comprising:
a storage; and
a processor configured to:
receive a first option for a first patch within an image;
receive a second option for a second patch within the image;
generate a first option score for a first patch;
generate a first relation score based on the consistency between the first option and the second option; and
generate a first global score for the first patch based on the first option score and the first relation score.

### Claim 49 (depends on 48)

The device of claim 48, further configured to designate the first option as a label for the first object in the image if the global score exceeds a global score threshold.

### Claim 50 (depends on 48)

The device of claim 48, further configured to:
receive a third option for the first patch within an image;
generate a second option score for a first patch;
generate a second relation score based on the consistency between the third option and the second option;
generate a second global score for the first patch based on the third option score and the second relation score;
compare the first global score to the second global score; and
designate the first option as a label for the first object in the image based on the comparison.
