# US 11,442,983 B2 · Contextually disambiguating queries

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,442,983 B2
- Document ID: US11442983B2
- Status: Granted
- Kind: B2
- Application number: 16/731,786
- Filed: 2019-12-31
- Published: 2020-08-06
- Granted: 2022-09-13
- Earliest claimed: 2017-03-20
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Ibrahim Badr; Nils Grimsmo; Gokhan H. Bakir; Kamil Anikiej; Aayush Kumar; Viacheslav Kuznetsov
- Web page: https://patentgrep.com/patent/US11442983B2

## CPC

- G06F 16/5846: {using extracted text}
- G06F 16/5866: {using information manually generated, e.g. tags, keywords, comments, manually generated location and time information}
- G06F 16/9032: Query formulation
- G06V 10/768: {using context analysis, e.g. recognition aided by known co-occurring patterns}
- G06V 20/63: {Scene text, e.g. street names}
- G06V 30/262: using context analysis, e.g. lexical, syntactic or semantic context
- G06V 30/10: Character recognition
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for contextually disambiguating queries are disclosed. In an aspect, a method includes receiving an image being presented on a display of a computing device and a transcription of an utterance spoken by a user of the computing device, identifying a particular sub-image that is included in the image, and based on performing image recognition on the particular sub-image, determining one or more first labels that indicate a context of the particular sub-image. The method also includes, based on performing text recognition on a portion of the image other than the particular sub-image, determining one or more second labels that indicate the context of the particular sub-image, based on the transcription, the first labels, and the second labels, generating a search query, and providing, for output, the search query.

## Claims

### Claim 1

A method implemented by one or more processors, the method comprising:
identifying an image that is displayed at a computing device;
receiving data indicating a user request at the computing device, wherein the user request is received while the image is displayed on the computing device;
processing the image to identify a plurality of sub-images included in the image and displayed on the computing device when the user request is received, wherein each sub-image of the plurality of sub-images included in the image comprises a separate portion of the image;
generating, based on the data indicating the user request, a plurality of confidence scores, wherein each of the confidence scores is for a corresponding one of the sub-images displayed on the computing device when the user request is received and indicates a corresponding likelihood that the corresponding one of the sub-images is a focus of the user request;
selecting a particular sub-image, of the sub-images, based on the confidence scores;
generating, based on performing image recognition on the particular sub-image, a plurality of labels, wherein each of the labels represents attributes of at least one entity included in a separate portion of the sub-image;
generating, for each of the labels, a corresponding label confidence score indicating a likelihood that the corresponding label corresponds to the separate portion of the sub-image that is relevant to the user request;
selecting a subset of the labels based on the corresponding label confidence scores;
generating, based on the subset of the labels and the data indicating the user request, a search request; and
providing, for output at the computing device, one or more search results responsive to the search request.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the search request based on the labels and the data indicating the user request comprises:
generating, based on the labels and the data indicating the user request, multiple candidate search requests, including the search request;
accessing historical user request data of a plurality of users;
comparing the multiple candidate search requests to the historical user request data; and
selecting the search request based on the comparison.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
identifying metadata associated with the particular sub-image;
wherein generating the labels is further based on the metadata associated with the particular sub-image.

### Claim 4 (depends on 1)

The method of claim 1, wherein generating the search request is further based on the corresponding label confidence scores for the subset of the labels.

### Claim 5

A computing device comprising:
a display;
one or more processors; and
one or more storage devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform actions comprising:
identifying an image that is displayed at the display of the computing device;
receiving data indicating a user request at the computing device, wherein the user request is received while the image is displayed at the display;
processing the image to identify a plurality of sub-images included in the image and displayed at the display when the user request is received, wherein each sub-image of the plurality of sub-images included in the image comprises a separate portion of the image;
generating, based on the data indicating the user request, a plurality of confidence scores, wherein each of the confidence scores is for a corresponding one of the sub-images displayed at the display when the user request is received and indicates a corresponding likelihood that the corresponding one of the sub-images is a focus of the user request;
selecting a particular sub-image, of the sub-images, based on the confidence scores;
generating, based on performing image recognition on the particular sub-image, a plurality of labels, wherein each of the labels represents attributes of at least one entity included in a separate portion of the sub-image;
generating, for each of the labels, a corresponding label confidence score indicating a likelihood that the corresponding label corresponds to the separate portion of the sub-image that is relevant to the user request;
selecting a subset of the labels based on the corresponding label confidence scores;
generating, based on the subset of the labels and the data indicating the user request, a search request; and
providing, for output at the computing device, one or more search results responsive to the search request.

### Claim 6 (depends on 5)

The computing device of claim 5, wherein generating the search request based on the labels and the data indicating the user request comprises:
generating, based on the labels and the data indicating the user request, multiple candidate search requests, including the search request;
accessing historical user request data of a plurality of users;
comparing the multiple candidate search requests to the historical user request data; and
selecting the search request based on the comparison.

### Claim 7 (depends on 5)

The computing device of claim 5, comprising further instructions that, when executed by the one or more processors, cause the one or more processors to perform further actions comprising:
identifying metadata associated with the particular sub-image;
wherein generating the labels is further based on the metadata associated with the particular sub-image.

### Claim 8 (depends on 5)

The computing device of claim 5, wherein generating the search request is further based on the corresponding label confidence scores for the subset of the labels.

### Claim 9

One or more non-transitory computer readable storage media storing thereon a computer program comprising instructions that when executed by a computer system, cause the computer system to perform operations including:
identifying an image that is displayed at a computing device;
receiving data indicating a user request at the computing device, wherein the user request is received while the image is displayed on the computing device;
processing the image to identify a plurality of sub-images included in the image and displayed on the computing device when the user request is received, wherein each sub-image of the plurality of sub-images included in the image comprises a separate portion of the image;
generating, based on the data indicating the user request, a plurality of confidence scores, wherein each of the confidence scores is for a corresponding one of the sub-images displayed on the computing device when the user request is received and indicates a corresponding likelihood that the corresponding one of the sub-images is a focus of the user request;
selecting a particular sub-image, of the sub-images, based on the confidence scores;
generating, based on performing image recognition on the particular sub-image, a plurality of labels, wherein each of the labels represents attributes of at least one entity included in a separate portion of the sub-image;
generating, for each of the labels, a corresponding label confidence score indicating a likelihood that the corresponding label corresponds to the separate portion of the sub-image that is relevant to the user request;
selecting a subset of the labels based on the corresponding label confidence scores;
generating, based on the subset of the labels and the data indicating the user request, a search request; and
providing, for output at the computing device, one or more search results responsive to the search request.

### Claim 10 (depends on 9)

The one or more non-transitory computer readable storage media of claim 9, wherein generating the search request based on the labels and the data indicating the user request comprises:
generating, based on the labels and the data indicating the user request, multiple candidate search requests, including the search request;
accessing historical user request data of a plurality of users;
comparing the multiple candidate search requests to the historical user request data; and
selecting the search request based on the comparison.

### Claim 11 (depends on 9)

The one or more non-transitory computer readable storage media of claim 9, comprising further instructions that, when executed by the one or more processors, cause the one or more processors to perform further actions comprising:
identifying metadata associated with the particular sub-image;
wherein generating the labels is further based on the metadata associated with the particular sub-image.

### Claim 12 (depends on 9)

The one or more non-transitory computer readable storage media of claim 9, wherein generating the search request is further based on the corresponding label confidence scores for the subset of the labels.