# US 2017/0371885 A1 · Contextual voice search suggestions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0371885 A1
- Document ID: US20170371885A1
- Status: Published application
- Kind: A1
- Application number: 15/194,153
- Filed: 2016-06-27
- Published: 2017-12-28
- Earliest claimed: 2016-06-27
- Assignees: Google Inc.
- Applicants: Google Inc.
- Inventors: Vikram Aggarwal; Aparna Rajan Ramchandran
- Web page: https://patentgrep.com/patent/US20170371885A1

## CPC

- G06F 16/3322: {using system suggestions (G06F16/3325 takes precedence)}
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 9/453: {Help systems}
- G06F 16/313: {Selection or weighting of terms for indexing}
- G06F 16/334: Query execution (filtering based on additional data G06F16/335)
- G06F 16/35: Clustering; Classification
- G06F 40/279: Recognition of textual entities
- G06F 40/30: Semantic analysis
- G06V 30/224: of printed characters having additional code marks or containing code marks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods, systems, and apparatus for receiving user input that invokes digital assistant functionality; obtaining screen data indicating content displayed on a screen of the computing device; determining a classification for an entity referenced in the content indicated by the screen data; determining a suggested request that refers to the entity based on the determined classification; and providing the suggested request in response to receiving the user input that invokes the digital assistant functionality.

## Claims

### Claim 1

A method performed by one or more computing devices, the method comprising:
receiving, by the one or more computing devices, user input that invokes digital assistant functionality;
obtaining, by the one or more computing devices, screen data indicating content displayed on a screen of the computing device;
determining, by the one or more computing devices, a classification for an entity referenced in the content indicated by the screen data;
determining, by the one or more computing devices, a suggested request that refers to the entity based on the determined classification; and
providing, by the one or more computing devices, the suggested request in response to receiving the user input that invokes the digital assistant functionality.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining the classification comprises selecting an entity type for the entity from among a plurality of entity types.

### Claim 3 (depends on 1)

The method of claim 1, wherein determining the suggested request comprises:
determining a pronoun for the entity; and
generating a suggested query that includes the pronoun to refer to the entity referenced in the content indicated by the screen data.

### Claim 4 (depends on 3)

The method of claim 3, determining the pronoun comprises:
accessing mapping data that indicates pronouns for a plurality of entities; and
selecting a pronoun that the mapping data indicates corresponding to the entity.

### Claim 5 (depends on 1)

The method of claim 1, wherein determining the suggested request comprises:
identifying one or more query types associated with the classification;
selecting a particular query type from the one or more query types; and
determining a suggested query of the particular query type.

### Claim 6 (depends on 1)

The method of claim 1, further comprising determining one or more attributes associated with the entity type, and determining a query that asks about one of the attributes.

### Claim 7 (depends on 1)

The method of claim 1, wherein determining the suggested request comprises determining a suggested request to initiate communication with a person or to navigate to a location.

### Claim 8 (depends on 1)

The method of claim 1, further comprising recognizing one or more terms displayed on the screen of the computing device using optical character recognition on the screen data; and
wherein determining the classification comprises determining a semantic classification for the recognized one or more terms.

### Claim 9 (depends on 8)

The method of claim 8, wherein recognizing the one or more terms comprises recognizing multiple different terms;
wherein the method further comprises ranking the recognized multiple different terms; and
wherein determining the suggested request comprises determining a suggested request that refers to the entity corresponding to the highest-ranking term.

### Claim 10 (depends on 8)

The method of claim 8, further comprising determining text formatting characteristics of the multiple different terms indicated by the screen data;
wherein ranking the recognized multiple different terms comprises ranking the multiple different terms based at least in part on the determined formatting characteristics.

### Claim 11 (depends on 8)

The method of claim 8, wherein ranking the recognized multiple different terms comprises ranking the multiple different terms based at least in part on (i) classifications assigned to the multiple different terms, (ii) sizes at which the multiple different terms are displayed, (iii) frequencies that the multiple different terms are displayed on the screen, (iv) quantities of actions that can be generated for the multiple different terms, or (v) measures of previous suggestions relating to the multiple different terms that were provided and acted upon by a user.

### Claim 12 (depends on 1)

The method of claim 1, further comprising determining that the screen references only a single entity or only a single entity of a particular entity type; and
wherein providing the suggested request is performed in response to determining only a single entity or only a single entity of a particular entity type.

### Claim 13

A system comprising:
one or more computing devices and more or more storage devices storing instructions that are operable, when executed by the one or more computing devices, to cause the one or more computing devices to perform operations comprising:
receiving, by the one or more computing devices, user input that invokes digital assistant functionality;
obtaining, by the one or more computing devices, screen data indicating content displayed on a screen of the computing device;
determining, by the one or more computing devices, a classification for an entity referenced in the content indicated by the screen data;
determining, by the one or more computing devices, a suggested request that refers to the entity based on the determined classification; and
providing, by the one or more computing devices, the suggested request in response to receiving the user input that invokes the digital assistant functionality.

### Claim 14 (depends on 13)

The system of claim 13, wherein determining the classification comprises selecting an entity type for the entity from among a plurality of entity types.

### Claim 15 (depends on 13)

The system of claim 13, wherein determining the suggested request comprises:
determining a pronoun for the entity; and
generating a suggested query that includes the pronoun to refer to the entity referenced in the content indicated by the screen data.

### Claim 16 (depends on 15)

The system of claim 15, determining the pronoun comprises:
accessing mapping data that indicates pronouns for a plurality of entities; and
selecting a pronoun that the mapping data indicates corresponding to the entity.

### Claim 17 (depends on 13)

The system of claim 13, wherein determining the suggested request comprises:
identifying one or more query types associated with the classification;
selecting a particular query type from the one or more query types; and
determining a suggested query of the particular query type.

### Claim 18 (depends on 13)

The system of claim 13, further comprising determining one or more attributes associated with the entity type, and determining a query that asks about one of the attributes.

### Claim 19 (depends on 13)

The system of claim 13, wherein determining the suggested request comprises determining a suggested request to initiate communication with a person or to navigate to a location.

### Claim 20

One or more non-transitory computer-readable storage media encoded with a software program, the program comprising instructions that, if executed by one or more computing devices, cause the one or more computing devices to perform operations comprising:
receiving, by the one or more computing devices, user input that invokes digital assistant functionality;
obtaining, by the one or more computing devices, screen data indicating content displayed on a screen of the computing device;
determining, by the one or more computing devices, a classification for an entity referenced in the content indicated by the screen data;
determining, by the one or more computing devices, a suggested request that refers to the entity based on the determined classification; and
providing, by the one or more computing devices, the suggested request in response to receiving the user input that invokes the digital assistant functionality.
