# US 2023/0066100 A1 · Determination and visual display of spoken menus for calls

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0066100 A1
- Document ID: US20230066100A1
- Status: Published application
- Kind: A1
- Application number: 17/540,895
- Filed: 2021-12-02
- Published: 2023-03-02
- Earliest claimed: 2021-08-24
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Joseph Joseph CHERUKARA; Amit SIVAN; Andrew George SHEBANOW; Yoav TZUR; Felix WANG; Yaniv LEVIATHAN
- Web page: https://patentgrep.com/patent/US20230066100A1

## CPC

- H04M 3/4936: {Speech interaction details (speech recognition per se G10L15/00)}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- H04M 1/72469: for operating the device by selecting functions from two or more displayed items, e.g. menus or icons
- H04M 3/42059: {Making use of the calling party identifier}
- H04M 3/4286: {Notifying a held subscriber when his held call is removed from hold}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2015/225: {Feedback of the input speech}
- H04M 3/493: Interactive information services, e.g. directory enquiries {; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals}
- H04M 2201/40: using speech recognition
- H04M 2201/42: Graphical user interfaces
- H04M 2203/2038: Call context notifications
- H04M 2203/251: where a voice mode or a visual mode can be used interchangeably
- H04M 2203/252: where a voice mode is enhanced with visual information
- H04M 2203/551: Call history
- H04M 2250/74: with voice recognition means
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Implementations relate to determination and visual display of spoken menus for calls. In some implementations, a computer-implemented method includes receiving audio data output in a call between a call device and a device associated with a target entity. The audio data includes speech indicating one or more selection options for a user of the call device to navigate through a call menu provided by the target entity in the call. Text is determined by programmatically analyzing the audio data, the text representing the speech. The selection options are determined based on programmatically analyzing at least one of the text or the audio data. At least a portion of the text is displayed by the call device during the call, as one or more visual options that correspond to the selection options. The visual options are each selectable via user input to cause corresponding navigation through the call menu.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving audio data in a call between a call device and a device associated with a target entity, the audio data including speech indicating one or more selection options for a user of the call device to navigate through a call menu provided by the target entity in the call;
programmatically analyzing the audio data to determine text representing the speech in the audio data;
determining the one or more selection options based on programmatically analyzing at least one of the text or the audio data; and
causing at least a portion of the text to be displayed by the call device during the call, wherein the portion of the text is displayed as one or more visual options corresponding to the one or more selection options, wherein the one or more visual options are each selectable via user input to cause corresponding navigation through the call menu.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising, in response to receiving a selection of a particular visual option of the one or more visual options, causing an indication of the selection to be sent to the device associated with the target entity, wherein the indication is one of:
a signal corresponding to a press of a key of a keypad, the key associated with the particular visual option, or
speech provided by the call device in the call, the speech comprising a designator associated with the particular visual option.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more visual options are each selectable via touch input on a touchscreen of the call device.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the audio data is first audio data, and wherein in response to receiving a selection of a particular visual option of the one or more visual options, the method further comprising:
receiving second audio data in the call, the second audio data including second speech indicating one or more second selection options for the user of the call device;
programmatically analyzing the second audio data to determine second text representing the second speech in the second audio data;
determining the one or more second selection options based on programmatically analyzing at least one of the second text or the second audio data; and
causing at least a portion of the second text to be displayed by the call device, wherein the at least a portion of the second text is displayed as one or more second visual options corresponding to the one or more second selection options, wherein the one or more second visual options are each selectable via second user input to cause corresponding navigation through the call menu.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more selection options are a plurality of selection options, and further comprising programmatically analyzing at least one of the text or the audio data to determine a hierarchical structure of the plurality of selection options in the call menu.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising, before receiving the audio data:
obtaining selection option data including the one or more selection options; and
causing the one or more visual options corresponding to the one or more selection options to be displayed by the call device before the call device receives the audio data that includes the speech indicating the one or more selection options.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the one or more selection options in the selection option data are determined by programmatically analyzing audio data received during previous calls.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein the obtained selection option data is cached in the call device prior to initiation of the call, wherein the obtained selection option data is associated with entity identifiers that have been previously called by callers in a geographic area of the call device, wherein the entity identifiers have been previously called at least a threshold number of times or have been previously called a higher number of times than other entity identifiers that are not associated with the obtained selection option data.

### Claim 9 (depends on 6)

The computer-implemented method of claim 6, further comprising:
causing a visual indicator to be displayed during the call, wherein the visual indicator highlights a particular portion of the text of the one or more visual options displayed during the call, the particular portion of the text currently being received during the call in the speech in the audio data.

### Claim 10 (depends on 6)

The computer-implemented method of claim 6, further comprising:
comparing the selection option data with the one or more selection options determined from the audio data; and
determining whether a mismatch exists between the selection option data and the one or more selection options determined from the audio data.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, further comprising, in response to determining a mismatch between the selection option data and the one or more selection options determined from the audio data, causing a notification of the mismatch to be output by the call device.

### Claim 12 (depends on 10)

The computer-implemented method of claim 10, further comprising, in response to determining a mismatch between the selection option data and the one or more selection options determined from the audio data, modifying the selection option data to match the one or more selection options determined from the audio data.

### Claim 13 (depends on 10)

The computer-implemented method of claim 10, wherein comparing the selection option data with the one or more selection options includes one of:
comparing text of the selection option data with the text of the one or more selection options; or
comparing audio data of the selection option data with the audio data received during the call.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1, further comprising:
causing the one or more selection options to be stored in at least one of: storage of the call device or storage of a remote device that communicates with the call device over a communication network; and
retrieving the one or more selection options for a next call between the call device and the target entity.

### Claim 15

A call device to display selection options for calls, the call device comprising:
a memory with instructions stored thereon;
a display device; and
at least one processor coupled to the memory, the at least one processor configured to access the instructions from the memory to perform operations comprising:
receiving audio data in a call between the call device and a device associated with a target entity, the audio data including speech indicating one or more selection options for a user of the call device to navigate through a call menu provided by the target entity in the call;
programmatically analyzing the audio data to determine text representing the speech in the audio data;
determining the one or more selection options based on programmatically analyzing at least one of the text or the audio data; and
causing at least a portion of the text to be displayed by the display device during the call, wherein the portion of the text is displayed as one or more visual options corresponding to the one or more selection options, wherein the one or more visual options are each selectable via user input to cause corresponding navigation through the call menu.

### Claim 16 (depends on 15)

The call device of claim 15, wherein the at least one processor performs further operations comprising, in response to receiving a selection of a particular visual option of the one or more visual options, causing an indication of the selection to be sent to the device associated with the target entity, wherein the indication is one of:
a signal corresponding to a press of a key of a keypad, the key associated with the particular visual option, or
speech provided by the call device in the call, the speech comprising a designator associated with the particular visual option.

### Claim 17 (depends on 15)

The call device of claim 15, wherein the at least one processor performs further operations comprising, before receiving the audio data:
obtaining selection option data including the one or more selection options and a hierarchical structure of the one or more selection options in the call menu; and
causing the one or more visual options corresponding to the one or more selection options to be displayed by the display device before the call device receives the audio data that includes the speech indicating the one or more selection options.

### Claim 18 (depends on 17)

The call device of claim 17, wherein the at least one processor performs further operations comprising:
causing a visual indicator to be displayed during the call, wherein the visual indicator highlights a particular portion of the text of the one or more visual options displayed during the call, the particular portion of the text currently being spoken during the call in the speech in the audio data.

### Claim 19 (depends on 17)

The call device of claim 17, wherein the at least one processor performs further operations comprising:
comparing the selection option data with the one or more selection options determined from the audio data; and
determining whether a mismatch exists between the selection option data and the one or more selection options determined from the audio data.

### Claim 20

A non-transitory computer readable medium having stored thereon instructions that, when executed by a processor, cause the processor to perform operations comprising:
receiving audio data in a call between a call device and a device associated with a target entity, the audio data including speech indicating one or more selection options for a user of the call device to navigate through a call menu provided by the target entity;
programmatically analyzing the audio data to determine text representing the speech in the audio data;
determining the one or more selection options based on programmatically analyzing at least one of the text or the audio data; and
causing at least a portion of the text to be displayed by the call device during the call, wherein the portion of the text is displayed as one or more visual options corresponding to the one or more selection options, wherein the one or more visual options are each selectable via user input to cause corresponding navigation through the call menu.
