# US 2022/0044684 A1 · Automated calling system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044684 A1
- Document ID: US20220044684A1
- Status: Published application
- Kind: A1
- Application number: 17/505,913
- Filed: 2021-10-20
- Published: 2022-02-10
- Earliest claimed: 2019-09-24
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Asaf Aharoni; Arun Narayanan; Nir Shabat; Parisa Haghani; Galen Tsai Chuang; Yaniv LEVIATHAN; Neeraj Gaur; Pedro J. Moreno Mengibar; Rohit Prakash Prabhavalkar; Zhongdi Qu; Austin Severn Waters; Tomer Amiaz; Michiel A.U. Bacchiani
- Web page: https://patentgrep.com/patent/US20220044684A1

## CPC

- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06F 40/30: Semantic analysis
- G10L 15/32: Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
- H04M 1/02: Constructional features of telephone sets
- H04M 1/663: Preventing unauthorised calls to a telephone set
- H04M 3/4286: {Notifying a held subscriber when his held call is removed from hold}
- H04M 3/4936: {Speech interaction details (speech recognition per se G10L15/00)}
- H04M 3/5191: {interacting with the Internet}
- G10L 15/1822: {Parsing for meaning understanding}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for an automated calling system are disclosed. In one aspect, a method includes the actions of receiving audio data of an utterance spoken by a user who is having a telephone conversation with a bot. The actions further include determining a context of the telephone conversation. The actions further include determining a user intent of a first previous portion of the telephone conversation spoken by the user and a bot intent of a second previous portion of the telephone conversation outputted by a speech synthesizer of the bot. The actions further include, based on the audio data of the utterance, the context of the telephone conversation, the user intent, and the bot intent, generating synthesized speech of a reply by the bot to the utterance. The actions further include, providing, for output, the synthesized speech.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving, by a server, and from a client device of a user, a request to perform a task, wherein the task includes causing a bot hosted at the server to initiate a telephone call with an entity;
initiating, by the server, the telephone call with the entity to perform the task; and
based on a telephone conversation conducted during the telephone call:
receiving, by the server, audio data that captures a spoken utterance provided by a human representative associated with the entity;
determining, by the server, a user intent of a first previous portion of the telephone conversation associated with the human representative and a bot intent of a second previous portion of the telephone conversation associated with the bot, wherein the first previous portion of the telephone conversation occurred prior to receiving the audio data of the utterance, and wherein the second previous portion of the telephone conversation also occurred prior to receiving the audio data of the utterance;
generating, by the server, and based on at least the audio data that captures the spoken utterance, the user intent, and the bot intent, synthesized speech capturing a reply to the spoken utterance; and
causing, by the server, the synthesized speech to be provided for audible presentation to the human representative.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
based on the telephone conversation conducted during the telephone call:
determining, by the server, a context of the telephone conversation,
wherein generating the synthesized speech capturing the reply to the spoken utterance is further based on the context of the telephone conversation.

### Claim 3 (depends on 2)

The method of claim 2, wherein the context of the telephone conversation comprises one or more of: an identity of the entity, a time the telephone call is initiated, an entity location associated with the entity, or a user location associated with the user.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
based on the telephone conversation conducted during the telephone call:
determining, by the server, whether the task has been completed; and
in response to determining that the task has been completed:
causing, by the server, the telephone call with the entity to be terminated.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
in response to determining that the task has not been completed:
continuing, by the server, the telephone call with the entity.

### Claim 6 (depends on 1)

The method of claim 1, wherein the server bypasses performance of speech recognition on the spoken utterance.

### Claim 7 (depends on 1)

The method of claim 1, wherein generating the synthesized speech capturing the reply to the spoken utterance based on at least the audio data that captures the spoken utterance, the user intent, and the bot intent comprises:
processing, using a machine learning model, at least the audio data that captures the spoken utterance, the user intent, and the bot intent to generate output;
determining, based on the output, an additional bot intent associated with the reply to the spoken utterance; and
generating, based on the additional bot intent associated with the reply associated with the spoken utterance, the synthesized speech capturing the reply to the spoken utterance.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
processing, using the machine learning model, and along with the audio data that captures the spoken utterance, the user intent, and the bot intent, a context of the telephone conversation to generate the output.

### Claim 9 (depends on 7)

The method of claim 7, wherein the machine learning model is trained based on historical data for previous telephone conversations, and wherein the historical data for the previous telephone conversation comprises, for each previous telephone conversation, at least (i) corresponding previous first speaker intents associated with a first speaker determined based on corresponding first portions of each previous telephone conversation, (ii) corresponding previous second speaker intents associated with a second speaker determined based on corresponding second portions of each previous telephone conversation, (iii) corresponding previous audio data that captures most recent spoken utterance of the first speaker or the second speaker during each previous telephone conversation, and (iv) a corresponding previous intent of a corresponding previous reply to corresponding the most recent spoken utterance.

### Claim 10 (depends on 9)

The method of claim 9, wherein the historical data for the previous telephone conversation further comprises, for each previous telephone conversation, (v) a corresponding previous context for each previous telephone conversation.

### Claim 11

A system comprising:
at least one processor; and
memory storing instructions that, when executed, cause the at least one processor to perform operations, the operations comprising:
receiving, from a client device of a user, a request to perform a task, wherein the task includes causing a bot hosted at the server to initiate a telephone call with an entity;
initiating the telephone call with the entity to perform the task; and
based on a telephone conversation conducted during the telephone call:
receiving audio data that captures a spoken utterance provided by a human representative associated with the entity;
determining a context of the telephone conversation;
determining a user intent of a first previous portion of the telephone conversation associated with the human representative and a bot intent of a second previous portion of the telephone conversation associated with the bot, wherein the first previous portion of the telephone conversation occurred prior to receiving the audio data of the utterance, and wherein the second previous portion of the telephone conversation also occurred prior to receiving the audio data of the utterance;
generating, based on at least the audio data that captures the spoken utterance, the user intent, and the bot intent, synthesized speech capturing a reply to the spoken utterance; and
causing the synthesized speech to be provided for audible presentation to the human representative.

### Claim 12 (depends on 11)

The system of claim 11, the operations further comprising:
based on the telephone conversation conducted during the telephone call:
determining a context of the telephone conversation,
wherein generating the synthesized speech capturing the reply to the spoken utterance is further based on the context of the telephone conversation.

### Claim 13 (depends on 12)

The system of claim 12, wherein the context of the telephone conversation comprises one or more of: an identity of the entity, a time the telephone call is initiated, an entity location associated with the entity, or a user location associated with the user.

### Claim 14 (depends on 11)

The system of claim 11, the operations further comprising:
based on the telephone conversation conducted during the telephone call:
determining whether the task has been completed; and
in response to determining that the task has been completed:
causing the telephone call with the entity to be terminated.

### Claim 15 (depends on 14)

The system of claim 14, the operations further comprising:
in response to determining that the task has not been completed:
continuing the telephone call with the entity.

### Claim 16 (depends on 11)

The system of claim 11, wherein the system bypasses performance of speech recognition on the spoken utterance.

### Claim 17 (depends on 11)

The method of claim 11, wherein generating the synthesized speech capturing the reply to the spoken utterance based on at least the audio data that captures the spoken utterance, the user intent, and the bot intent comprises:
processing, using a machine learning model, at least the audio data that captures the spoken utterance, the user intent, and the bot intent to generate output;
determining, based on the output, an additional bot intent associated with the reply to the spoken utterance; and
generating, based on the additional bot intent associated with the reply associated with the spoken utterance, the synthesized speech capturing the reply to the spoken utterance.

### Claim 18 (depends on 17)

The method of claim 17, the operations further comprising:
processing, using the machine learning model, and along with the audio data that captures the spoken utterance, the user intent, and the bot intent, a context of the telephone conversation to generate the output.

### Claim 19 (depends on 17)

The method of claim 17, wherein the machine learning model is trained based on historical data for previous telephone conversations, and wherein the historical data for the previous telephone conversation comprises, for each previous telephone conversation, (i) a corresponding previous context for each previous telephone conversation, (ii) corresponding previous first speaker intents associated with a first speaker determined based on corresponding first portions of each previous telephone conversation, (iii) corresponding previous second speaker intents associated with a second speaker determined based on corresponding second portions of each previous telephone conversation, (iv) corresponding previous audio data that captures most recent spoken utterance of the first speaker or the second speaker during each previous telephone conversation, (v) a corresponding previous intent of a corresponding previous reply to corresponding the most recent spoken utterance.

### Claim 20

A non-transitory computer-readable storage medium storing instructions that, when executed cause at least one processor to perform operations, the operations comprising:
receiving, by a server, and from a client device of a user, a request to perform a task, wherein the task includes causing a bot hosted at the server to initiate a telephone call with an entity;
initiating, by the server, the telephone call with the entity to perform the task; and
based on a telephone conversation conducted during the telephone call:
receiving, by the server, audio data that captures a spoken utterance provided by a human representative associated with the entity;
determining, by the server, a user intent of a first previous portion of the telephone conversation associated with the human representative and a bot intent of a second previous portion of the telephone conversation associated with the bot, wherein the first previous portion of the telephone conversation occurred prior to receiving the audio data of the utterance, and wherein the second previous portion of the telephone conversation also occurred prior to receiving the audio data of the utterance;
generating, by the server, and based on at least the audio data that captures the spoken utterance, the user intent, and the bot intent, synthesized speech capturing a reply to the spoken utterance; and
causing, by the server, the synthesized speech to be provided for audible presentation to the human representative.
