# US 11,443,730 B2 · Initiating synthesized speech output from a voice-controlled device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,730 B2
- Document ID: US11443730B2
- Status: Granted
- Kind: B2
- Application number: 16/774,122
- Filed: 2020-01-28
- Published: 2020-05-21
- Granted: 2022-09-13
- Earliest claimed: 2017-12-26
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Shang Qing Guo; Jonathan Lenchner
- Web page: https://patentgrep.com/patent/US11443730B2

## CPC

- G10L 13/02: Methods for producing synthetic speech; Speech synthesisers
- B25J 9/0003: {Home robots, i.e. small robots for domestic use}
- B25J 11/0005: {Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means}
- B25J 13/003: {by means of an audio-responsive input (audible safety signals B25J19/061)}
- G06V 40/166: {using acquisition arrangements}
- G06V 40/168: {Feature extraction; Face representation}
- G06V 40/172: {Classification, e.g. identification}
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 15/18: using natural language modelling
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 25/84: for discriminating voice from noise
- G10L 2015/088: {Word spotting}
- G10L 2021/02087: {the noise being separate speech, e.g. cocktail party}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system, a computer program product, and method for controlling synthesized speech output on a voice-controlled device. A sensor is used to capture an image of a face of a person. A database of previously stored images of facial features is accessed. In response to i) not recognizing the at least one person the voice-controlled device selects a first set of conversational starters; ii) recognizing the person and recognizing previous communications with the person, the voice-controlled device selects a second set of conversational starters; iii) recognizing the person and not recognizing previous communications with the person, the voice-controlled device selects a third set of conversational starters; or iv) recognizing the at least one person and recognizing previous communications with the person selecting but do not know the person's name selecting a fourth set of conversational starters. The voice controlled device outputs the selected set of conversational starters.

## Claims

### Claim 1

A computer-implemented method on a voice-controlled device for controlling synthesized speech output, the method comprising:
capturing, with a sensor, an image of a face of at least one person;
accessing a database of previously stored images of facial features;
in response to
not recognizing the at least one person then selecting a first set of conversational starters;
recognizing the at least one person and recognizing a previous communication with the person then selecting a second set of conversational starters; and
recognizing the at least one person and not recognizing a previous communication with the person then selecting a third set of conversational starters;
detecting, with at least one sensor, whether the at least one person remains within a first settable distance from the voice-controlled device; and
in response to the person being detected, then outputting synthesized speech using one of the first set, the second set, or the third set of conversational starters which have been selected.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
in response to recognizing the at least one person and recognizing previous communications with the person but do not know the person's identity then selecting a fourth set of conversational starters.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein in response to the person being detected, further includes determining if a settable period of time having elapsed, then outputting synthesized speech using one of the set of conversational starters.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the voice-controlled device is a robot.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the at least one sensor is a visible light camera sensor, an infra-red camera sensor, laser pulse-based radar sensor, an acoustical sensor, or a combination thereof.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising:
capturing audio input while outputting synthesized speech;
in response to the captured audio is
recognized as speech, pausing the output of synthesized speech; and
not recognized as speech and above a settable background noise threshold, pausing the output of synthesized speech.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, further comprising:
in response to the captured audio being not recognized as speech and a volume of the audio input below or equal to a settable background noise threshold, and the pausing of the output of synthesized speech being within a settable pause timeframe, resuming the output of synthesized speech based on the settable pause timeframe elapsing.

### Claim 8

A voice-controlled device for controlling synthesized speech output, the voice-controlled device comprising:
a processor device; and
a memory operably coupled to the processor device and storing computer-executable instructions causing:
capturing, with a sensor, an image of a face of at least one person;
accessing a database of previously stored images of facial features;
in response to
not recognizing the at least one person then selecting a first set of conversational starters;
recognizing the at least one person and recognizing a previous communication with the person then selecting a second set of conversational starters;
detecting, with at least one sensor, whether one person of the at least one person remains at a first settable distance or two or more persons of the at least one person remains within a second settable distance from the voice-controlled device; and
in response to the one person being detected, then outputting synthesized speech using one of the set of conversational starters, otherwise, in response to two or more persons being detected waiting for an attention word before outputting synthesized speech.

### Claim 9 (depends on 8)

The voice-controlled device of claim 8, further comprising:
in response to recognizing the at least one person and not recognizing a previous communication with the person then selecting a third set of conversational starters.

### Claim 10 (depends on 9)

The voice-controlled device of claim 9, further comprising:
in response to recognizing the at least one person and recognizing previous communications with the person but do not know the person's identity then selecting a fourth set of conversational starters.

### Claim 11 (depends on 8)

The voice-controlled device of claim 8, wherein in response to the person being detected, further includes determining if a settable period of time having elapsed, then outputting synthesized speech using one of the set of conversational starters.

### Claim 12 (depends on 8)

The voice-controlled device of claim 8, wherein the voice-controlled device is a robot.

### Claim 13 (depends on 8)

The voice-controlled device of claim 8, wherein the at least one sensor is a visible light camera sensor, an infra-red camera sensor, laser pulse-based radar sensor, an acoustical sensor, or a combination thereof.

### Claim 14 (depends on 8)

The voice-controlled device of claim 8, further comprising:
capturing audio input while outputting synthesized speech;
in response to the captured audio is
recognized as speech, pausing the output of synthesized speech; and
not recognized as speech and above a settable background noise threshold, pausing the output of synthesized speech based.

### Claim 15 (depends on 14)

The voice-controlled device of claim 14, further comprising:
in response to the captured audio being not recognized as speech and a volume of the audio input below or equal to a settable background noise threshold, and the pausing of the output of synthesized speech being within a settable pause timeframe, resuming the output of synthesized speech.

### Claim 16

A computer program product for controlling synthesized speech output, the computer program product comprising:
a non-transitory computer readable storage medium readable by a processing device as part of a voice-controlled device and storing program instructions for execution by the processing device, said program instructions comprising:
capturing, with a sensor, an image of a face of at least one person;
accessing a database of previously stored images of facial features;
in response to
not recognizing the at least one person then selecting a first set of conversational starters;
recognizing the at least one person and recognizing a previous communication with the person then selecting a second set of conversational starters;
detecting, with at least one sensor, whether one person of the at least one person remains at a first settable distance or two or more persons of the at least one person remains within a second settable distance from the voice-controlled device; and
in response to the one person being detected, then outputting synthesized speech using one of the set of conversational starters, otherwise, in response to two or more persons being detected waiting for an attention word before outputting synthesized speech.

### Claim 17 (depends on 16)

The computer program product of claim 16, further comprising:
in response to recognizing the at least one person and not recognizing a previous communication with the person then selecting a third set of conversational starters.

### Claim 18 (depends on 16)

The computer program product of claim 16, further comprising:
in response to recognizing the at least one person and recognizing previous communications with the person but do not know the person's identity then selecting a fourth set of conversational starters.

### Claim 19 (depends on 16)

The computer program product of claim 16, wherein in response to the person being detected, further includes determining if a settable period of time having elapsed, then outputting synthesized speech using one of the set of conversational starters.
