# US 2020/0404097 A1 · Semiautomated relay method and apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0404097 A1
- Document ID: US20200404097A1
- Status: Published application
- Kind: A1
- Application number: 16/537,196
- Filed: 2019-08-09
- Published: 2020-12-24
- Earliest claimed: 2014-02-28
- Assignees: Ultratec, Inc.
- Applicants: Ultratec, Inc.
- Inventors: Robert M. Engelke; Kevin R. Colwell; Christopher Engelke
- Web page: https://patentgrep.com/patent/US20200404097A1

## CPC

- H04M 3/42391: {where the subscribers are hearing-impaired persons, e.g. telephone devices for the deaf}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 25/48: specially adapted for particular use
- H04M 1/2475: {for a hearing impaired user}
- H04M 1/72478: for hearing-impaired users
- H04W 4/06: Selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS]; Services to user groups; One-way selective calling services
- H04W 4/12: Messaging; Mailboxes; Announcements
- H04W 4/16: Communication-related supplementary services, e.g. call-transfer or call-hold
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 21/10: Transforming into visible information
- H04M 1/72433: for voice messaging, e.g. dictaphones (for answering incoming calls H04M1/64)
- H04M 3/53366: {Message disposing or creating aspects}
- H04M 2201/18: Comparators
- H04M 2201/40: using speech recognition
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

A method and system to transcribe communications the method comprising the steps of obtaining an audio message originating at a first device during a voice communication session between the first device and a second device, providing the audio message to a first speech recognition system to generate a first transcript of the audio message, directing the first transcript to the second device, in response to obtaining an indication that indicates a quality of the first transcript is below a quality threshold, using a second speech recognition system to generate a second transcript based on the audio message while continuing to provide the audio data to the first speech recognition system to generate the first transcript and, in response to occurrence of an event that indicates the second transcript is to be directed to the second device, directing the second transcript to the second device instead of directing the first transcript.

## Claims

### Claim 1

A method to transcribe communications, the method comprising:
obtaining an audio message originating at a first device during a voice communication session between the first device and a second device;
providing the audio message to a first automated speech recognition system that works independent of human interaction to generate a first transcript of the audio message;
directing the first transcript to the second device;
in response to obtaining an indication that indicates a quality of the first transcript is below a quality threshold and while continuing to provide the audio message to the first automated speech recognition system to generate the first transcript, the method including:
providing the audio message to a second speech recognition system;
broadcasting, by the second speech recognition system, audio based on the audio message to a call assistant;
obtaining, by the second speech recognition system, a second audio message based on a re-voicing of the broadcast audio; and
generating, by the second speech recognition system a second transcript using the second audio message; and
in response to occurrence of an event, directing the second transcript to the second device instead of directing the first transcript to the second device and ceasing providing the audio message to the first automated speech recognition system.

### Claim 2 (depends on 1)

The method of claim 1 further comprising:
obtaining a confidence score of the first transcript from the first automated speech recognition system; and
using the confidence score to assess transcription accuracy and generate the indication that indicates quality.

### Claim 3 (depends on 1)

The method of claim 1 wherein the indication that indicates quality is obtained from the second device.

### Claim 4 (depends on 1)

The method of claim 1 wherein the event includes the second transcript including at least one captioned word based on the second audio message.

### Claim 5 (depends on 1)

At least one memory device storing at least one software program including instructions that when executed by at least one processor cause or direct a system to perform the method of claim 1.

### Claim 6

A method to transcribe communications, the method comprising:
obtaining an audio message originating at a first device during a voice communication session between the first device and a second device;
providing the audio message to a first speech recognition system to generate a first transcript of the audio message;
directing the first transcript to the second device;
in response to obtaining an indication that indicates a quality of the first transcript is below a quality threshold, providing the audio message to a second speech recognition system to generate a second transcript based on the audio message while continuing to provide the audio message to the first speech recognition system to generate the first transcript and continuing to direct the first transcript to the second device; and
in response to occurrence of an event that indicates the second transcript is to be directed to the second device instead of the first transcript, directing the second transcript to the second device instead of directing the first transcript.

### Claim 7 (depends on 6)

The method of claim 6 wherein the first speech recognition system and the second speech recognition system are automated speech recognition systems that work independent of human interaction.

### Claim 8 (depends on 6)

The method of claim 6 wherein the first speech recognition system is an automated speech recognition system that works independent of human interaction and the generation of the second transcript by the second speech recognition system includes:
(i) broadcasting the audio message; and
(ii) obtaining a second audio message based on a re-voicing of the broadcast audio message, wherein the second transcript is generated based on the second audio message.

### Claim 9 (depends on 6)

The method of claim 6 further comprising obtaining a confidence score of the first transcript from the first speech recognition system and obtaining the indication of quality based on a comparison of the confidence score to the quality threshold.

### Claim 10 (depends on 6)

The method of claim 6 wherein the quality indication is obtained from the second device.

### Claim 11 (depends on 6)

The method of claim 6 wherein the event includes the second transcript including at least one captioned word based on the audio message.

### Claim 12 (depends on 6)

The method of claim 6 further including, in response to the event, ceasing providing the audio message to the first speech recognition system.

### Claim 13 (depends on 6)

The method of claim 6 wherein the event includes passage of a short delay.

### Claim 14

A system to transcribe communications, the system comprising:
at least one processor; and
at least one memory device linked to the processor and storing at least one software program including instructions that when executed by at least one processor cause or direct a system to perform operations comprising:
obtaining an audio message originating at a first device during a voice communication session between the first device and a second device;
providing the audio message to a first automated speech recognition system to generate a first transcript of the audio message;
directing the first transcription to the second device;
in response to obtaining an indication that indicates a quality of the first transcript is below a quality threshold, provide the audio message to a second speech recognition system to generate a second transcript based on the audio message while continuing to provide the audio message to the first speech recognition system to generate the first transcript and continuing to direct the first transcript to the second device; and
in response to an event that indicates the second transcript is to be directed to the second device instead of the first transcript where the event occurs after the audio message is provided to the second speech recognition system, direct the second transcript to the second device instead of the first transcript.

### Claim 15 (depends on 14)

The system of claim 14, wherein the first speech recognition system and the second speech recognition system are automated speech recognition systems that work independent of human interaction.

### Claim 16 (depends on 14)

The system of claim 14, wherein the first speech recognition system is an automated speech recognition system that works independent of human interaction and the generation of the second transcript by the second speech recognition system includes operations comprising:
broadcast audio based on the audio message; and
obtain a second audio message based on a re-voicing of the broadcast audio, wherein the second transcript is generated based on the second audio message.

### Claim 17 (depends on 14)

The system of claim 14, wherein the operations further comprise:
obtaining a confidence score of the first transcript from the first speech recognition system; and
obtaining the quality indication based on a comparison of the confidence score to the quality threshold.

### Claim 18 (depends on 14)

The system of claim 14, wherein the quality indication is obtained from the second device.

### Claim 19 (depends on 14)

The system of claim 14 wherein the event includes the second transcript including at least one captioned word based on the audio message.

### Claim 20 (depends on 14)

The system of claim 14, wherein the operations further comprise in response to occurrence of the event, ceasing providing the audio message to the first speech recognition system.

### Claim 21 (depends on 6)

The method of claim 6 wherein the second device includes an input device for indicating that a user of the second device desires a call assistant to assist in transcription, the method generating a quality indication upon selection of the input.

### Claim 22 (depends on 6)

The method of claim 6 further including the steps of, while directing the second transcript to the second device, continuing to provide the audio message to the first speech recognition system to generate the first transcript, comparing the first and second transcripts to identify errors in the first transcript and using those errors to train the first speech recognition system while the voice communication session is ongoing.

### Claim 23 (depends on 6)

The method of claim 6 wherein the event is the end of a delay so that the first speech recognition system and the second speech recognition system each generate text corresponding to at least a segment of the audio message.

### Claim 24 (depends on 6)

The method of claim 6 wherein a call assistant reviews the first transcript on a display screen and generates an input signal when the quality of the first transcript is below the quality threshold.

### Claim 25 (depends on 6)

The method of claim 6 wherein the audio message is duplicated so that the audio message is provided to each of the first and second speech recognition systems in parallel during at least a short duration of time.

### Claim 26

A method to transcribe communications, the method comprising:
obtaining an audio message originating at a first device during a voice communication session between the first device and a second device;
providing the audio message to a first speech recognition system to generate a first transcript of the audio message;
directing the first transcript to the second device;
in response to obtaining an indication that indicates a quality of the first transcript is below a quality threshold, using a second speech recognition system to generate a second transcript based on the audio message while continuing to provide the audio message to the first speech recognition system to generate the first transcript; and
in response to occurrence of an event that indicates the second transcript is to be directed to the second device, directing the second transcript to the second device instead of directing the first transcript.

### Claim 27 (depends on 26)

The method of claim 26 wherein the first transcript is directed to the second device until occurrence of the event that indicates that the second transcript is to be directed to the second device.

### Claim 28 (depends on 26)

The method of claim 26 further including, in response to the event, ceasing providing the audio message to the first speech recognition system.

### Claim 29 (depends on 26)

The method of claim 26 wherein the first speech recognition system and the second speech recognition system are automated speech recognition systems that work independent of human interaction.

### Claim 30 (depends on 26)

The method of claim 26 wherein at least one of the first and second speech recognition systems is an automated speech recognition systems that work independent of human interaction and the other of the first and second speech recognition system requires human interaction to generate a transcript.

### Claim 31 (depends on 1)

The method of claim 1 wherein a remote relay provides the first automated speech recognition system and the second speech recognition system, the step of providing the audio message to the first automated speech recognition system and the step of providing the audio message to the second speech recognition system including transmitting the audio message to the remote relay.

### Claim 32 (depends on 31)

The method of claim 31 wherein the step of transmitting the audio message to the remote relay includes receiving the audio message at the second device and transmitting the audio message from the second device to the remote relay.

### Claim 33 (depends on 1)

The method of claim 1 wherein the first device includes a phone device and the second device includes a captioned device.

### Claim 34 (depends on 1)

The method of claim 1 wherein the second device is a portable computing device.

### Claim 35 (depends on 1)

The method of claim 1 wherein the second device includes a speaker and a microphone and wherein the second device obtains an assisted user's voice signal and transmits that voice signal to the first device, the second device receiving the audio message from the first device and broadcasting the audio message via the speaker.

### Claim 36 (depends on 35)

The method of claim 35 wherein the step of obtaining an indication includes obtaining an indication from the second device that the quality of the first transcript is below a quality threshold.

### Claim 37 (depends on 36)

The method of claim 36 wherein the step of providing the audio message to the second speech recognition system includes transmitting the audio message to a remote relay that operates the second speech recognition system.

### Claim 38 (depends on 6)

The method of claim 6 wherein the first device provides the first speech recognition system.

### Claim 39 (depends on 38)

The method of claim 38 wherein a remote relay provides the second speech recognition system and wherein the step of providing the audio message to the second speech recognition system includes transmitting the audio message to the remote relay.

### Claim 40 (depends on 39)

The method of claim 39 wherein the second device presents at least one of the first and second transcripts for each portion of the audio signal message via a display screen.

### Claim 41 (depends on 40)

The method of claim 40 wherein the audio message signal is also presented to the second device and wherein the second device controls a speaker to broadcast the audio message signal.

### Claim 42 (depends on 6)

The method of claim 6 wherein the second device presents the first transcript as captions via a second device display screen during a first portion of the voice communication session and presents the second transcript as captions via the second device display screen during a second portion of the voice communication session.

### Claim 43 (depends on 42)

The method of claim 42 further including presenting an indication via the display screen that the first speech recognition system generated the first transcript when the first transcript is presented on the display screen and presenting an indication via the display screen that the second speech recognition system generated the second transcript when the second transcript is presented on the display screen.

### Claim 44 (depends on 43)

The method of claim 43 further including presenting a first on screen icon on the display screen that is selectable to switch from the first speech recognition engine to the second speech recognition engine while the first transcript is presented on the display screen and presenting a second on screen icon on the display screen that is selectable to switch from the second speech recognition engine to the first speech recognition engine while the second transcript is presented on the display screen.

### Claim 45 (depends on 6)

The method of claim 6 wherein the second device provides the first speech recognition system.

### Claim 46 (depends on 6)

The method of claim 6 wherein a remote relay provides each of the first and second speech recognition systems and wherein the audio message is transmitted to the relay.

### Claim 47 (depends on 6)

The method of claim 6 wherein the step of providing the audio message to a first speech recognition system includes:
(i) broadcasting the audio message to a call assistant; and
(ii) obtaining a second audio message based on a re-voicing of the broadcast audio message by the call assistant, wherein the first transcript is generated based on the second audio message.

### Claim 48 (depends on 47)

The method of claim 47 wherein the second audio message is provided to a first automated speech recognition system that is trained to a voice of the call assistant.

### Claim 49 (depends on 48)

The method of claim 48 wherein the second speech recognition system includes a second automated speech recognition system.

### Claim 50 (depends on 49)

The method of claim 49 wherein the second speech recognition system trains during the voice communication session to a voice of a user speaking the audio message.

### Claim 51 (depends on 6)

The method of claim 6 wherein the first transcript is presented via a second device display prior to occurrence of the event and the second transcript is presented via the second device display subsequent to occurrence of the event.

### Claim 52 (depends on 51)

The method of claim 51 wherein the second device includes a speaker and wherein the audio message is provided to the second device and is broadcast as a voice message via the speaker.

### Claim 53 (depends on 52)

The method of claim 52 wherein transcript captions presented via the display lag the voice message, the method further including presenting an indication via the display screen indicating a magnitude of a lag period between a most recently broadcast voice message and a most recently presented transcript caption.

### Claim 54 (depends on 53)

The method of claim 53 wherein the lag period is expressed in seconds.

### Claim 55 (depends on 6)

The method of claim 6 wherein the audio message is a first audio message and wherein the method further includes obtaining a second audio message originating at the second device during the voice communication session and providing the second audio message to a speech recognition system to generate a third transcript associated with the second audio message.

### Claim 56 (depends on 55)

The method of claim 55 further including presenting one of the first and second transcripts as well as the third transcript via a second device display screen.

### Claim 57 (depends on 56)

The method of claim 56 wherein the first and second transcripts are presented in a first column on the second device display screen and the third transcript is presented in a second column spaced from the first column on the second device display screen.

### Claim 58 (depends on 6)

The method of claim 6 wherein the second device includes a display device and a separate audio device, at least one of the transcripts presented as captions via the display device and the audio message broadcast via the audio device.

### Claim 59 (depends on 6)

The method of claim 6 further including the steps of, subsequent to directing the second transcript to the second device, continuing to generate the first transcript via the first speech recognition system, using differences between the first and second transcripts to train the first speech recognition system, determining when a quality of the first transcript is above a second quality threshold and, when the quality of the first transcript is above a second quality threshold, directing the first transcript to the second device instead of directing the second transcript.

### Claim 60 (depends on 6)

The method of claim 6 wherein a remote relay provides the first speech recognition system and the second speech recognition system, the step of providing the audio message to the first speech recognition system and the step of providing the audio message to the second speech recognition system including transmitting the audio message to the remote relay.

### Claim 61 (depends on 60)

The method of claim 60 wherein the step of transmitting the audio message to the remote relay includes receiving the audio message at the second device and transmitting the audio message from the second device to the remote relay.

### Claim 62 (depends on 26)

The method of claim 26 wherein the first device includes a phone device and the second device includes an assisted user's captioned device.

### Claim 63 (depends on 26)

The method of claim 26 wherein the second device includes a speaker and a microphone and wherein the second device obtains an assisted user's voice signal and transmits that voice signal to the first device, the second device receiving the audio message from the first device and broadcasting the audio message via the speaker.

### Claim 64 (depends on 63)

The method of claim 63 wherein the step of obtaining an indication includes obtaining an indication from the second device that the quality of the first transcript is below a quality threshold.

### Claim 65 (depends on 64)

The method of claim 64 wherein the step of providing the audio message to the second speech recognition system includes transmitting the audio message to a remote relay that operates the second speech recognition system.

### Claim 66 (depends on 26)

The method of claim 26 wherein the first device provides the first speech recognition system.

### Claim 67 (depends on 66)

The method of claim 66 wherein a remote relay provides the second speech recognition system and wherein the step of providing the audio message to the second speech recognition system includes transmitting the audio message to the remote relay.

### Claim 68 (depends on 26)

The method of claim 26 wherein the second device presents the first transcript as captions via a second device display screen during a first portion of the voice communication session prior to occurrence of the event and presents the second transcript as captions via the second device display screen during a second portion of the voice communication session after occurrence of the event.

### Claim 69 (depends on 26)

The system of claim 26 wherein a remote relay provides each of the first and second speech recognition systems and wherein the audio message is transmitted to the relay.

### Claim 70 (depends on 26)

The method of claim 26 wherein the step of providing the audio message to a first speech recognition system includes:
(i) broadcasting the audio message to a call assistant; and
(ii) obtaining a second audio message based on a re-voicing of the broadcast audio message by the call assistant, wherein the first transcript is generated based on the second audio message.

### Claim 71 (depends on 70)

The method of claim 70 wherein the second speech recognition system includes an automated speech recognition system.

### Claim 72 (depends on 6)

The method of claim 6 wherein the audio message is a first audio message and wherein the method further includes obtaining a second audio message originating at the second device during the voice communication session and providing the second audio message to a third speech recognition system to generate a third transcript associated with the second audio message.

### Claim 73 (depends on 72)

The method of claim 72 wherein each of the first and third speech recognition systems includes an automated recognition system that operates independent of call assistant input.