# US 2023/0274744 A1 · Semiautomated relay method and apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274744 A1
- Document ID: US20230274744A1
- Status: Published application
- Kind: A1
- Application number: 18/144,644
- Filed: 2023-05-08
- Published: 2023-08-31
- Earliest claimed: 2014-02-28
- Assignees: Ultratec, Inc.
- Applicants: Ultratec, Inc.
- Inventors: Robert M. Engelke; Kevin R. Colwell; Christopher Engelke
- Web page: https://patentgrep.com/patent/US20230274744A1

## CPC

- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 15/01: Assessment or evaluation of speech recognition systems
- H04M 1/2475: {for a hearing impaired user}
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 25/48: specially adapted for particular use
- G10L 25/60: for measuring the quality of voice signals
- H04M 3/42391: {where the subscribers are hearing-impaired persons, e.g. telephone devices for the deaf}
- H04M 2201/40: using speech recognition
- H04M 2201/60: Medium conversion
- H04M 2203/2061: Language aspects
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)

## Abstract

A method to transcribe communications includes the steps of obtaining a plurality of hypothesis transcriptions of a voice signal generated by a speech recognition system, determining consistent words that are included in at least first and second of the plurality of hypothesis transcriptions, in response to determining the consistent words, providing the consistent words to a device for presentation of the consistent words to an assisted user, and presenting the consistent words via a display screen on the device, wherein a rate of the presentation of the words on the display screen is variable.

## Claims

### Claim 1

A method of providing captions to an assisted user (AU) using an AU device to communicate via a voice call with a hearing user (HU) using an HU device, the AU device including a display screen, the method comprising the steps of:
receiving HU voice signal at the AU device;
using an automated speech recognition (ASR) engine run by the AU device to generate ASR captions for at least a subset of words in the HU voice signal;
identifying any instance of a line check words (LCWs) that occur in the ASR captions;
immediately presenting each LCW via the display screen immediately upon identification;
transmitting the HU voice signal to a relay for transcription;
receiving relay generated captions associated with the HU voice signal; and
presenting the relay generated captions via the display in line with the LCWs.

### Claim 2 (depends on 1)

The method of claim 1 wherein the ASR engine is programmed to only identify LCWs and not other words uttered as part of the HU voice signal.

### Claim 3 (depends on 1)

The method of claim 1 wherein LCWs are presented via the display in a manner that visually distinguishes those words from other caption words presented on the display.

### Claim 4 (depends on 1)

The method of claim 1 further including, upon presenting an LCW via the display, transmitting a message to the HU indicating that transcription is lagging.

### Claim 5 (depends on 1)

The method of claim 1 further including, upon presenting an LCW via the display, broadcasting a message to the AU indicating that transcription is lagging.

### Claim 6 (depends on 1)

The method of claim 1 wherein the step of presenting the LCWs via the display includes presenting the LCWs in line with captions being generated at the relay with at least one intermediate blank representing words that have yet to be received back from the relay.

### Claim 7 (depends on 1)

The method of claim 1 wherein the step of presenting the LCWs via the display includes presenting the LCWs in line with captions being generated at the relay with a separate intermediate blank representing each word that has yet to be received back from the relay.

### Claim 8 (depends on 7)

The method of claim 7 wherein the relay captions are used to replace the intermediate blanks as the relay captions are generated.

### Claim 9 (depends on 1)

The method of claim 1 wherein the step of presenting the LCWs via the display includes presenting the LCWs in line with captions being generated at the relay with some indication representing words that have yet to be received back from the relay.

### Claim 10 (depends on 9)

The method of claim 9 wherein the ASR generates captions for the entire HU voice signal and only presents LCWs via the display.

### Claim 11 (depends on 1)

The method of claim 1 further including estimating the captioning delay and presenting an indication of the captioning delay via the display.

### Claim 12 (depends on 1)

The method of claim 1 further including presenting a catch-up option to the AU via the AU device and skipping the relay ahead to a current HU voice signal upon selection of the catch-up option.

### Claim 13 (depends on 1)

The method of claim 1 wherein the ASR engine generates captions for the entire HU voice signal and presents all ASR captions immediately upon being generated, the method further including visually distinguishing the LCWs on the display from non-LCWs.

### Claim 14 (depends on 13)

The method of claim 13 wherein the step of presenting relay generated captions via the display includes using the relay captions to make error corrections in line within the ASR captions presented on the display.

### Claim 15 (depends on 14)

The method of claim 14 further including visually distinguishing any text that corresponds to an error correction via the display.

### Claim 16 (depends on 1)

The method of claim 1 wherein the HU voice signal is transmitted to the relay immediately upon being received.

### Claim 17 (depends on 1)

The method of claim 1 wherein the AU device includes a speaker, the method further including broadcasting the HU voice signal to the AU via teh speaker.

### Claim 18 (depends on 17)

The method of claim 17 wherein the HU voice signal is delayed prior to broadcasting to the AU via the speaker so that the broadcast voice signal is more temporally aligned with presentation of the relay captions.

### Claim 19

A method of providing captions to an assisted user (AU) using an AU device to communicate via a voice call with a hearing user (HU) using an HU device, the AU device including a display screen, the method comprising the steps of:
receiving HU voice signal at the AU device;
using an automated speech recognition (ASR) engine run by the AU device to generate ASR captions for at least a subset of words in the HU voice signal;
identifying line check words (LCWs) that occur in the ASR captions; and
presenting an intermediate blank representing each non-LCW and LCWs in line with the intermediate blanks via the display.

### Claim 20

A method of providing captions to an assisted user (AU) using an AU device to communicate via a voice call with a hearing user (HU) using an HU device, the AU device including a display screen, the method comprising the steps of:
receiving HU voice signal at the AU device;
using an automated speech recognition (ASR) engine run by the AU device to generate ASR captions for at least a subset of words in the HU voice signal;
identifying line check words (LCWs) that occur in the ASR captions;
presenting an intermediate blank representing each non-LCW and LCWs in line with the intermediate blanks via the display;
transmitting the HU voice signal to a relay for transcription;
receiving relay generated captions associated with the HU voice signal; and
replacing the intermediate blanks with the relay generated captions via the display in line with the LCWs as the relay captions are generated.
