# US 2021/0174823 A1 · System for and Method of Converting Spoken Words and Audio Cues into Spatially Accurate Caption Text for Augmented Reality Glasses

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174823 A1
- Document ID: US20210174823A1
- Status: Published application
- Kind: A1
- Application number: 16/927,699
- Filed: 2020-07-13
- Published: 2021-06-10
- Earliest claimed: 2019-12-10
- Assignees: Spectrum Accountable Care Company
- Applicants: Spectrum Accountable Care Company
- Inventors: Barry Goldstein; Quyen Tang Kiet
- Web page: https://patentgrep.com/patent/US20210174823A1

## CPC

- H04S 7/40: {Visual indication of stereophonic sound image}
- G06T 11/60: Creating or editing images; Combining images with text
- G09G 5/02: characterised by the way in which colour is displayed {(details of colour display specific for CRTs G09G1/28; specific for flat matrix panels other than liquid crystal displays G09G3/2003; specific for liquid crystal displays G09G3/3607)}
- G09G 5/32: with means for controlling the display position {(see provisionally G09G5/42)}
- G10L 15/183: using context dependencies, e.g. language models
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 21/0232: Processing in the frequency domain
- G10L 21/10: Transforming into visible information
- G10L 25/84: for discriminating voice from noise
- H04R 1/406: {microphones}
- H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)}
- H04R 25/407: {Circuits for combining signals of a plurality of transducers}
- G06F 3/147: using display panels
- G09G 2354/00: Aspects of interface with display user
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 2021/065: {Aids for the handicapped in understanding}
- H04R 2201/401: 2D or 3D arrays of transducers
- H04S: STEREOPHONIC SYSTEMS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A wearable device with augmented reality glasses that allows a user to see the text information and a microphone array that positions the source of sound in three-dimensional space around the user. When the wearable device is worn by a hearing impaired user, the user would be able to see captioned dialogue that is spoken by those around him along with the position information of the speaker.

## Claims

### Claim 1

A method for displaying text on augmented reality glasses with a plurality of microphones, the method comprising:
capturing audio from a speaker into an audio file;
converting the audio file into a text file;
determining a position of the speaker;
if the position is out of a visual range, displaying the text file with an out of range indicator on a display screen of the augmented reality glasses; and
if the position is within the visual range, displaying the text file on the display screen of the augmented reality glasses adjacent to the position of the speaker on the display screen.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
receiving language preference from a user; and
receiving visual range setting from the user.

### Claim 3 (depends on 1)

The method of claim 1, further comprising displaying turning information if the position is out of the visual range.

### Claim 4 (depends on 1)

The method of claim 1, further comprising checking if audio conversion feature is turned on.

### Claim 5 (depends on 1)

The method of claim 1, further comprising storing the audio file and the text file into a storage unit.

### Claim 6 (depends on 1)

The method of claim 1, wherein displaying the text file with an out of range indicator on a display screen of the augmented reality glasses further comprising displaying the text file in a different color on a display screen of the augmented reality glasses.

### Claim 7 (depends on 1)

The method of claim 1, further comprising filtering out noise from the audio file.

### Claim 8 (depends on 1)

The method of claim 1, further comprising translating the audio file into a second language.

### Claim 9

An augmented reality apparatus for hearing-impaired people comprising:
a frame;
display lens connected to the frame;
a plurality of microphones connected to the frame, the plurality of microphones capturing a speech from a nearby speaker; and
a controller in communication with the plurality of microphones and the display lens,
wherein
the controller converts the captured speech into text, calculates a position for the nearby speaker, and displays the text along with information on the position on the display lens.

### Claim 10 (depends on 9)

The augmented reality apparatus of claim 9, wherein the controller receives language preference and visual range setting from a user.

### Claim 11 (depends on 10)

The augmented reality apparatus of claim 10, wherein the controller displays turning information if the position is out of the visual range.

### Claim 12 (depends on 10)

The augmented reality apparatus of claim 10, wherein the controller displays the text with an out of range indicator on the display lens if the position of the speaker is out of visual range.

### Claim 13 (depends on 10)

The augmented reality apparatus of claim 10, wherein the controller displays the text in a different color on the display screen if the position of the speaker is out of visual range.

### Claim 14 (depends on 9)

The augmented reality apparatus of claim 9, wherein the controller checks if audio conversion feature is turned on.

### Claim 15 (depends on 9)

The augmented reality apparatus of claim 9, wherein the controller stores the speech and the text file a storage unit.

### Claim 16 (depends on 9)

The augmented reality apparatus of claim 9, wherein the controller filters out noise from the speech.

### Claim 17 (depends on 9)

The augmented reality apparatus of claim 9, wherein the controller translates the speech into a second language.
