# US 2024/0265935 A1 · Wearable speech input-based to moving lips display overlay

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0265935 A1
- Document ID: US20240265935A1
- Status: Published application
- Kind: A1
- Application number: 18/624,869
- Filed: 2024-04-02
- Published: 2024-08-08
- Earliest claimed: 2021-08-23
- Assignees: Snap Inc.
- Inventors: Kathleen Worthington McMahon
- Web page: https://patentgrep.com/patent/US20240265935A1

## CPC

- G10L 21/10: Transforming into visible information
- G02B 27/0101: {characterised by optical features (G02B27/0172 takes precedence)}
- G02B 27/017: {Head mounted}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G10L 2021/105: {Synthesis of the lips movements from speech, e.g. for talking heads}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Eyewear having a speech to moving lips algorithm that receives and translates speech and utterances of a person viewed through the eyewear, and then displays an overlay of moving lips corresponding to the speech and utterances on a mask of the viewed person. A database having text to moving lips information is utilized to translate the speech and generate the moving lips in near-real time with little latency. This translation provides the deaf/hearing impaired users the ability to understand and communicate with the person viewed through the eyewear when they are wearing a mask. The translation may include automatic speech recognition (ASR) and natural language understanding (NLU) as a sound recognition engine.

## Claims

### Claim 1

A device, comprising:
a body;
a display supported by the body; and
a processor configured to:
receive an image of a person wearing a mask covering lips of the person;
receive a signal indicative of speech of the person;
process the signal and translate the speech;
generate a moving lip image indicative of the translated speech; and
overlay the moving lip image on the display over the mask covering the lips of the person.

### Claim 2 (depends on 1)

The device of claim 1, further comprising a database storing text matched to a respective moving lip image, wherein the processor is configured to translate the speech to text and use the database to generate the moving lip image indicative of the translated speech.

### Claim 3 (depends on 2)

The device of claim 2, wherein the processor is configured to generate a series of moving lip images, including a moving lip image for each word of the speech.

### Claim 4 (depends on 2)

The device of claim 2, wherein the processor is configured to generate a series of moving lip images, including a moving lip image for each word and utterance of the speech.

### Claim 5 (depends on 2)

The device of claim 2, wherein the processor is configured to display the moving lip image over displayed mask at a size that is enlarged and disproportionate to a human head.

### Claim 6 (depends on 1)

The device of claim 1, wherein the processor is configured to use automatic speech recognition (ASR) to translate the speech to text.

### Claim 7 (depends on 2)

The device of claim 2, wherein the processor is configured to also use natural language understanding (NLU) to translate the speech to text.

### Claim 8 (depends on 1)

The device of claim 1, wherein the processor is configured to generate the moving lip image in near real time.

### Claim 9

A method of use of a device having a body, a display supported by the body, and a processor, the processor:
receiving an image of a person wearing a mask covering lips of the person;
receive a signal indicative of speech of the person;
processing the signal and translating the speech;
generating a moving lip image indicative of the translated speech; and
overlaying the moving lip image on the display over the mask covering the lips of the person.

### Claim 10 (depends on 9)

The method of claim 9, the device further comprising a database storing text matched to a respective moving lip image, wherein the processor translates the speech to text and uses the database to generate the moving lip image indicative of the translated speech.

### Claim 11 (depends on 10)

The method of claim 10, wherein the processor generates a series of moving lip images, including a moving lip image for each word of the speech.

### Claim 12 (depends on 10)

The method of claim 10, wherein the processor generates a series of moving lip images, including a moving lip image for each word and utterance of the speech.

### Claim 13 (depends on 10)

The method of claim 10, wherein the processor displays the moving lip image over displayed mask at a size that is enlarged and disproportionate to a human head.

### Claim 14 (depends on 9)

The method of claim 9, wherein the processor uses automatic speech recognition (ASR) to translate the speech to text.

### Claim 15 (depends on 14)

The method of claim 14, wherein the processor also uses natural language understanding (NLU) to translate the speech to text.

### Claim 16 (depends on 9)

The method of claim 9, wherein the processor generates the moving lip image in near real time.

### Claim 17

A non-transitory computer-readable medium storing program code which, when executed, is operative to cause a processor of a device having a body, a display supported by the body to perform the steps of:
receiving an image of a person wearing a mask covering lips of the person;
receive a signal indicative of speech of the person;
processing the signal and translating the speech;
generating a moving lip image indicative of the translated speech; and
overlaying the moving lip image on the display over the mask covering the lips of the person.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the program code, when executed, is operative to cause the processor to use a database storing text matched to a respective moving lip image and translate the speech to text and using the database to generate the moving lip image indicative of the translated speech.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein the program code, when executed, is operative to cause the processor to generate a series of moving lip images, including a moving lip image for each word and utterance of the speech.

### Claim 20 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein the program code, when executed, is operative to cause the processor to display the moving lip image over displayed mask at a size that is enlarged and disproportionate to a human head.
