# US 2023/0056847 A1 · Wearable speech input-based to moving lips display overlay

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0056847 A1
- Document ID: US20230056847A1
- Status: Published application
- Kind: A1
- Application number: 17/408,820
- Filed: 2021-08-23
- Published: 2023-02-23
- Earliest claimed: 2021-08-23
- Assignees: Snap Inc.
- Inventors: Kathleen Worthington McMahon
- Web page: https://patentgrep.com/patent/US20230056847A1

## CPC

- G10L 21/10: Transforming into visible information
- G02B 27/0101: {characterised by optical features (G02B27/0172 takes precedence)}
- G02B 27/017: {Head mounted}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G10L 2021/105: {Synthesis of the lips movements from speech, e.g. for talking heads}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Eyewear having a speech to moving lips algorithm that receives and translates speech and utterances of a person viewed through the eyewear, and then displays an overlay of moving lips corresponding to the speech and utterances on a mask of the viewed person. A database having text to moving lips information is utilized to translate the speech and generate the moving lips in near-real time with little latency. This translation provides the deaf/hearing impaired users the ability to understand and communicate with the person viewed through the eyewear when they are wearing a mask. The translation may include automatic speech recognition (ASR) and natural language understanding (NLU) as a sound recognition engine.

## Claims

### Claim 1

Eyewear, comprising:
a frame;
a camera supported by the frame and configured to generate an image of a person wearing a mask viewed through the frame;
a display supported by the frame;
a microphone supported by the frame; and
an electronic processor configured to:
receive a signal indicative of speech from the user via the microphone;
process the signal and translate the speech;
generate a moving lip image indicative of the translated speech; and
overlay the moving lip image on the display over the viewed mask of the person.

### Claim 2 (depends on 1)

The eyewear of claim 1, further comprising a database storing text matched to a respective moving lip image, wherein the processor is configured to translate the speech to text and use the database to generate the moving lip image indicative of the translated speech.

### Claim 3 (depends on 2)

The eyewear of claim 2, wherein the processor is configured to generate a series of moving lip images, including a moving lip image for each word of the speech.

### Claim 4 (depends on 3)

The eyewear of claim 3, wherein the processor is configured to generate a series of moving lip images, including a moving lip image for each word and utterance of the speech.

### Claim 5 (depends on 2)

The eyewear of claim 2, wherein the processor is configured to display the moving lip image over displayed mask at a size that is enlarged and disproportionate to a human head.

### Claim 6 (depends on 1)

The eyewear of claim 1, wherein the processor is configured to use automatic speech recognition (ASR) to translate the speech to text.

### Claim 7 (depends on 2)

The eyewear of claim 2, wherein the processor is configured to also use natural language understanding (NLU) to translate the speech to text.

### Claim 8 (depends on 1)

The eyewear of claim 1, wherein the processor is configured to generate the moving lip image in near real time.

### Claim 9

A method of use of eyewear, the eyewear having a frame, a camera supported by the frame and configured to generate an image of a person wearing a mask viewed through the frame, a display supported by the frame, a microphone supported by the frame, and a processor, the electronic processor:
receiving a signal indicative of speech from the person via the microphone;
processing the signal and translating the speech;
generating a moving lip image indicative of the translated speech; and
overlaying the moving lip image on the display over the viewed mask of the person.

### Claim 10 (depends on 9)

The method of claim 9, the eyewear further comprising a database storing text matched to a respective moving lip image, wherein the processor translates the speech to text and uses the database to generate the moving lip image indicative of the translated speech.

### Claim 11 (depends on 10)

The method of claim 10, wherein the processor generates a series of moving lip images, including a moving lip image for each word of the speech.

### Claim 12 (depends on 10)

The method of claim 10, wherein the processor generates a series of moving lip images, including a moving lip image for each word and utterance of the speech.

### Claim 13 (depends on 10)

The method of claim 10, wherein the processor displays the moving lip image over displayed mask at a size that is enlarged and disproportionate to a human head.

### Claim 14 (depends on 9)

The method of claim 9, wherein the processor uses automatic speech recognition (ASR) to translate the speech to text.

### Claim 15 (depends on 14)

The method of claim 14, wherein the processor also uses natural language understanding (NLU) to translate the speech to text.

### Claim 16 (depends on 9)

The method of claim 9, wherein the processor generates the moving lip image in near real time.

### Claim 17

A non-transitory computer-readable medium storing program code which, when executed, is operative to cause a processor of eyewear having a frame, a camera supported by the frame and configured to generate an image of a person wearing a mask viewed through the frame, a display supported by the frame, a microphone supported by the frame to perform the steps of:
receiving a signal indicative of speech from the person via the microphone;
processing the signal and translating the speech;
generating a moving lip image indicative of the translated speech; and
overlaying the moving lip image on the display over the viewed mask of the person.

### Claim 18 (depends on 17)

The non-transitory computer readable medium as specified in claim 17, wherein the program code, when executed, is operative to cause the processor to use a database storing text matched to a respective moving lip image, and translate the speech to text and using the database to generate the moving lip image indicative of the translated speech.

### Claim 19 (depends on 18)

The non-transitory computer readable medium as specified in claim 18, wherein the program code, when executed, is operative to cause the processor to generate a series of moving lip images, including a moving lip image for each word and utterance of the speech.

### Claim 20 (depends on 18)

The non-transitory computer readable medium as specified in claim 18, wherein the program code, when executed, is operative to cause the processor to display the moving lip image over displayed mask at a size that is enlarged and disproportionate to a human head.
