# US 2021/0280191 A1 · Lip language recognition method and mobile terminal

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0280191 A1
- Document ID: US20210280191A1
- Status: Published application
- Kind: A1
- Application number: 16/346,815
- Filed: 2018-11-26
- Published: 2021-09-09
- Earliest claimed: 2018-01-02
- Assignees: Boe Technology Group Co., Ltd.
- Applicants: Boe Technology Group Co., Ltd.
- Inventors: Lihua GENG; Xitong MA; Zhiguo ZHANG
- Web page: https://patentgrep.com/patent/US20210280191A1

## CPC

- G06V 10/82: using neural networks
- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V 40/176: {Dynamic expression}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G10L 13/0335: {Pitch control}
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/063: {Training}
- G10L 15/16: using artificial neural networks
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/25: using position of the lips, movement of the lips or face analysis
- H04R 1/08: Mouthpieces; {Microphones;} Attachments therefor
- H04R 1/10: Earpieces; Attachments therefor {; Earphones; Monophonic headphones (H04R1/28 takes precedence; stereophonic headphones H04R5/033)}
- H04R 2499/11: Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)

## Abstract

A lip language recognition method, applied to a mobile terminal having a sound mode and a silent mode, includes: training a deep neural network in the sound mode; collecting a user's lip images in the silent mode; and identifying content corresponding to the user's lip images with the deep neural network trained in the sound mode.

## Claims

### Claim 1

A lip language recognition method, applied to a mobile terminal having a sound mode and a silent mode, the method comprising:
training a deep neural network in the sound mode;
collecting a user's lip images in the silent mode; and
identifying content corresponding to the user's lip images with the deep neural network trained in the sound mode.

### Claim 2 (depends on 1)

The method of claim 1, wherein the training comprises:
collecting lip images and corresponding voice data for training;
obtaining image data corresponding to the collected lip images for training, the image data comprising pixel information; and
training the deep neural network based on the image data and the voice data for training.

### Claim 3 (depends on 1)

The method of claim 1, wherein the training of the deep neural network comprises:
collecting lip images and corresponding voice data for training;
obtaining image data corresponding to the collected lip images for training, the image data comprising pixel information;
obtaining text encoding corresponding to the voice data for training; and
training the deep neural network based on the image data and the text encoding for training.

### Claim 4 (depends on 3)

The method of claim 3, wherein the identifying the content corresponding to the user's lip images with the deep neural network comprises:
obtaining user image data corresponding to the user's lip images; and
identifying user text encoding corresponding to the user's lip images by applying the deep neural network on the user image data.

### Claim 5 (depends on 2, 3)

The method of claim 2 or claim 3, further comprising extracting the user's voice features based on the voice data for training.

### Claim 6 (depends on 5)

The method of claim 5, wherein the user's voice features comprise at least one of tone color, pitch, or volume.

### Claim 7 (depends on 6)

The method of claim 6, further comprising synthesizing user voice data having the user's voice features based on the extracted user's voice features and the content corresponding to the user's lip images.

### Claim 8

A mobile terminal having a sound mode and a silent mode, comprising:
an acquisition portion; and
a processing portion; wherein:
the acquisition portion is configured to acquire a user's lip images in the silent mode; and
the processing portion is in communication with the acquisition portion and configured to identify content corresponding to user's the lip images with a deep neural network established in the sound mode.

### Claim 9 (depends on 8)

The mobile terminal of claim 8, wherein:
in the sound mode, the acquisition portion is configured to collect, with an imaging device and a microphone device, lip images and corresponding voice data for training; and
the processing portion is configured to:
obtain image data including pixel information based on the collected lip images for training; and
train the deep neural network according to the image data and the voice data for training.

### Claim 10 (depends on 8)

The mobile terminal of claim 8, wherein:
in the sound mode, the acquisition portion is configured to acquire lip images and corresponding voice data for training;
the processing portion is configured to:
obtain image data corresponding to lip images for training;
obtain text encoding for training corresponding to the voice data for training; and
train the deep neural network according to the image data and the text encoding for training.

### Claim 11 (depends on 10)

The mobile terminal of claim 10, wherein the processing portion is further configured to identify the text encoding for training using the deep neural network.

### Claim 12 (depends on 10)

The mobile terminal of claim 10, further comprising a feature extraction portion configured to obtain a user's voice features according to the voice data for training; wherein the voice features comprise at least one of tone color, pitch, or volume.

### Claim 13 (depends on 12)

The mobile terminal of claim 12, further comprising a speech synthesis portion configured to synthesize voice data with the user's voice features according to the obtained voice features and the identified content.

### Claim 14 (depends on 13)

The mobile terminal of claim 13, wherein the acquisition portion comprises an imaging device disposed at a bottom portion of the mobile terminal.

### Claim 15 (depends on 14)

The mobile terminal according to claim 14, further comprising:
a sending portion configured to encode the synthesized voice data and send the encoded synthesized voice data to a communication station wirelessly;
a receiving portion configured to receive a signal from the communication station and perform decoding and conversion into user-recognizable voice data; and
an earpiece configured to play the user-recognizable voice data decoded and converted by the receiving portion.

### Claim 16 (depends on 15)

A non-transitory computer-readable medium having instructions stored on for execution by the mobile terminal of claim 15 for lip language recognition, the instructions comprising:
an imaging device capturing the lip images for training in a voice communication;
a microphone collecting the voice data corresponding to the lip images for training;
the processing portion obtaining the image data according to the lip images and text encoding corresponding to the voice data, and training the deep neural network based on the image data and the text encoding;
saving training results to guide the lip image recognition in the silent mode; and
a feature extraction portion extracting from the voice data user's voice features and saving the extracted voice features.

### Claim 17 (depends on 16)

The non-transitory computer-readable medium of claim 16, wherein the instructions further comprise:
the processing portion identifying the text encoding from the user's image data using the trained deep neural network, and transmitting the recognized text encoding to the speech synthesis portion;
the speech synthesis portion synthesizing voice data having the voice features based on the voice features saved in the sound mode and the recognized text encoding;
the sending portion encoding and sending the voice data having the voice features to a communication station wirelessly; and
the receiving portion receiving from the communication station the voice for decoding.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the instructions further comprise:
downloading sound recording;
the feature extraction portion extracting sound features from the downloaded sound recording; and
mixing the extracted sound features with the saved voice features prior to the synthesizing.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein the instructions further comprise:
obtaining user feedbacks on the text encoding for training; and
training the deep neural network with the obtained user feedbacks.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the instructions further comprise:
starting the silent mode with a user input of a key phrase;
wherein the key phrase is recognized by the mobile terminal through the user's lip movements without associated voice.
