# US 11,114,101 B2 · Speech recognition with image signal

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,114,101 B2
- Document ID: US11114101B2
- Status: Granted
- Kind: B2
- Application number: 16/962,734
- Filed: 2019-01-25
- Published: 2020-11-12
- Granted: 2021-09-07
- Earliest claimed: 2018-01-31
- Assignees: Iebm B.V.
- Applicants: Iebm B.V.
- Inventors: Olaf Petrus Quirinus Mossinkoff; Johannes Leonardus Jozef Meijer
- Web page: https://patentgrep.com/patent/US11114101B2

## CPC

- G10L 15/25: using position of the lips, movement of the lips or face analysis
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- H04N 23/56: provided with illuminating means
- H04N 23/611: where the recognised objects include parts of the human body
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method of speech recognition and person identification based thereon, comprising: recording speech from a speech signal using a microphone; illuminating a speaking mouth; recording a degree of light reflected by the mouth from a reflection signal using a sensor; and recording combined parameters of the speech signal and of the reflection signal, and coupling them to letters associated therewith, per predetermined time duration; comparing a combination occurring in speech of parameters of the speech signal and of the reflection signal to the recorded combined parameters of the speech signal and of the reflection signal which are coupled to letters; and deciding on the basis of the comparison to which letter the combination occurring in the speech of parameters of the speech signal and of the reflection signal corresponds, using block-width modulation of the reflection signal.

## Claims

### Claim 1

A method of speech recognition, comprising:
recording speech from a speech signal using a microphone;
illuminating a speaking mouth;
recording a degree of light reflected by the mouth from a reflection signal using a sensor;
for speech recognition, in one time duration of the predetermined length of speech to be recognized:
comparing a combination of parameters of the speech signal and of the reflection signal with pre-recorded combined parameters of a speech signal obtained during a training phase and of a reflection signal obtained during said training phase of at least some of a plurality of pre-recorded time durations, wherein said combination of parameters includes information regarding said degree of light reflected by the mouth, and wherein said pre-recorded combined parameters include information regarding said degree of light reflected by the mouth determined during said training phase;
wherein the training speech signal and the training reflection signal are coupled to letters associated therewith per time duration; and
deciding on the basis of the comparison to which letter the combination occurring in the speech of parameters of the speech signal and of the reflection signal corresponds,
wherein block-width modulation is applied to the reflection signal.

### Claim 2 (depends on 1)

The method of claim 1, wherein the parameter of the speech signal is taken from a group comprising within the one time duration of speech to be recognized at least:
volume dispersion of a difference between a highest and a lowest value of a volume of the speech signal; and
a ratio of sound of the speech signal within and outside a noise level (signal-to-noise ratio).

### Claim 3 (depends on 1)

The method of claim 1, wherein the parameter of the reflection signal is taken from a group comprising within the one time duration at least:
an average of the reflection signal; and
a degree of increase or decrease of the reflection signal.

### Claim 4 (depends on 3)

The method of claim 3, further comprising determining the average of the reflection signal as an average over the one time duration of half a block duration of a block wave used in block wave modulation.

### Claim 5 (depends on 3)

The method of claim 3, further comprising determining the degree of increase or decrease of the reflection signal as a measurement in degrees.

### Claim 6 (depends on 1)

The method of claim 1, wherein the time duration has a predetermined length of 1, 2, 3, 4 or 5 milliseconds.

### Claim 7 (depends on 1)

The method of claim 1, further comprising subdividing the speech signal into portions corresponding to letters, and indicating at least one of:
starting and ending times of letters in the speech signal; and
time durations expressed in a number of times of the one time duration.

### Claim 8 (depends on 1)

The method of claim 1, further comprising determining a maximum and minima of a degree of light reflected by the mouth in the reflection signal, and normalizing the reflection signal on the basis of the maximum and minima.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
forming a preselection of at least one potential subsequent letter on the basis of the decision to which letter the combination occurring in the speech of parameters of the speech signal and of the reflection signal corresponds to.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
synchronously registering speech and registering a degree of light reflected by the mouth in the reflection signal.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
person recognition based on recognition from the speech signal and the reflection signal of viseme and phoneme combinations.