# US 10,482,872 B2 · Speech recognition apparatus and speech recognition method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,482,872 B2
- Document ID: US10482872B2
- Status: Granted
- Kind: B2
- Application number: 15/825,087
- Filed: 2017-11-28
- Published: 2018-06-07
- Granted: 2019-11-19
- Earliest claimed: 2016-12-01
- Assignees: Olympus Corporation
- Applicants: Olympus Corporation
- Inventors: Hiroyuki Tokiwa; Kenta Yumoto; Osamu Nonaka
- Web page: https://patentgrep.com/patent/US10482872B2

## CPC

- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G06V 40/16: Human faces, e.g. facial parts, sketches or expressions
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G10L 15/25: using position of the lips, movement of the lips or face analysis
- G10L 25/57: for processing of video signals
- G10L 2015/022: {Demisyllables, biphones or triphones being the recognition units}
- G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units}
- G10L 2015/027: {Syllables being the recognition units}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A speech recognition apparatus according to an embodiment includes a microphone that acquires an audio stream in which speech vocalized by a person is recorded, a camera that acquires an image data in which at least a mouth of the person is captured, and an operation element that recognizes speech including a consonant vocalized by the person, based on the audio stream, estimates the consonant vocalized by the person, based on the shape of the mouth of the person in the image data, and specifies the consonant based on the estimated consonant and the speech-recognized consonant.

## Claims

### Claim 1

A speech recognition apparatus comprising:
a microphone configured to acquire an audio stream in which speech vocalized by a person is recorded;
a camera configured to acquire an image data in which at least a mouth of the person is captured;
an operation element configured to recognize speech including a consonant vocalized by the person based on the audio stream, estimate the consonant vocalized by the person based on a mouth shape of the person in the image data, and specify the consonant based on the estimated consonant and the speech-recognized consonant,
wherein the operation element specifies a consonant vocalization frame which is a frame in which it is estimated that the person in the image data has vocalized a consonant before a vowel based on a timing when the vowel is detected, and estimates the consonant vocalized by the person based on the mouth shape of the person in the consonant vocalization frame and a mouth shape model that indicates a change in mouth shape for each consonant.

### Claim 2 (depends on 1)

The speech recognition apparatus according to claim 1, wherein each of a plurality of frames until a vowel is detected is the consonant vocalization frame, and the operation element estimates a consonant vocalized by the person based on a change in mouth shape of the person in the plurality of consonant vocalization frames and the mouth shape model that indicates a change in mouth shape for each consonant.

### Claim 3 (depends on 2)

The speech recognition apparatus according to claim 2, wherein the operation element determines a timing before a timing when a vowel is detected by a predetermined time as a head of the consonant vocalization frames.

### Claim 4 (depends on 2)

The speech recognition apparatus according to claim 2, wherein the operation element determines a timing when an opening area becomes a predetermined value or greater as a head of the consonant vocalization frames.

### Claim 5 (depends on 2)

The speech recognition apparatus according to claim 2, wherein the operation element determines a timing when a ratio between breadth and length of a mouth becomes a predetermined value or greater as a head of the consonant vocalization frames.

### Claim 6 (depends on 2)

The speech recognition apparatus according to claim 2, wherein the operation element determines a timing when a sound of a predetermined volume or greater is detected as a head of the consonant vocalization frames.

### Claim 7 (depends on 2)

The speech recognition apparatus according to claim 2, wherein the operation element determines a timing before a timing when a sound of a predetermined volume or greater is detected by a predetermined time as a head of the consonant vocalization frames.

### Claim 8 (depends on 1)

The speech recognition apparatus according to claim 1, wherein the operation element detects a timing when the person vocalizes a vowel based on the audio stream.

### Claim 9 (depends on 1)

The speech recognition apparatus according to claim 1, wherein the operation element detects a timing when the person has vocalized a vowel based on the image data.