# US 2013/0158998 A1 · Systems and Methods for Extracting Meaning from Multimodal Inputs Using Finite-State Devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0158998 A1
- Document ID: US20130158998A1
- Status: Published application
- Kind: A1
- Application number: 13/690,037
- Filed: 2012-11-30
- Published: 2013-06-20
- Earliest claimed: 2001-07-12
- Assignees: AT&T Intellectual Property II, L.P.
- Inventors: Srinivas Bangalore; Michael J. Johnston
- Web page: https://patentgrep.com/patent/US20130158998A1

## CPC

- G10L 15/00: Speech recognition (G10L17/00 takes precedence)
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G10L 15/24: Speech recognition using non-acoustical features
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Multimodal utterances contain a number of different modes. These modes can include speech, gestures, and pen, haptic, and gaze inputs, and the like. This invention use recognition results from one or more of these modes to provide compensation to the recognition process of one or more other ones of these modes. In various exemplary embodiments, a multimodal recognition system inputs one or more recognition lattices from one or more of these modes, and generates one or more models to be used by one or more mode recognizers to recognize the one or more other modes. In one exemplary embodiment, a gesture recognizer inputs a gesture input and outputs a gesture recognition lattice to a multimodal parser. The multimodal parser generates a language model and outputs it to an automatic speech recognition system, which uses the received language model to recognize the speech input that corresponds to the recognized gesture input.

## Claims

### Claim at position 1

1-31. (canceled)

### Claim 32

A method for recognizing an utterance comprising:
receiving an utterance comprising a first portion having a first mode and a second portion having a second mode;
generating a first mode recognition lattice and a first finite-state transducer, based on the first mode;
relating the first portion of the utterance to the second portion of the utterance based on the first finite-state transducer;
generating a second finite-state transducer, comprising a gesture and speech recognition model finite-state transducer, based on the first mode recognition lattice and the first finite-state transducer; and
outputting a recognition result based on the utterance and the second finite-state transducer.

### Claim 33 (depends on 32)

The method of claim 32, further comprising:
recognizing the second mode based on the second finite-state transducer.

### Claim 34 (depends on 32)

The method of claim 32 wherein the first mode is a gesture mode.

### Claim 35 (depends on 32)

The method of claim 32 wherein the second mode is a speech mode.

### Claim 36 (depends on 32)

The method of claim 32 wherein the generating the first mode recognition lattice is further based on a first mode feature lattice.

### Claim 37

A method for recognizing an utterance comprising:
receiving an utterance comprising a plurality of modes;
relating a first portion of the utterance comprising a first mode of the plurality of modes to a second portion of the utterance comprising a second mode of the plurality of modes, based on a first finite-state transducer;
generating a second finite-state transducer, comprising a gesture and speech recognition model finite-state transducer, based on a first mode recognition lattice and the first finite-state transducer; and
outputting a recognition result based on the utterance and the second finite-state transducer.

### Claim 38 (depends on 37)

The method of claim 37 further comprising:
generating the first mode recognition lattice and the first finite-state transducer, based on the first mode.

### Claim 39 (depends on 37)

The method of claim 37 further comprising:
recognizing the second mode based on the second finite-state transducer.

### Claim 40 (depends on 37)

The method of claim 37 wherein the first mode is a gesture mode.

### Claim 41 (depends on 37)

The method of claim 37 wherein the second mode is a speech mode.

### Claim 42 (depends on 38)

The method of claim 38 wherein the generating the first mode recognition lattice is further based on a first mode feature lattice.

### Claim 43 (depends on 42)

The method of claim 42 further comprising:
generating the first mode feature lattice based on the utterance.

### Claim 44

A method for extracting meaning from multimodal inputs comprising:
receiving a multimodal input;
relating a first portion of the multimodal input comprising a first mode to a second portion of the multimodal input comprising a second mode, based on a first finite-state transducer;
generating a second finite-state transducer, comprising a gesture and speech recognition model finite-state transducer, based on a first mode recognition lattice and the first finite-state transducer; and
outputting a recognition result based on the multimodal input and the second finite-state transducer.

### Claim 45 (depends on 44)

The method of claim 44 further comprising:
generating the first mode recognition lattice and the first finite-state transducer, based on the first mode.

### Claim 46 (depends on 44)

The method of claim 44 wherein the first mode is a gesture mode.

### Claim 47 (depends on 44)

The method of claim 44 wherein the second mode is a speech mode.

### Claim 48 (depends on 45)

The method of claim 45 wherein the generating the first mode recognition lattice is further based on a first mode feature lattice.

### Claim 49 (depends on 48)

The method of claim 48 further comprising:
generating the first mode feature lattice based on the multimodal input.
