# US 10,770,066 B2 · Slot filling in spoken language understanding with joint pointer and attention

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,770,066 B2
- Document ID: US10770066B2
- Status: Granted
- Kind: B2
- Application number: 16/057,521
- Filed: 2018-08-07
- Published: 2019-12-05
- Granted: 2020-09-08
- Earliest claimed: 2018-05-31
- Assignees: Robert Bosch GmbH
- Applicants: Robert Bosch GmbH
- Inventors: Lin Zhao; Zhe Feng
- Web page: https://patentgrep.com/patent/US10770066B2

## CPC

- G10L 15/197: Probabilistic grammars, e.g. word n-grams
- G06F 40/174: Form filling; Merging
- G10L 15/16: using artificial neural networks
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method for slot-filling in an automated spoken language understanding (SLU) system includes receiving words in a text phrase, generating an encoded output and attention weights using a recurrent neural network (RNN) encoder with an attention mechanism, generating a first and second probability distributions of slot words in an extended slot vocabulary database using an RNN decoder and a pointer network, respectively, for a time step, generating a combined probability distribution based on the first and second probability distributions, and identifying a slot word corresponding to one word in the extended slot vocabulary database having a highest probability value in the combined probability distribution for the time step.

## Claims

### Claim 1

A method for operation of a spoken language understanding (SLU) system comprising:
receiving, with a processor in the SLU system, an input sequence including a plurality of words in a text phrase;
generating, with the processor, an encoded output corresponding to the plurality of words and a plurality of attention weights corresponding to the plurality of words using a recurrent neural network (RNN) encoder with an attention mechanism that receives the plurality of words as inputs to the RNN encoder;
generating, with the processor, a first probability distribution corresponding to a plurality of words in an extended slot vocabulary database being a slot word at a time step using an RNN decoder that receives the encoded output from the RNN encoder as input;
generating, with the processor, a second probability distribution corresponding to the plurality of words in the input sequence being a slot word at the time step using a pointer network that receives the plurality of attention weights as input;
generating, with the processor, a combined probability distribution based on the first probability distribution and the second probability distribution;
identifying, with the processor, a slot word corresponding to one word in the extended slot vocabulary database having a highest probability value in the combined probability distribution for the time step; and
performing, with the processor, a command using the slot word as a parameter to the command.

### Claim 2 (depends on 1)

The method of claim 1, the generating of the combined probability distribution further comprising:
generating, with the processor, a weight value based on a state of the RNN decoder; and
generating, with the processor, the combined probability distribution as weighted sum of the first probability distribution and the second probability distribution using the weight value.

### Claim 3 (depends on 2)

The method of claim 2 further comprising:
generating, with the processor, the weight value as pt at the time step t based on the following function pt=σ(wcct+wsst+wddt+b)
where ct is a context vector of the attention mechanism,
st is a is a hidden state value of the RNN decoder,
dt is an input to the RNN decoder,
wc, ws, and wd, are a plurality of trainable weight values, and
b is a bias term.

### Claim 4 (depends on 3)

The method of claim 3 further comprising:
generating, with the processor, the combined probability distribution including a probability value P(w) where w is a word in the plurality of words in the extended slot vocabulary database based on the following function:
P(w)=PtPgen(w)+(1−pt)Pptr(w),
where Pgen is the first probability distribution, and
Pptr is the second probability distribution.

### Claim 5 (depends on 2)

The method of claim 2 wherein the processor generates the weight value to favor the first probability distribution generated by the RNN decoder in response to the first word being an in-vocabulary word and the processor generates the weight value to favor the second probability distribution generated by the pointer network in response to the first word being an out-of-vocabulary word.

### Claim 6 (depends on 1)

The method of claim 1 wherein the RNN encoder is a bi-directional RNN encoder.

### Claim 7 (depends on 6)

The method of claim 6 wherein the RNN encoder includes at least two gated recurrent units.

### Claim 8 (depends on 7)

The method of claim 7 wherein the RNN decoder includes at least one gated recurrent unit.

### Claim 9 (depends on 6)

The method of claim 6 wherein the RNN decoder is a uni-directional RNN decoder.

### Claim 10 (depends on 1)

The method of claim 1 further comprising:
generating, with the processor, the plurality of words in the text phrase using an automated speech recognition engine based on audio data received using an audio input device.

### Claim 11

A system for spoken language understanding comprising:
an input device;
an output device;
a memory; and
a processor operatively connected to the input device, the output device, and the memory, the processor being configured to:
receive an input sequence including a plurality of words in a text phrase based on an input received with the input device;
generate an encoded output corresponding to the plurality of words and a plurality of attention weights corresponding to the plurality of words using a recurrent neural network (RNN) encoder with an attention mechanism stored in the memory that receives the plurality of words as inputs to the RNN encoder;
generate a first probability distribution corresponding to a plurality of words in an extended slot vocabulary database stored in the memory being a slot word at a time step using an RNN decoder stored in the memory that receives the encoded output from the RNN encoder as input;
generate a second probability distribution corresponding to the plurality of words in the input sequence being a slot word at the time step using a pointer network that receives the plurality of attention weights as input;
generate a combined probability distribution based on the first probability distribution and the second probability distribution;
identify a slot word corresponding to one word in the extended slot vocabulary database having a highest probability value in the combined probability distribution for the time step; and
perform a command using the slot word as a parameter to the command to produce an output with the output device.

### Claim 12 (depends on 11)

The system of claim 11, the processor being further configured to:
generate a weight value based on a state of the RNN decoder; and
generate the combined probability distribution as weighted sum of the first probability distribution and the second probability distribution using the weight value.

### Claim 13 (depends on 12)

The system of claim 12, the processor being further configured to:
generate the weight value as pt at the time step t based on the following function Pt=σ(wcct+wsst+wddt+b)
where ct is a context vector of the attention mechanism,
st is a is a hidden state value of the RNN decoder,
dt is an input to the RNN decoder,
wc, ws, and wd, are a plurality of trainable weight values, and
b is a bias term.

### Claim 14 (depends on 13)

The system of claim 13, the processor being further configured to:
generate the combined probability distribution including a probability value P(w) where w is a word in the plurality of words in the extended slot vocabulary database based on the following function:
P(w)ptPgen(w)+(1−pt)Pptr(w),
where Pgen is the first probability distribution, and
Pptr is the second probability distribution.

### Claim 15 (depends on 12)

The system of claim 12, the processor being further configured to generate the weight value to favor the first probability distribution generated by the RNN decoder in response to the first word being an in-vocabulary word and the processor generates the weight value to favor the second probability distribution generated by the pointer network in response to the first word being an out-of-vocabulary word.

### Claim 16 (depends on 11)

The system of claim 11 wherein the RNN encoder is a bi-directional RNN encoder.

### Claim 17 (depends on 16)

The system of claim 16 wherein the RNN encoder includes at least two gated recurrent units.

### Claim 18 (depends on 17)

The system of claim 17 wherein the RNN decoder includes at least one gated recurrent unit.

### Claim 19 (depends on 16)

The system of claim 16 wherein the RNN decoder is a uni-directional RNN decoder.

### Claim 20 (depends on 11)

The system of claim 11, the input device further comprising:
an audio input device; and
the processor being further configured to generate the plurality of words in the text phrase using an automated speech recognition engine based on audio data received using the audio input device.