# US 2022/0044671 A1 · Spoken language understanding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044671 A1
- Document ID: US20220044671A1
- Status: Published application
- Kind: A1
- Application number: 16/989,012
- Filed: 2020-08-10
- Published: 2022-02-10
- Earliest claimed: 2020-08-10
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: NIKITA KAPOOR; Jaya Dodeja; Nikaash Puri
- Web page: https://patentgrep.com/patent/US20220044671A1

## CPC

- G10L 15/063: {Training}
- G06F 40/30: Semantic analysis
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/16: using artificial neural networks
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 19/04: using predictive techniques
- G10L 25/51: for comparison or discrimination
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and methods for spoken language understanding are described. Embodiments of the systems and methods receive audio data for a spoken language expression, encode the audio data using a multi-stage encoder comprising a basic encoder and a sequential encoder, wherein the basic encoder is trained to generate character features during a first training phase and the sequential encoder is trained to generate token features during a second training phase, and decode the token features to generate semantic information representing the spoken language expression.

## Claims

### Claim 1

A method of spoken language understanding, the method comprising:
receiving audio data for a spoken language expression;
encoding the audio data using a multi-stage encoder comprising a basic encoder and a sequential encoder, wherein the basic encoder is trained to generate character features during a first training phase and the sequential encoder is trained to generate token features during a second training phase; and
decoding the token features to generate semantic information representing the spoken language expression.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
generating a spectrogram based on the audio data; and
dividing the spectrogram into a plurality of frames, wherein the multi-stage encoder takes the frames as input.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
generating a sequence of character feature vectors using the basic encoder, wherein each of the sequence of character feature vectors corresponds to one of the frames; and
generating a sequence of token feature vectors based on the sequence of character feature vectors using the sequential encoder.

### Claim 4 (depends on 3)

The method of claim 3, wherein:
the character feature vectors are generated independently of each other.

### Claim 5 (depends on 3)

The method of claim 3, further comprising:
generating a first token feature vector; and
generating a second token feature vector based at least in part on the first token feature vector.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
identifying a decoding position;
computing a ratio between an output sequence of the multi-stage encoder and a subsequent input sequence; and
computing a sum of encoder states based on the decoding position and the ratio, wherein the decoding is based on the sum of encoder states.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
generating a response to the spoken language expression based on the semantic information.

### Claim 8 (depends on 1)

The method of claim 1, wherein:
the semantic information comprises contextual information.

### Claim 9 (depends on 1)

The method of claim 1, wherein:
the semantic information includes attribute names, attribute values, or both.

### Claim 10

A method of training a neural network for spoken language understanding, the method comprising:
training a basic encoder to generate character features based on a spoken language expression;
training a sequential encoder to generate token features based on the spoken language expression;
combining the basic encoder, the sequential encoder and a decoder in sequence to produce an end-to-end neural network for spoken language understanding; and
training the end-to-end network to generate semantic information for the spoken language expression.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
appending one or more linear layers with a log-softmax output function to the basic encoder;
predicting the character features for the spoken language expression using the one or more linear layer;
comparing the predicted character features to ground-truth character features; and
adjusting parameters of the basic encoder based on the comparison.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
removing the log-softmax output function prior to combining the basic encoder, the sequential encoder, and the decoder.

### Claim 13 (depends on 10)

The method of claim 10, further comprising:
appending a linear layer and a sequential decoder to layers of a basic model;
predicting token features for the spoken language expression using the linear layer and the sequential decoder;
comparing the predicted token features to ground-truth token features; and
adjusting parameters of the layers of the basic model, the linear layer, and the sequential decoder based on the comparison.

### Claim 14 (depends on 10)

The method of claim 10, further comprising:
predicting semantic information for the spoken language expression using the end-to-end network;
comparing the predicted semantic information to ground truth semantic information; and
updating parameters of the basic encoder, the sequential encoder and the decoder based on the comparison.

### Claim 15 (depends on 10)

The method of claim 10, further comprising:
training the end-to-end network is based on a connectionist temporal classification (CTC) loss.

### Claim 16

An apparatus for spoken language understanding, comprising:
a basic encoder configured to generate character features based on audio data for a spoken language expression, wherein the basic encoder is trained in a first training phase;
a sequential encoder configured to generate token features based on an output of the basic encoder, wherein the sequential encoder is trained during a second training phase; and
a decoder configured to generate semantic information for the spoken language expression based on an output of the sequential encoder, wherein the decoder is trained together with the basic encoder and the sequential encoder during a third training phase.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein:
the basic encoder comprises one or more convolutional neural network (CNN) layers and one or more recurrent neural network (RNN) layers.

### Claim 18 (depends on 16)

The apparatus of claim 16, wherein:
the basic encoder is trained using a linear layer with a log-softmax output function.

### Claim 19 (depends on 16)

The apparatus of claim 16, wherein:
the sequential encoder comprises one or more CNN layers, one or more RNN layers, a linear layer, and a sequential decoder.

### Claim 20 (depends on 16)

The apparatus of claim 16, wherein:
the neural network is trained using an incremental training process including the first training phase for training the basic encoder, the second training phase for training the sequential encoder, and the third training phase for training the neural network as a whole.