# US 2014/0278424 A1 · Kernel deep convex networks and end-to-end learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0278424 A1
- Document ID: US20140278424A1
- Status: Published application
- Kind: A1
- Application number: 13/798,284
- Filed: 2013-03-13
- Published: 2014-09-18
- Earliest claimed: 2013-03-13
- Assignees: Microsoft Corporation
- Applicants: Microsoft Corporation
- Inventors: Li Deng; Xiaodeng He; Gokhan Tur; Dilek Hakkani-Tur
- Web page: https://patentgrep.com/patent/US20140278424A1

## CPC

- G10L 15/063: {Training}
- G06N 3/045: Combinations of networks
- G06N 3/0499: Feedforward networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G10L 15/083: {Recognition networks (G10L15/142, G10L15/16 take precedence)}
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/1822: {Parsing for meaning understanding}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Data associated with spoken language may be obtained. An analysis of the obtained data may be initiated for understanding of the spoken language using a deep convex network that is integrated with a kernel trick. The resulting kernel deep convex network may also be constructed by stacking one shallow kernel network over another with concatenation of the output vector of the lower network with the input data vector. A probability associated with a slot that is associated with slot-filling may be determined, based on local, discriminative features that are extracted using the kernel deep convex network.

## Claims

### Claim 1

A system comprising:
a device that includes at least one processor, the device including a language understanding engine comprising instructions tangibly embodied on a computer readable storage medium for execution by the at least one processor, the language understanding engine including:
a feature acquisition component configured to obtain local, discriminative features that are associated with an input spoken language string;
a slot-filling component configured to determine a plurality of probabilities associated with a plurality of respective slots that are associated with a slot-filling task in spoken language understanding (SLU);
a softmax interface configured to provide an interface between the feature acquisition component and the slot-filling component, using a softmax function; and
an end-to-end learning component configured to train parameters for the softmax interface, based on an objective function taking a value of a model-based expectation of slot-filling accuracy over an entire training set.

### Claim 2 (depends on 1)

The system of claim 1, wherein:
the feature acquisition component is configured to obtain the local, discriminative features that are associated with the input spoken language string, based on n-gram contexts associated with words in the input spoken language string.

### Claim 3 (depends on 1)

The system of claim 1, further comprising:
a string acquisition component configured to obtain data associated with the input spoken language string; and
a deep convex network (DCN) analysis component configured to initiate an analysis of the obtained data for understanding of the spoken language string using a deep convex network, wherein
the feature acquisition component is configured to obtain the local, discriminative features that are associated with an input spoken language string, based on obtaining results of the analysis of the DCN analysis component.

### Claim 4 (depends on 3)

The system of claim 3, wherein:
the DCN analysis component is configured to determine the local, discriminative features that are associated with an input spoken language string, based on obtaining results of the analysis, wherein the deep convex network includes one or more of:
a deep convex network that includes sigmoidal hidden layers, or
a kernel deep convex network that includes a deep convex network with effectively infinite hidden units that avoids computation of the hidden units, based on using a kernel trick.

### Claim 5

A method comprising:
obtaining data associated with spoken language;
initiating an analysis of the obtained data for understanding of the spoken language using a deep convex network that is integrated with a kernel trick; and
determining a probability associated with a slot that is associated with slot-filling, based on local, discriminative features that are extracted using the deep convex network that is integrated with the kernel trick.

### Claim 6 (depends on 5)

The method of claim 5, wherein:
the analysis of the obtained data includes one or more of:
classifying one or more domains of the spoken language using the deep convex network that is integrated with the kernel trick, or
classifying one or more intents of the spoken language using the deep convex network that is integrated with the kernel trick.

### Claim 7 (depends on 5)

The method of claim 5, wherein:
the deep convex network includes a plurality of interconnected, stacked modules, wherein each module includes a plurality of layers, wherein each of the modules includes an input data layer, a kernel layer that is based on the kernel trick, and an output predictions layer.

### Claim 8 (depends on 7)

The method of claim 7, wherein:
at least one of the modules includes an input layer that includes raw input data and other input that includes a plurality of predictions from one of the output predictions layers that is associated with another one of the modules that is interconnected to the at least one of the modules as an adjacent module.

### Claim 9 (depends on 5)

The method of claim 5, wherein:
determining the probability associated with the slot includes determining the probability associated with the slot that is associated with slot-filling, using a softmax function.

### Claim 10 (depends on 9)

The method of claim 9, wherein:
one or more parameters associated with the softmax function are trained using an end-to-end learning algorithm.

### Claim 11 (depends on 10)

The method of claim 10, wherein:
an objective function for learning a matrix of the parameters using the end-to-end learning algorithm is based on a model-based expectation of slot-filling accuracy over an entire training set.

### Claim 12

A computer program product tangibly embodied on a computer-readable storage medium and including executable code that causes at least one data processing apparatus to:
obtain data associated with spoken language; and
initiate an analysis of the obtained data for understanding of the spoken language using a deep convex network that is integrated with a kernel trick.

### Claim 13 (depends on 12)

The computer program product of claim 12, wherein:
the analysis of the obtained data includes one or more of:
classifying one or more domains of the spoken language using the deep convex network that is integrated with the kernel trick, or
classifying one or more intents of the spoken language using the deep convex network that is integrated with the kernel trick.

### Claim 14 (depends on 12)

The computer program product of claim 12, wherein:
the deep convex network includes a plurality of interconnected, stacked modules, wherein each module includes a plurality of layers, wherein each of the modules includes an input data layer, a kernel layer that is based on the kernel trick, and an output predictions layer.

### Claim 15 (depends on 14)

The computer program product of claim 14, wherein:
each one of the modules is associated with respective module-dependent hyper-parameters that include kernel smoothing parameters and regularization parameters.

### Claim 16 (depends on 14)

The computer program product of claim 14, wherein:
each one of the modules includes the respective kernel layer that is based on the kernel trick, in lieu of explicitly determining outputs of one or more sigmoidal hidden layers, wherein the kernel trick provides an equivalent of an infinite number of hidden units.

### Claim 17 (depends on 14)

The computer program product of claim 14, wherein:
at least one of the modules includes an input layer that includes raw input data and other input that includes a plurality of predictions from one of the output predictions layers that is associated with another one of the modules that is interconnected to the at least one of the modules as an adjacent module.

### Claim 18 (depends on 12)

The computer program product of claim 12, wherein the executable code is configured to cause the at least one data processing apparatus to:
determine a probability associated with a slot that is associated with slot-filling, based on local, discriminative features that are extracted using the deep convex network that is integrated with the kernel trick.

### Claim 19 (depends on 18)

The computer program product of claim 18, wherein:
determining the probability associated with the slot includes determining the probability associated with the slot that is associated with slot-filling, using a softmax function, wherein
one or more parameters associated with the softmax function are trained using an end-to-end learning algorithm.

### Claim 20 (depends on 19)

The computer program product of claim 19, wherein:
an objective function for learning a matrix of the parameters using the end-to-end learning algorithm is based on a model-based expectation of slot-filling accuracy over an entire training set.
