# US 2012/0254086 A1 · Deep convex network with joint use of nonlinear random projection, restricted boltzmann machine and batch-based parallelizable optimization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0254086 A1
- Document ID: US20120254086A1
- Status: Published application
- Kind: A1
- Application number: 13/077,978
- Filed: 2011-03-31
- Published: 2012-10-04
- Earliest claimed: 2011-03-31
- Assignees: Microsoft Corporation
- Inventors: Li Deng; Dong Yu; Alejandro Acero
- Web page: https://patentgrep.com/patent/US20120254086A1

## CPC

- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0499: Feedforward networks
- G06N 3/09: Supervised learning
- G06N 3/02: Neural networks
- G10L 15/16: using artificial neural networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A method is disclosed herein that includes an act of causing a processor to access a deep-structured, layered or hierarchical model, called deep convex network, retained in a computer-readable medium, wherein the deep-structured model comprises a plurality of layers with weights assigned thereto. This layered model can produce the output serving as the scores to combine with transition probabilities between states in a hidden Markov model and language model scores to form a full speech recognizer. The method makes joint use of nonlinear random projections and RBM weights, and it stacks a lower module's output with the raw data to establish its immediately higher module. Batch-based, convex optimization is performed to learn a portion of the deep convex network's weights, rendering it appropriate for parallel computation to accomplish the training. The method can further include the act of jointly substantially optimizing the weights, the transition probabilities, and the language model scores of the deep-structured model using the optimization criterion based on a sequence rather than a set of unrelated frames.

## Claims

### Claim 1

A method comprising the following computer-executable acts:
receiving training data for training a deep convex network that comprises a plurality of interconnected modules, wherein each module in the plurality of interconnected modules comprises a linear layer and a non-linear layer; and
training a portion of the deep convex network in a batch-based manner based at least in part upon the training data, wherein training the deep convex network comprises learning a weight matrix corresponding to output of the non-linear layer of at least one module.

### Claim 2 (depends on 1)

The method of claim 1 configured for execution across a plurality of computing devices.

### Claim 3 (depends on 1)

The method of claim 1, wherein each module is a single-hidden layer neural network.

### Claim 4 (depends on 3)

The method of claim 3, wherein a hidden layer in the single-hidden layer neural network comprises a plurality of sigmoidal units.

### Claim 5 (depends on 1)

The method of claim 1, wherein the deep convex network comprises a lowest module, and wherein input to the lowest module is raw training data.

### Claim 6 (depends on 5)

The method of claim 5, wherein the deep convex network comprises a higher module that is interconnected with the lowest module, and wherein input to the higher module comprises output of the lower module appended with the raw training data.

### Claim 7 (depends on 6)

The method of claim 6, wherein a first weight matrix pertaining to connections between units in the linear layer that correspond to the raw training and units in the non-linear layer in the higher module comprises random numbers generated by a random number generator.

### Claim 8 (depends on 7)

The method of claim 7, wherein a second weight matrix pertaining to connections between units in the linear layer that correspond to output units from the lowest module and the units in the non-linear layer in the higher module comprises weights of separately trained Restricted Boltzmann Machines.

### Claim 9 (depends on 8)

The method of claim 8, wherein a third weight matrix pertaining to connections between units in the non-linear layer and output units of the higher module is learned by way of convex optimization.

### Claim 10 (depends on 9)

The method of claim 9, wherein an algorithm utilized to perform the convex optimization is U=pinv(H)T, where H=σ(ΣWX), where T represents supervised output values in the training data, W is a weight matrix between the input modules and units in the non-linear layer, and X represents the input units corresponding to the raw training data, and wherein σ(ΣWX)=1/(1+exp(−ΣWX)) applied element-wise.

### Claim 11 (depends on 1)

The method of claim 1, wherein the training data is speech data.

### Claim 12 (depends on 1)

The method of claim 1, wherein the training data is image data.

### Claim 13

A system comprising:
a processor; and
a memory that comprises a plurality of components that are executable by the processor, the plurality of components comprising:
a data receiver component that receives training data; and
a training component that trains a deep convex network in a batch-training manner based at least in part upon the training data, wherein the deep convex network comprises a plurality of interconnected modules that each include a linear layer that comprises a plurality of input units and a non-linear layer that comprises a plurality of hidden units, wherein input units of a first module include units output by the plurality of hidden units in a second module.

### Claim 14 (depends on 13)

The system of claim 13, wherein the training component learns a first weight matrix corresponding to the plurality of hidden units of the first module and the plurality of hidden units of the first module through utilization of convex optimization.

### Claim 15 (depends on 14)

The system of claim 14, wherein the training data is speech data.

### Claim 16 (depends on 14)

The system of claim 14, wherein the training data is image data.

### Claim 17 (depends on 14)

The system of claim 14, wherein the first weight matrix is based at least in part upon a second weight matrix that has been assigned to connections between input units of the first module and hidden units of the first module.

### Claim 18 (depends on 17)

The system of claim 17, wherein the second weight matrix comprises a plurality of randomly generated numbers.

### Claim 19 (depends on 17)

The system of claim 17, wherein the second weight matrix comprises weights obtained by training RBMs.

### Claim 20

A computer-readable medium comprising instructions that, when executed by a processor, cause the processor to perform acts comprising:
receiving speech training data;
training a deep convex network in a batch manner through utilization of the training data, wherein the deep convex network comprises a plurality of layered modules that each include a one-hidden-layer neural network, wherein a hidden layer in a first module comprises a plurality of hidden units that are interconnected to a plurality of input units in a linear layer of a second module, wherein training the deep convex network comprises:
learning a plurality of weight matrices corresponding to the plurality of layered modules, such that a weight matrix is learned for each layered module, and wherein the weight matrix is learned by way of complex optimization.
