# US 2013/0212052 A1 · Tensor deep stacked neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0212052 A1
- Document ID: US20130212052A1
- Status: Published application
- Kind: A1
- Application number: 13/397,580
- Filed: 2012-02-15
- Published: 2013-08-15
- Earliest claimed: 2012-02-15
- Assignees: Microsoft Corporation
- Inventors: Dong Yu; Li Deng; Brian Hutchinson
- Web page: https://patentgrep.com/patent/US20130212052A1

## CPC

- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A tensor deep stacked neural (T-DSN) network for obtaining predictions for discriminative modeling problems. The T-DSN network and method use bilinear modeling with a tensor representation to map a hidden layer to the predication layer. The T-DSN network is constructed by stacking blocks of a single hidden layer tensor neural network (SHLTNN) on top of each other. The single hidden layer for each block then is separated or divided into a plurality of two or more sections. In some embodiments, the hidden layer is separated into a first hidden layer section and a second hidden layer section. These multiple sections of the hidden layer are combined using a product operator to obtain an implicit hidden layer having a single section. In some embodiments the product operator is a Khatri-Rao product. A prediction is made using the implicit hidden layer and weights, and the output prediction layer is consequently obtained.

## Claims

### Claim 1

A method for mapping from a hidden layer to a prediction layer in a neural network, comprising:
separating the hidden layer into multiple sections;
obtaining a product of each of the multiple sections; and
obtaining the prediction layer using the product.

### Claim 2 (depends on 1)

The method of claim 1, further comprising combining the multiple sections using the product to obtain an implicit hidden layer.

### Claim 3 (depends on 2)

The method of claim 2, further comprising using a Khatri-Rao product as the product.

### Claim 4 (depends on 1)

The method of claim 1, further comprising associating a lower-layer weight with each of the multiple sections.

### Claim 5 (depends on 1)

The method of claim 1, further comprising combining the multiple sections using the product to predict the output layer.

### Claim 6 (depends on 2)

The method of claim 2, further comprising separating the hidden layer into a first hidden layer section and a second hidden layer section.

### Claim 7 (depends on 6)

The method of claim 6, further comprising modeling a three-way interaction between the first hidden layer section, the second hidden layer section, and the prediction layer.

### Claim 8 (depends on 7)

The method of claim 7, further comprising modeling the three-way interaction using a bilinear mapping with a tensor representation to map from both the first hidden layer section and the second hidden layer section to the prediction layer.

### Claim 9 (depends on 6)

The method of claim 6, further comprising using a Khatri-Rao product to combine the two first hidden layer section and the second hidden layer section to obtain the implicit hidden layer.

### Claim 10 (depends on 2)

The method of claim 2, further comprising performing a linear mapping from the implicit hidden layer to the prediction layer.

### Claim 11

A method for constructing a tensor deep stacked neural network, comprising:
stacking blocks to create a deep stacked neural network, where each of the blocks is a single-hidden-layer tensor neural network containing a hidden layer;
separating the hidden layer in each block into multiple sections; and
combining the multiple sections with a product operator to obtain an implicit hidden layer to construct the tensor deep stacked neural network.

### Claim 12 (depends on 11)

The method of claim 11, further comprising obtaining a prediction layer from the implicit hidden layer.

### Claim 13 (depends on 12)

The method of claim 12, further comprising performing the above-mentioned process for each of the blocks.

### Claim 14 (depends on 11)

The method of claim 11, further comprising separating the hidden layer into two sections.

### Claim 15 (depends on 11)

The method of claim 11, further comprising obtaining the prediction layer by mapping from the implicit hidden layer to the prediction layer.

### Claim 16 (depends on 15)

The method of claim 15, further comprising using a Khatri-Rao product to perform the mapping.

### Claim 17

A tensor deep stack neural network, comprising:
a plurality of blocks stacked one block upon another block, each of the plurality of blocks further comprising:
input data;
a first hidden layer section having a corresponding first lower-layer weight for processing the input data;
a second hidden layer section having a corresponding second lower-layer weight for processing the input data;
a prediction layer that is output for a block; and
a bilinear model with a tensor representation that models a three-way interaction between the first hidden layer section, the second hidden layer section, and the prediction layer.

### Claim 18 (depends on 17)

The tensor deep stack neural network of claim 17, further comprising an implicit hidden layer that is a combination of the first hidden layer section and the second hidden layer section.

### Claim 19 (depends on 18)

The tensor deep stack neural network of claim 18, further comprising a Khatri-Rao product that is used to combine the first hidden layer section and the second hidden layer section.

### Claim 20 (depends on 19)

The tensor deep stack neural network of claim 19, wherein each of the plurality of blocks is a single hidden layer tensor neural network.
