# US 5,473,730 A · High efficiency learning network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 5,473,730 A
- Document ID: US5473730A
- Status: Granted
- Kind: A
- Application number: 08/150,677
- Filed: 1993-11-09
- Granted: 1995-12-05
- Earliest claimed: 1993-11-09
- Assignees: AT&T IPM Corp.
- Inventors: Patrice Y. Simard
- Web page: https://patentgrep.com/patent/US5473730A

## CPC

- G06N 3/063: using electronic means
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Nodal outputs are discretized to values of S2.sup.n where n is an integer and S is equal to +1 or -1. During forward propagation, this offers the advantage of forming a product of a nodal output and a weight using a simple shift operation rather than a multiply operation. Replacing multiply operations with shift operations through out a neural network improves response times and permits building larger networks that have broader applicability. Training is also improved by increasing the efficiency of backward propagation. The multiplications involved in backward propagation are reduced To shift operations by discretizing the errors associated with each node so that they are represented as 2.sup.n where n is an integer and S is equal to +1 or -1.

## Claims

### Claim 1

A computational network that produces a plurality of network outputs in response to a plurality of network inputs, comprising:
a plurality of first layer computational nodes forming a first layer of a computational network, each of said first layer computational nodes receiving at least one input resulting from at least one of a plurality of computational network inputs to produce a first layer output, said first layer output being discretized to a nonzero integer power of 2; and;
a plurality of second layer computational nodes forming a second layer of the computational network, each of said plurality of second layer computational nodes receiving a plurality of inputs to produce a second layer output, each of said plurality of inputs being a product of a weight value and said first layer output produced by one of said plurality of first layer computational nodes, said second layer output being used to produce at least one of a plurality of network outputs.

### Claim 2 (depends on 1)

The computational network of claim 1, wherein said first layer output is in the form of S2.sup.n where n is a nonzero integer and S is .+-.1.

### Claim 3 (depends on 2)

The computational network of claim 2, further comprising product means for producing said product by shifting said weight value n times.

### Claim 4 (depends on 2)

The computational network of claim 2, further comprising product means for producing said product by summing n and an exponent portion of said weight value.

### Claim 5 (depends on 1)

The computational network of claim 1, wherein said plurality of network inputs are discretized to an integer power of 2.

### Claim 6

A computational network that produces a plurality of network outputs in response to a plurality of network inputs, comprising:
a plurality of first layer computational nodes forming a first layer of a computational network, each of said first layer computational nodes receiving at least one first layer input resulting from at least one of a plurality of computational network inputs to produce a first layer output, said first layer output being discretized to a nonzero integer power of 2; and;
a plurality of second layer computational nodes forming a second layer of the computational network, each of said plurality of second layer computational nodes receiving a plurality of second layer inputs to produce a second layer output, each of said plurality of second layer inputs being a product of a weight value and said first layer output produced by one of said plurality of first layer computational nodes, said second layer output being produced by using a sum of said plurality of inputs as an operand of a first activation function, said second layer output being used to produce at least one of a plurality of network outputs.

### Claim 7 (depends on 6)

The computational network of claim 6, wherein said first layer output is in the form of S2.sup.n where n is a nonzero integer and S is .+-.1.

### Claim 8 (depends on 7)

The computational network of claim 7, further comprising product means for producing said product by shifting said weight value n times.

### Claim 9 (depends on 8)

The computational network of claim 8, wherein said product means shifts said weight value in a first direction for positive values of n and in a second direction for negative values of n.

### Claim 10 (depends on 7)

The computational network of claim 7, further comprising product means for producing said product by summing n and an exponent portion of said weight value.

### Claim 11 (depends on 6)

The computational network of claim 6, wherein said plurality of network inputs are discretized to an integer power of 2.

### Claim 12 (depends on 6)

The computational network of claim 6, wherein each of said first layer computational nodes produces said first layer output by using a sum of a plurality of first layer inputs as an operand of a second activation function.

### Claim 13 (depends on 12)

The computational network of claim 12, wherein said first layer output is in the form of S2.sup.n where n is a nonzero integer and S is .+-.1.

### Claim 14 (depends on 13)

The computational network of claim 13, further comprising product means for producing said product by shifting said weight value n times.

### Claim 15 (depends on 14)

The computational network of claim 14, wherein said product means shifts said weight value in a first direction for positive values of n and in a second direction for negative values of n.

### Claim 16 (depends on 13)

The computational network of claim 13, further comprising product means for producing said product by summing n and an exponent portion of said weight value.

### Claim 17 (depends on 12)

The computational network of claim 12, wherein said plurality of network inputs are discretized to an integer power of 2.

### Claim 18 (depends on 12)

The computational network of claim 12, wherein said first and second activation functions are the same.

### Claim 19

A method of recognizing a user's input, comprising the steps of:
forming a first plurality of values representative of the user's input, said first plurality of values being discretized to a nonzero integer power of 2;
producing a first plurality of products using said first plurality of values and a first plurality of weights;
producing a second plurality of products using said first plurality of values and a second plurality of weights;
using a first computational node belonging to a plurality of first layer computational nodes to produce a first output by forming a sum of said first plurality of products and by using said sum of said first plurality of products as an operand of a first activation function;
using a second computational node belonging to said plurality of first layer computational nodes to produce a second output by forming a sum of said second plurality of products and by using said sum of said second plurality of products as an operand of a second activation function;
forming a first product using said first output and a first weight value;
forming a second product using said second output and a second weight value;
using a second layer computational node belonging to a plurality of second layer computational nodes to produce a second layer output by forming a sum of said first and second products and by using said sum of said first and second products as an operand of a third activation function; and
using said second layer output to produce at least one of a plurality of network outputs that identify the user's input.

### Claim 20 (depends on 19)

The method of claim 19, wherein said user's input is an utterance.

### Claim 21 (depends on 19)

The method of claim 19, wherein said user's input is a handwritten symbol.

### Claim 22 (depends on 19)

The method of claim 19, further comprising adjusting said first plurality of weights using an error value discretized to an integer power of 2.

### Claim 23 (depends on 22)

The method of claim 22, further comprising adjusting said first plurality of weights using a learning rate discretized to an integer power of 2.

### Claim 24 (depends on 22)

The method of claim 22 wherein said error value is in the form S2.sup.n where n is an integer and S is .+-.1.

### Claim 25 (depends on 22)

The method of claim 22, wherein said step of adjusting said first plurality of weights comprises forming a product of said error value and a derivative value discretized to a power of 2, said discretized derivative value obtained by using said sum of said first plurality of products as an argument of a derivative function, said derivative function being a derivative of an approximation of said first activation function.

### Claim 26 (depends on 25)

The method of claim 25, wherein said derivative value is in the form S2.sup.n where n is an integer and S is .+-.1.

### Claim 27 (depends on 25)

The method of claim 25, wherein said derivative value is discretized to a value of 1 or 0.

### Claim 28 (depends on 25)

The method of claim 25, further comprising adjusting said first plurality of weights using a learning rate discretized to an integer power of 2.

### Claim 29 (depends on 19)

The method of claim 19, further comprising adjusting said first plurality of weights using a derivative value discretized to a power of 2, said discretized derivative value obtained by using said sum of said first plurality of products as an argument of a derivative function, said derivative function being a derivative of an approximation of said first activation function.

### Claim 30 (depends on 29)

The method of claim 29, wherein said derivative value is in the form S2.sup.n where n is an integer and S is .+-.1.

### Claim 31 (depends on 29)

The method of claim 29, further comprising adjusting said first plurality of weights using a learning rate discretized to a power of 2.
