# US 11,443,536 B2 · System and methods for efficiently implementing a convolutional neural network incorporating binarized filter and convolution operation for performing image classification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,536 B2
- Document ID: US11443536B2
- Status: Granted
- Kind: B2
- Application number: 16/430,123
- Filed: 2019-06-03
- Published: 2019-09-19
- Granted: 2022-09-13
- Earliest claimed: 2016-04-14
- Assignees: The Allen Institute for Artificial Intelligence
- Applicants: The Allen Institute for Artificial Intelligence
- Inventors: Ali Farhadi; Mohammad Rastegari; Vicente Ignacio Ordonez Roman
- Web page: https://patentgrep.com/patent/US11443536B2

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 18/24: Classification techniques
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/09: Supervised learning
- G06V 10/82: using neural networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems, apparatuses, and methods for efficiently and accurately processing an image in order to detect and identify one or more objects contained in the image, and methods that may be implemented on mobile or other resource constrained devices. Embodiments of the invention introduce simple, efficient, and accurate approximations to the functions performed by a convolutional neural network (CNN); this is achieved by binarization (i.e., converting one form of data to binary values) of the weights and of the intermediate representations of data in a convolutional neural network. The inventive binarization methods include optimization processes that determine the best approximations of the convolution operations that are part of implementing a CNN using binary operations.

## Claims

### Claim 1

A method comprising:
inputting data representing an image into a convolutional neural network (CNN), the CNN including a plurality of convolutional layers, a set of filters or weights for at least one of the convolutional layers, and a set of input data to the at least one of the convolutional layers;
performing a convolution operation between the set of input data and the set of filters or weights by determining a product of a scaling factor and a binary representation of the set of filters or weights convolved with the set of input data; and
applying an operation to an output of a last of the plurality of convolutional layers.

### Claim 2 (depends on 1)

The method of claim 1, wherein the binary representation of the set of filters or weights is a sign of weight values, and the scaling factor is an average of absolute weight values.

### Claim 3 (depends on 1)

The method of claim 1, further comprising representing the set of input data by a product of a scaling factor and a binary representation of the set of input data.

### Claim 4 (depends on 3)

The method of claim 3, wherein the binary representation of the set of input data is a sign of input data values, and the scaling factor is an average of absolute input data values.

### Claim 5 (depends on 4)

The method of claim 4, wherein the convolution operation is implemented by a combination of an XNOR operation and a bit-counting operation.

### Claim 6 (depends on 1)

The method of claim 1, further comprising processing the data representing the image prior to inputting the data into the convolutional neural network, wherein the processing the data includes one or more of reducing noise in the data, scaling the data, or sampling the data.

### Claim 7 (depends on 1)

The method of claim 1, wherein the scaling factor is determined by solving an optimization problem of a form given by:
J(B,α)=∥W−αB∥2
α*,B*=argmin J(B,α)
α,B,
where α is the scaling factor, B is a binary representation of the set of filters or weights, and W is the set of filters or weights.

### Claim 8 (depends on 1)

The method of claim 1, further comprising identifying, based on applying the operation to the output, an object contained in the image.

### Claim 9 (depends on 8)

The method of claim 8, wherein the object contained in the image is a person.

### Claim 10 (depends on 8)

The method of claim 8, further comprising authorizing use based on the identified object.

### Claim 11 (depends on 8)

The method of claim 8, further comprising authorizing access based on the identified object.

### Claim 12 (depends on 8)

The method of claim 8, further comprising controlling a vehicle based on the identified object.

### Claim 13

A method comprising:
inputting data representing an image into a convolutional neural network (CNN), the CNN including a plurality of convolutional layers, a set of filters or weights for at least one of the convolutional layers, and a set of input data to the at least one of the convolutional layers;
performing a convolution operation between the set of input data and the set of filters or weights by determining a product of a scaling factor and a binary representation of the set of input data convolved with the set of filters or weights; and
applying an operation to an output of a last of the plurality of convolutional layers.

### Claim 14 (depends on 13)

The method of claim 13, wherein the binary representation of the set of input data is a sign of data values, and the scaling factor is an average of absolute input data values.

### Claim 15 (depends on 13)

The method of claim 13, further comprising representing the set of filters or weights by a product of a scaling factor and a binary representation of the set of filters or weights.

### Claim 16 (depends on 15)

The method of claim 15, wherein the binary representation of the set of filters or weights is a sign of weight values, and the scaling factor is an average of absolute weight values.

### Claim 17 (depends on 16)

The method of claim 16, wherein the convolution operation is implemented by a combination of an XNOR operation and a bit-counting operation.

### Claim 18 (depends on 13)

The method of claim 13, further comprising processing the data representing the image prior to inputting the data into the convolutional neural network, wherein the processing the data includes one or more of reducing noise in the data, scaling the data, or sampling the data.

### Claim 19 (depends on 13)

The method of claim 13, wherein the scaling factor is determined by solving an optimization problem of a form given by:
J(B,α)=∥W−αB∥2
α*,B*=argmin J(B,α)
α,B,
where α is the scaling factor, B is a binary representation of the set of filters or weights, and W is the set of filters or weights.

### Claim 20 (depends on 13)

The method of claim 13, further comprising identifying, based on applying the operation to the output, an object contained in the image.

### Claim 21 (depends on 20)

The method of claim 20, wherein the object contained in the image is a person.

### Claim 22 (depends on 20)

The method of claim 20, further comprising authorizing use based on the identified object.

### Claim 23 (depends on 20)

The method of claim 20, further comprising authorizing access based on the identified object.

### Claim 24 (depends on 20)

The method of claim 20, further comprising controlling a vehicle based on the identified object.

### Claim 25

A system comprising:
a set of processing blocks, wherein the set of processing blocks includes a normalization block, an activation block, a convolution block, and a pooling block, with image input data being processed in that order;
a computer-implemented process for representing the convolution block, wherein the computer-implemented process represents a convolution operation between a set of data and a set of filters or weights by a product of a scaling factor and a binary representation of the set of filters or weights convolved with the set of data; and
a classifier applied to an output of the pooling block.

### Claim 26 (depends on 25)

The system of claim 25, further comprising a computer-implemented process for representing data by a product of a scaling factor and a binary representation of the data.

### Claim 27 (depends on 26)

The system of claim 26, wherein the convolution operation is implemented by a combination of an XNOR operation and a bit-counting operation.

### Claim 28 (depends on 26)

The system of claim 26, wherein the binary representation of the set of data is a sign of data values, and the scaling factor is an average of absolute data values.

### Claim 29 (depends on 25)

The system of claim 25, wherein the binary representation of the set of filters or weights is a sign of weight values, and the scaling factor is an average of absolute weight values.

### Claim 30 (depends on 25)

The system of claim 25, wherein the scaling factor is determined by solving an optimization problem of a form given by:
J(B,α)=∥W−αB∥2
α*,B*=argmin J(B,α)
α,B,
where α is the scaling factor, B is a binary representation of the set of filters or weights, and W is the set of filters or weights.

### Claim 31 (depends on 25)

The system of claim 25, wherein the computer implemented process further comprising identifying, based on an output of the classifier, an object contained in the image input data.

### Claim 32 (depends on 31)

The system of claim 31, wherein the object contained in the image input data is a person.

### Claim 33 (depends on 31)

The system of claim 31, wherein the computer implemented process further comprises authorizing use based on the identified object.

### Claim 34 (depends on 31)

The system of claim 31, wherein the computer implemented process further comprises authorizing access based on the identified object.

### Claim 35 (depends on 31)

The system of claim 31, wherein the computer implemented process further comprises controlling a vehicle based on the identified object.