# US 10,762,164 B2 · Vector and matrix computing device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,762,164 B2
- Document ID: US10762164B2
- Status: Granted
- Kind: B2
- Application number: 16/039,803
- Filed: 2018-07-19
- Published: 2018-11-15
- Granted: 2020-09-01
- Earliest claimed: 2016-01-20
- Assignees: Cambricon Technologies Corporation Limited
- Applicants: Cambricon Technologies Corporation Limited
- Inventors: Tianshi Chen; Xiao Zhang; Shaoli Liu; Yunji Chen
- Web page: https://patentgrep.com/patent/US10762164B2

## CPC

- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06F 7/4873: {Dividing}
- G06F 7/4876: {Multiplying}
- G06F 9/30036: {Instructions to perform operations on packed data, e.g. vector, tile or matrix operations}
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computing device and related products are provided. The computing device is configured to perform machine learning calculations. The computing device includes an operation unit, a controller unit, and a storage unit. The storage unit includes a data input/output (I/O) unit, a register, and a cache. Technical solution provided by the present disclosure has advantages of fast calculation speed and energy saving.

## Claims

### Claim 1

A computing device, configured to perform machine learning calculations, comprising:
a storage unit that includes a data input/output (I/O) unit and one or any combination of a register and a cache;
a data I/O unit configured to acquire data, a machine learning model, and calculation instructions, wherein the storage unit is configured to store the machine learning model and the data; and
a controller unit configured to extract a first calculation instruction from the storage unit, to obtain an operation code and an operation field of the first calculation instruction by parsing the first calculation instruction, to extract data corresponding to the operation field, and to send the operation code and the data corresponding to the operation field to the operation unit;
wherein the operation field includes at least a register number that identifies the register or an immediate value;
wherein the operation code is an operation code of a matrix calculation instruction; and
wherein the operation unit is configured to obtain a result of the first calculation instruction by performing an operation corresponding to the operation code on the data corresponding to the operation field, according to the operation code.

### Claim 2 (depends on 1)

The device of claim 1, wherein the acquired data includes input data, weight data, and output data.

### Claim 3 (depends on 2)

The device of claim 2, wherein the operation unit comprises:
a plurality of operation modules, configured to form a structure of n-stage pipeline and to perform calculations of the n-stage pipeline; and
the operation unit is configured to:
obtain a first result by performing a calculation of a first stage pipeline on the input data and the weight data;
obtain a second result by performing a calculation of a second stage pipeline on the first result input in the second stage pipeline;
obtain a nth result by performing a calculation of a nth stage pipeline on a (n−1)th result input in the nth stage pipeline; and
input the nth result to the storage unit; wherein n is an integer greater than or equal to 2.

### Claim 4 (depends on 3)

The device of claim 3, wherein the operation unit comprises a structure of two-stage pipeline, and the first calculation instruction executed comprises a matrix and scalar calculation instruction;
wherein the matrix and scalar calculation instruction comprises:
a vector-add-scalar instruction (VAS), a scalar-subtract-vector instruction (SSV), a vector-multiply-scalar instruction (VMS), a scalar-divide-vector instruction (SDV), a matrix-add-scalar instruction, a scalar-subtract-matrix instruction, a matrix-multiply-scalar instruction, a scalar-divide-matrix instruction, a vector addition instruction (VA) or a vector addition instruction (VA), a matrix addition instruction (MA), a matrix-multiply-scalar instruction (VMS), a matrix addition instruction (MA), and a matrix subtraction instruction (MS);
wherein the controller unit is configured to:
extract the matrix and scalar calculation instruction from the storage unit;
obtain a first operation code and a first operation field of the matrix and scalar calculation instruction by parsing the matrix and scalar calculation instruction;
extract input data and weight data corresponding to the first operation field; and
send the first operation code as well as the input data and the weight data corresponding to the first operation field to the operation unit;
wherein an operation module in a structure of a first stage pipeline of the operation unit is configured to obtain an intermediate result by performing an operation corresponding to the first operation code on the input data and the weight data; and
wherein an operation module in a structure of a second stage pipeline of the operation unit is configured to obtain the result of the first calculation instruction by performing a subsequent operation on the intermediate result.

### Claim 5 (depends on 3)

The device of claim 3, wherein the operation unit comprises a structure of three-stage pipeline, and the first calculation instruction executed comprises a multiplication instruction;
wherein the multiplication instruction comprises:
a vector-multiply-vector instruction (VMV), a matrix-multiply-matrix instruction, a matrix-multiply-vector instruction (MMV), and a vector-multiply-matrix instruction (VMM);
wherein the controller unit is configured to:
extract the multiplication instruction from the storage unit;
obtain a first operation code and a first operation field of the multiplication instruction by parsing the multiplication instruction;
extract input data and weight data corresponding to the first operation field; and
send the first operation code as well as the input data and the weight data corresponding to the first operation field to the operation unit;
wherein an operation module in a structure of a first stage pipeline of the operation unit is configured to obtain a multiplication result by performing a multiplication operation on the input data and the weight data received;
wherein an operation module in a structure of a second stage pipeline of the operation unit is configured to:
obtain an intermediate result by performing an addition operation on the multiplication result sent to the operation module in the structure of the second stage pipeline; and
send the intermediate result to an operation module in a structure of a third stage pipeline; and
wherein the operation module in the structure of the third stage pipeline is configured to obtain a result of the multiplication instruction by stitching and combining the intermediate result.

### Claim 6 (depends on 3)

The device of claim 3, wherein the operation unit comprises a structure of three-stage pipeline, and the first calculation instruction executed comprises a division instruction;
wherein the division instruction comprises a vector division instruction (VD) and a matrix division instruction;
wherein the controller unit is configured to:
extract the division instruction from the storage unit;
obtain a first operation code and a first operation field of the division instruction by parsing the division instruction;
extract input data and weight data corresponding to the first operation field; and
send the first operation code as well as the input data and the weight data corresponding to the first operation field to the operation unit;
wherein an operation module in a structure of a first stage pipeline is configured to:
obtain inverse weight data by performing an inversion on the weight data; and
send the input data and the inverse weight data to an operation module in a structure of a second stage pipeline;
wherein the operation module in the structure of the second stage pipeline is configured to:
obtain an intermediate result by performing an inner product operation on the input data and the inverse weight data; and
send the intermediate result to an operation module in a structure of a third stage pipeline; and
wherein the operation module in the structure of the third stage pipeline is configured to obtain a result of the division instruction by stitching and combining the intermediate result.

### Claim 7 (depends on 1)

The device of claim 1, wherein the controller unit comprises:
an instruction cache unit configured to cache the first calculation instruction;
an instruction processing unit configured to obtain the operation code and the operation field of the first calculation instruction by parsing the first calculation instruction; and
a storage queue unit configured to store an instruction queue, wherein the instruction queue includes one or more calculation instructions or operation codes to be executed arranged in a sequence of the instruction queue.

### Claim 8 (depends on 7)

The device of claim 7, wherein the controller unit further comprises
a dependency relationship processing unit configured to:
determine whether there is an associated relationship between the first calculation instruction and a zeroth calculation instruction stored prior to the first calculation instruction;
cache the first calculation instruction to the instruction cache unit, based on a determination that there is the associated relationship between the first calculation instruction and the zeroth calculation instruction; and
transmit the first calculation instruction extracted from the instruction cache unit to the operation unit after execution of the zeroth calculation instruction is completed;
the dependency relationship processing unit configured to determine whether there is the associated relationship between the first calculation instruction and the zeroth calculation instruction prior to the first calculation instruction is configured to:
extract a first storage address space of the operation field of the first calculation instruction, according to the first calculation instruction;
extract a zeroth storage address space of an operation field of the zeroth calculation instruction, according to the zeroth calculation instruction; and
determine there is the associated relationship between the first calculation instruction and the zeroth calculation instruction when there is an overlapping area of the first storage address space and the zeroth storage address space; or
determine there is no associated relationship between the first calculation instruction and the zeroth calculation instruction when there is no overlapping area of the first storage address space and the zeroth storage address space.

### Claim 9 (depends on 1)

The device of claim 1, wherein the matrix calculation instruction comprises one or any combination of:
a vector AND vector instruction (VAV), a vector AND instruction (VAND), a vector OR vector instruction (VOV), a vector OR instruction (VOR), a vector exponent instruction (VE), a vector logarithm instruction (VL), a vector greater than instruction (VGT), a vector equal decision instruction (VEQ), a vector invert instruction (VINV), a vector merge instruction (VMER), a vector maximum instruction (VMAX), a scalar to vector instruction (STV), a scalar to vector Pos N instruction (STVPN), a vector Pos N to scalar instruction (VPNTS), a vector retrieval instruction (VR), a vector dot product instruction (VP), a random vector instruction (RV), a vector cyclic shift instruction (VCS), a vector load instruction (VLOAD), a vector storage instruction (VS), a vector movement instruction (VMOVE), a matrix retrieval instruction (MR), a matrix load instruction (ML), a matrix storage instruction (MS), and a matrix movement instruction (MMOVE).

### Claim 10 (depends on 1)

A machine learning operation device, comprising at least one computing device of claim 1, wherein
the machine learning operation device is configured to acquire data to be operated and control information from other processing devices, to perform a specified machine learning operation, and to send a result of performing the specified machine learning operation to other processing devices through an input/output (I/O) interface;
at least two computing devices are configured to connect with each other and to transmit data through a specific structure when the machine learning operation device comprises the at least two computing devices;
the at least two computing devices are configured to interconnect with each other and to transmit data via a peripheral component interconnect express (PCIE) bus to support larger-scale machine learning operations; and
the at least two computing devices share a same control system or have their own control systems, share a memory or have their own memories, and have an interconnection mode of an arbitrary interconnection topology.

### Claim 11

A machine learning computing method, applicable to a computing device, the computing device comprising an operation unit, a controller unit, and a storage unit;
wherein the storage unit comprises a data input/output (I/O) unit and one or any combination of a register and a cache;
acquiring, by the data I/O unit, data, a machine learning model, and calculation instructions, wherein the storage unit is configured to store the machine learning model and the data;
extracting, by the controller unit, a first calculation instruction from the storage unit, obtaining, by the controller unit, an operation code and an operation field of the first calculation instruction by parsing the first calculation instruction, extracting, by the controller unit, data corresponding to the operation field, and sending, by the controller unit, the operation code and the data corresponding to the operation field to the operation unit;
wherein the operation field includes at least a register number that identifies the register or an immediate value;
wherein the operation code is an operation code of a matrix calculation instruction; and
obtaining, by the operation unit, a result of the first calculation instruction by performing an operation corresponding to the operation code on the data corresponding to the operation field, according to the operation code.

### Claim 12 (depends on 11)

The method of claim 11, wherein the acquired data comprises input data, weight data, and output data.

### Claim 13 (depends on 12)

The method of claim 12, wherein
the operation unit comprises:
a plurality of operation modules, configured to form a structure of n-stage pipeline and to perform calculations of the n-stage pipeline; and
the operation unit is configured to:
obtain a first result by performing a calculation of a first stage pipeline on the input data and the weight data;
obtain a second result by performing a calculation of a second stage pipeline on the first result input in the second stage pipeline;
obtain a nth result by performing a calculation of a nth stage pipeline on a (n−1)th result input in the nth stage pipeline; and
input the nth result to the storage unit; wherein n is an integer greater than or equal to 2.

### Claim 14 (depends on 13)

The method of claim 13, wherein the operation unit comprises a structure of two-stage pipeline, and executing the first calculation instruction comprises:
executing a matrix and scalar calculation instruction;
wherein the matrix and scalar calculation instruction comprises:
a vector-add-scalar instruction (VAS), a scalar-subtract-vector instruction (SSV), a vector-multiply-scalar instruction (VMS), a scalar-divide-vector instruction (SDV), a matrix-add-scalar instruction, a scalar-subtract-matrix instruction, a matrix-multiply-scalar instruction, a scalar-divide-matrix instruction, a vector addition instruction (VA), a matrix addition instruction (MA), a matrix-multiply-scalar instruction (MMS), a matrix addition instruction (MA), and a matrix subtraction instruction (MS);
extracting, by the controller unit, the matrix and scalar calculation instruction from the storage unit;
obtaining, by the controller unit, a first operation code and a first operation field of the matrix and scalar calculation instruction by parsing the matrix and scalar calculation instruction;
extracting, by the controller unit, input data and weight data corresponding to the first operation field;
sending, by the controller unit, the first operation code as well as the input data and the weight data corresponding to the first operation field to the operation unit;
obtaining, by an operation module in a structure of a first stage pipeline of the operation unit, an intermediate result by performing an operation corresponding to the first operation code on the input data and the weight data; and
obtaining, by an operation module in a structure of a second stage pipeline of the operation unit, the result of the first calculation instruction by performing a subsequent operation on the intermediate result.

### Claim 15 (depends on 13)

The method of claim 13, wherein the operation unit comprises a structure of three-stage pipeline, and executing the first calculation instruction comprises:
executing a multiplication instruction;
wherein the multiplication instruction comprises:
a vector-multiply-vector instruction (VMV), a matrix-multiply-matrix instruction, a matrix-multiply-vector instruction (MMV), and a vector-multiply-matrix instruction (VMM);
extracting, by the controller unit, the multiplication instruction from the storage unit;
obtaining, by the controller unit, a first operation code and a first operation field of the multiplication instruction by parsing the multiplication instruction;
extracting, by the controller unit, input data and weight data corresponding to the first operation field;
sending, by the controller unit, the first operation code as well as the input data and the weight data corresponding to the first operation field to the operation unit;
obtaining, by an operation module in a structure of a first stage pipeline of the operation unit, a multiplication result by performing a multiplication operation on the input data and the weight data received;
obtaining, by an operation module in a structure of a second stage pipeline of the operation unit, an intermediate result by performing an addition operation on the multiplication result sent to the operation module in the structure of the second stage pipeline;
sending, by the operation module in the structure of the second stage pipeline of the operation unit, the intermediate result to an operation module in a structure of a third stage pipeline; and
obtaining, by the operation module in the structure of the third stage pipeline, a result of the multiplication instruction by stitching and combining the intermediate result.

### Claim 16 (depends on 13)

The method of claim 13, wherein the operation unit comprises a structure of three-stage pipeline, and executing the first calculation instruction comprises:
executing a division instruction;
wherein the division instruction comprises a vector division instruction (VD) and a matrix division instruction;
extracting, by the controller unit, the division instruction from the storage unit;
obtaining, by the controller unit, a first operation code and a first operation field of the division instruction by parsing the division instruction;
extracting, by the controller unit, input data and weight data corresponding to the first operation field;
sending, by the controller unit, the first operation code as well as the input data and the weight data corresponding to the first operation field to the operation unit;
obtaining, by an operation module in a structure of a first stage pipeline, inverse weight data by performing an inversion on the weight data;
sending, by the operation module in the structure of the first stage pipeline, the input data and the inverse weight data to an operation module in a structure of a second stage pipeline;
obtaining, by the operation module in the structure of the second stage pipeline, an intermediate result by performing an inner product operation on the input data and the inverse weight data;
sending, by the operation module in the structure of the second stage pipeline, the intermediate result to an operation module in a structure of a third stage pipeline; and
obtaining, by the operation module in the structure of the third stage pipeline, a result of the division instruction by stitching and combining the intermediate result.

### Claim 17 (depends on 11)

The method of claim 11, wherein the controller unit comprises:
an instruction cache unit, configured to cache the first calculation instruction;
an instruction processing unit, configured to obtain the operation code and the operation field of the first calculation instruction by parsing the first calculation instruction; and
a storage queue unit, configured to store an instruction queue, wherein the instruction queue comprises a plurality of calculation instructions or operation codes to be executed arranged in a sequence of the instruction queue.

### Claim 18 (depends on 17)

The method of claim 17, wherein
the controller unit further comprises a dependency relationship processing unit, configured to:
determine whether there is an associated relationship between the first calculation instruction and a zeroth calculation instruction prior to the first calculation instruction;
cache the first calculation instruction to the instruction cache unit, based on a determination that there is the associated relationship between the first calculation instruction and the zeroth calculation instruction; and
transmit the first calculation instruction extracted from the instruction cache unit to the operation unit after execution of the zeroth calculation instruction is completed;
determining whether there is the associated relationship between the first calculation instruction and the zeroth calculation instruction prior to the first calculation instruction comprises:
extracting a first storage address space of the operation field of the first calculation instruction, according to the first calculation instruction;
extracting a zeroth storage address space of an operation field of the zeroth calculation instruction, according to the zeroth calculation instruction; and
determining there is the associated relationship between the first calculation instruction and the zeroth calculation instruction when there is an overlapping area of the first storage address space and the zeroth storage address space; or
determining there is no associated relationship between the first calculation instruction and the zeroth calculation instruction when there is no overlapping area of the first storage address space and the zeroth storage address space.

### Claim 19 (depends on 11)

The method of claim 11, wherein the matrix calculation instruction comprises one or any combination of:
a vector AND vector instruction (VAV), a vector AND instruction (VAND), a vector OR vector instruction (VOV), a vector OR instruction (VOR), a vector exponent instruction (VE), a vector logarithm instruction (VL), a vector greater than instruction (VGT), a vector equal decision instruction (VEQ), a vector invert instruction (VINV), a vector merge instruction (VMER), a vector maximum instruction (VMAX), a scalar to vector instruction (STV), a scalar to vector Pos N instruction (STVPN), a vector Pos N to scalar instruction (VPNTS), a vector retrieval instruction (VR), a vector dot product instruction (VP), a random vector instruction (RV), a vector cyclic shift instruction (VCS), a vector load instruction (VLOAD), a vector storage instruction (VS), a vector movement instruction (VMOVE), a matrix retrieval instruction (MR), a matrix load instruction (ML), a matrix storage instruction (MS), and a matrix movement instruction (MMOVE).