Vector engine for efficient beam search
AssigneeAmazon Technologies, Inc.
InventorPaul Gilbert Meyer
Life of this patent
Estimating the term…
Abstract
To provide native hardware support for complex operations such as a beam search operation, an integrated circuit device can be implemented with multiple computational circuit blocks coupled in series to form a pipeline. Each computational circuit block includes an arithmetic logic unit (ALU) circuit having a first numeric input, a second numeric input, a primary result output, and a secondary output. The ALU circuit is programmable to perform an arithmetic operation on the first numeric input and the second numeric input to generate the primary result output, and provide one of the first numeric input or the second numeric input to the secondary output, which is fed back to the ALU circuit.
Claims20
A method for operating a processing circuit to perform a beam search operation, the processing circuit having a plurality of arithmetic logic unit (ALU) stages coupled in series in an ALU pipeline, each ALU stage having a first numeric input, a second numeric input, a primary result output that couples to the first numeric input of a next ALU stage, and a secondary output coupled to a feedback register to provide a feedback value selectable between the first numeric input and the secondary numeric input to feedback to the second numeric input of the ALU stage, the method comprising:
- initializing feedback registers of the ALU stages to a minimum value;
- configuring each ALU to perform a MIN operation, wherein the MIN operation selects a minimum of the first and second numeric inputs to output on the primary result output, and outputs an unselected input of the first and second numeric inputs on the secondary output;
- streaming elements of a tensor into the first numeric input of a first ALU stage along the ALU pipeline to cause a set of largest values of the tensor to be stored in the feedback registers;
- streaming the elements of the tensor into the first numeric input of the first ALU stage along the ALU pipeline while the set of largest values of the tensor are stored in the feedback registers;
- streaming indices corresponding to the elements of the tensor along a shift register in parallel with the elements of the tensor;
- and
- for each element of the tensor that is a first match to a value in the set of largest values of the tensor stored in the feedback registers, storing a corresponding index of that element in an index register.
An integrated circuit device comprising:
- a plurality of computational circuit blocks coupled in series in a pipeline, wherein each computational circuit block includes:
- an arithmetic logic unit (ALU) circuit having a first numeric input, a second numeric input, a primary result output, and a secondary output, wherein the ALU circuit is programmable to perform an arithmetic operation on the first numeric input and the second numeric input to generate the primary result output, and select between the first numeric input and the second numeric input to provide to the secondary output;
- a first multiplexer coupled to the ALU circuit to select the first numeric input for the ALU circuit;
- a second multiplexer coupled to the ALU circuit to select the second numeric input for the ALU circuit;
- an output register configured to receive the primary result output of the ALU and provide the primary result output of the ALU circuit as an output of the computational circuit block;
- and
- a feedback register configured to receive the secondary output and provide the secondary output to the second multiplexer of the ALU.
A method comprising:
- initializing feedback registers of a pipeline of computational circuit blocks to a minimum value, wherein each computational circuit block includes:
- an arithmetic logic unit (ALU) circuit having a first numeric input selected by a first multiplexer, a second numeric input selected by a second multiplexer, a primary result output, and a secondary output;
- a feedback register coupled to the second numeric input;
- and
- an output register to provide the primary result output of the ALU circuit as an output of the computational circuit block:
- configuring each computational circuit block to select a minimum of the first numeric input and the second numeric input to provide on the output of the computational circuit block, and to store an unselected one of the first numeric input and the second numeric input in the corresponding feedback register of the computational circuit block via a feedback path from the secondary output of the ALU circuit;
- inputting a set of data elements into the pipeline of computational circuit blocks to store a set of largest values of the set of data elements in the feedback registers;
- and
- inputting a set of constant values into the pipeline of computational circuit blocks to output the set of largest values of the set of data elements from the pipeline of computational circuit blocks.
Dependencies are inferred from the claim text.