# US 2024/0201952 A1 · Artificial intelligence operation system and method thereof > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2024/0201952 A1 - Document ID: US20240201952A1 - Status: Published application - Kind: A1 - Application number: 18/393,565 - Filed: 2023-12-21 - Published: 2024-06-20 - Earliest claimed: 2022-12-19 - Assignees: Electronics and Telecommunications Research Institute - Applicants: Electronics and Telecommunications Research Institute - Inventors: Hyunjeong KWON - Web page: https://patentgrep.com/patent/US20240201952A1 ## CPC - G06F 7/5443: {Sum of products (for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00)} - G06F 9/3001: {Arithmetic instructions} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Disclosed is an artificial intelligence (AI) operation system. The AI operation system includes a plurality of operators, and a host configured to merge nodes constituting a specific attention layer in a transformer model, pre-process specific matrix data among data of the merged node, distribute the preprocessed data and non-preprocessed data to the plurality of operators, and add and normalize operation results of the plurality of operators, wherein the plurality of operators perform a GEneral Matrix Matrix Multiplication (GEMM) operation in parallel using the distributed data. ## Claims ### Claim 1 An artificial intelligence (AI) operation system, comprising: a plurality of operators; and a host configured to merge nodes constituting a specific attention layer in a transformer model, pre-process specific matrix data among data of the merged node, distribute the preprocessed data and non-preprocessed data to the plurality of operators, and add and normalize operation results of the plurality of operators, wherein the plurality of operators perform a GEneral Matrix Matrix Multiplication (GEMM) operation in parallel using the distributed data. ### Claim 2 (depends on 1) The AI operation system of claim 1, wherein the host merges the nodes constituting the attention layer consisting of GEneral Matrix Vector Multiplication (GEMV), Softmax, and GEMV into a single node. ### Claim 3 (depends on 1) The AI operation system of claim 1, wherein the data of the merged node includes at least one of query feature map data, key feature map data, and value feature map data. ### Claim 4 (depends on 3) The AI operation system of claim 3, wherein the host preprocesses the value feature map data among the data of the merged node. ### Claim 5 (depends on 4) The AI operation system of claim 4, wherein the host performs a logarithmic operation on each element of the value feature map data, and performs preprocessing by dividing a value obtained by performing the logarithmic operation by a sum of each element of the query feature map data. ### Claim 6 (depends on 1) The AI operation system of claim 1, further comprising: a memory, wherein the host stores the preprocessed data and the non-preprocessed data in the memory, and extracts data necessary for each operator from the memory and distributes the extracted data. ### Claim 7 (depends on 1) The AI operation system of claim 1, wherein the host divides a new operation generated by merging the nodes into independent operations, distributes the divided operations to the plurality of operators, and distributes data necessary for an operation to be performed by each operator among the preprocessed data and the non-preprocessed data, to each operator. ### Claim 8 (depends on 1) The AI operation system of claim 1, wherein each of the plurality of operators includes an internal memory configured to store the distributed data, a processing element array including a plurality of processing elements, and a controller configured to control the operation of the processing element and data movement between the processing elements in response to a request from the host. ### Claim 9 (depends on 8) The AI operation system of claim 8, wherein the plurality of processing elements include a plurality of adders configured to receive row-direction data and column-direction data among the data stored in the internal memory through the controller and adding the received data, a plurality of multipliers configured to receive the row-direction data among the data stored in the internal memory through the controller and multiply the row-direction data by an output value of the plurality of adders, and a plurality of exponent operators and multipliers configured to perform an exponent operation on an output value of the plurality of multipliers and perform a cumulative multiplication operation on exponent output values. ### Claim 10 (depends on 9) The AI operation system of claim 9, wherein each of the plurality of exponent operators and multipliers includes a lookup table outputting an exponent for the output value of each multiplier. ### Claim 11 (depends on 8) The AI operation system of claim 8, wherein the processing element array is based on a systolic array. ### Claim 12 An AI operation system comprising: a plurality of operators; a host configured to merge nodes constituting a specific attention layer in a transformer model, preprocess specific matrix data among data of the merged nodes to convert GEMV into GEMM, distribute the preprocessed data and non-preprocessed data to the plurality of operators to perform GEMM in parallel in the plurality of operators, and add and normalize operation results of the plurality of operators; and a memory configured to store the preprocessed data and the non-preprocessed data, wherein each of the plurality of operators receives row/column-direction data based on the distributed data and performs an addition operation, receives row-direction data based on the distributed data and performs a multiplication operation on the received row-direction data and a value obtained by performing the addition operation, performs an exponent operation on a value obtained by performing the multiplication operation, and performs a GEMM operation in parallel by performing a cumulative multiplication operation on values obtained by performing the exponent operation. ### Claim 13 (depends on 12) The AI operation system of claim 12, wherein the host merges the nodes constituting the attention layer consisting of GEMV, Softmax, and GEMV, into a single node. ### Claim 14 (depends on 12) The AI operation system of claim 12, wherein the data of the merged node includes at least one of query feature map data, key feature map data, and value feature map data, and the host performs a logarithmic operation on each element of the value feature map data among the data of the merged node, and performs preprocessing by dividing a value obtained by performing the logarithmic operation by a sum of each element of the query feature map data. ### Claim 15 (depends on 12) The AI operation system of claim 12, wherein each of the plurality of operators includes an inner memory configured to store the distributed data; a processing element array including a plurality of processing elements, and a controller configured to control the operation of the processing element and data movement between the processing elements in response to a request from the host. ### Claim 16 (depends on 15) The AI operation system of claim 15, wherein the plurality of processing elements include a plurality of adders configured to receive row-direction data and column-direction data among the data stored in the internal memory through the controller and adding the received data, a plurality of multipliers configured to receive the row-direction data among the data stored in the internal memory through the controller and multiplying the row-direction data by an output value of the plurality of adders, and a plurality of exponent operators and multipliers configured to perform an exponent operation on an output value of the plurality of multipliers and perform a cumulative multiplication operation on exponent output values. ### Claim 17 An AI operation method comprising: merging, by a host, nodes constituting a specific attention layer in a transformer model; preprocessing, by the host, specific matrix data among data of the merged node; distributing, by the host, the preprocessed data and non-preprocessed data to a plurality of operators; performing, by the plurality of operators, a parallel operation using the distributed data; and adding and normalizing, by the host, operation results of the plurality of operators. ### Claim 18 (depends on 17) The AI operation method of claim 17, wherein the merging of the nodes includes merging, by the host, the nodes constituting the attention layer consisting of GEMV, Softmax, and GEMV into a single node. ### Claim 19 (depends on 17) The AI operation method of claim 17, wherein the preprocessing of the specific matrix data includes performing, by the host, a logarithmic operation on each element of value feature map data among data of the merged node including at least one of query feature map data, key feature map data, and the value feature map data. ### Claim 20 (depends on 17) The AI operation method of claim 17, wherein the performing of the parallel operation includes storing, by a controller of each operator, the distributed data in an inner memory, receiving, by a plurality of adders of each operator, row-direction data and column-direction data among the data stored in the internal memory through the controller, and adding the received data, receiving, by a plurality of multiplexers of each operator, the row-direction data among the data stored in the internal memory through the controller and multiplexing the row-direction data by an output value of the plurality of adders, and performing, by a plurality of exponent operators and multipliers of each operator, an exponent operation on an output value of the plurality of multiplexers and performing a cumulative multiplication operation on exponent output values.