# US 2024/0201992 A1 · Artificial intelligence processing apparatus, and data prefetching device and method for artificial intelligence processor > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2024/0201992 A1 - Document ID: US20240201992A1 - Status: Published application - Kind: A1 - Application number: 18/539,012 - Filed: 2023-12-13 - Published: 2024-06-20 - Earliest claimed: 2022-12-15 - Assignees: Electronics and Telecommunications Research Institute - Applicants: Electronics and Telecommunications Research Institute - Inventors: Hyun-Mi KIM - Web page: https://patentgrep.com/patent/US20240201992A1 ## CPC - G06F 9/30036: {Instructions to perform operations on packed data, e.g. vector, tile or matrix operations} - G06F 12/0862: with prefetch - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Disclosed herein are a prefetching device and method for an artificial intelligence processor. The prefetching method includes prefetching data, stored in external off-chip memory, into internal on-chip memory in the artificial intelligence processor, and storing information including an address value and a total amount of matrix operation data in at least one control and status register, as a kernel program is executed, extracting a matrix operation instruction among instructions provided from an instruction cache of the off-chip memory, determining whether prefetching is enabled based on a result of extracting the matrix operation instruction, as prefetching is enabled, determining a number of blocks to be prefetched based on the information stored in the at least one control and status register, and determining a bus burst value corresponding to the determined number of blocks and transmitting the bus burst value as a data request signal through a bus interface. ## Claims ### Claim 1 An artificial intelligence processing apparatus, comprising: an off-chip memory; a main central processing unit configured to execute an artificial intelligence program; and one or more artificial intelligence processors, wherein each of the one or more artificial intelligence processors comprises: a processing core; an on-chip memory; and a prefetcher configured to load data stored in the off-chip memory into the on-chip memory. ### Claim 2 (depends on 1) The artificial intelligence processing apparatus of claim 1, wherein the processing core is implemented as a pair of a floating-point operation-based tensor processing unit configured to perform a matrix operation and a central processing unit configured to perform a normal operation. ### Claim 3 (depends on 2) The artificial intelligence processing apparatus of claim 2, wherein the on-chip memory comprises: an instruction cache configured to store an instruction of the artificial intelligence program; and a data cache configured to store operation data of the artificial intelligence program. ### Claim 4 (depends on 3) The artificial intelligence processing apparatus of claim 3, wherein the prefetcher comprises at least one control and status register configured to store information including an address value and a total amount of matrix operation data. ### Claim 5 (depends on 4) The artificial intelligence processing apparatus of claim 4, wherein the main central processing unit executes: a compiler configured to create machine code by optimizing the artificial intelligence program; and runtime software configured to execute the created machine code, wherein information extracted by the compiler and the runtime software is recorded in the at least one control and status register. ### Claim 6 (depends on 5) The artificial intelligence processing apparatus of claim 5, wherein the main central processing unit records the information extracted by the compiler and the runtime software in the control and status register through an Advanced Peripheral Bus (APB) interface. ### Claim 7 (depends on 5) The artificial intelligence processing apparatus of claim 5, wherein the compiler performs: extracting a matrix operation represented by a nested loop in the artificial intelligence program; performing tiling to allocate the extracted matrix operation data to each of multiple processing cores; generating a matrix operation instruction based on a result of tiling; and creating machine code dedicated for the multiple processing cores from the matrix operation instruction. ### Claim 8 (depends on 5) The artificial intelligence processing apparatus of claim 5, wherein the runtime software performs: separating a kernel program by decoding the artificial intelligence program; allocating a dynamic memory; allocating an address of the matrix operation data; and setting the address of the matrix operation data in the control and status register. ### Claim 9 (depends on 4) The artificial intelligence processing apparatus of claim 4, wherein the prefetcher performs: as a kernel program is executed, extracting a matrix operation instruction among instructions provided from the instruction cache; determining whether prefetching is enabled based on a result of extracting the matrix operation instruction; as prefetching is enabled, determining a number of blocks to be prefetched; and determining a bus burst value corresponding to the determined number of blocks and transmitting the bus burst value as a data request signal through a bus interface. ### Claim 10 (depends on 9) The artificial intelligence processing apparatus of claim 9, wherein the prefetcher further performs: receiving a Program Counter (PC) value of the central processing unit and adjusting the program counter value so that a difference between the address value of the artificial intelligence program, read from the instruction cache, and the program counter value is not increased to a certain distance or more. ### Claim 11 (depends on 9) The artificial intelligence processing apparatus of claim 9, wherein the prefetcher is configured to, when determining whether prefetching is enabled, determine that prefetching is enabled only when a first matrix operation instruction is extracted. ### Claim 12 (depends on 9) The artificial intelligence processing apparatus of claim 9, wherein the prefetcher is configured to, when determining the number of blocks, determine the number of blocks based on the address value and the total amount of data of the matrix operation data, stored in the control and status register. ### Claim 13 (depends on 9) The artificial intelligence processing apparatus of claim 9, wherein the prefetcher is configured to, when determining the bus burst value, determine the bus burst value based on a size of one block of the data cache and a data bandwidth of a bus interface. ### Claim 14 A prefetching device for an artificial intelligence processor, comprising: at least one control and status register configured to prefetch data, stored in an external off-chip memory, into an internal on-chip memory in an artificial intelligence processor, and to store information including an address value and a total amount of matrix operation data; a matrix operation discrimination unit configured to, as a kernel program is executed, extract a matrix operation instruction among instructions provided from an instruction cache of the off-chip memory; a prefetching/non-prefetching determination unit configured to determine whether prefetching is enabled based on a result of extracting the matrix operation instruction; a prefetch block number determination unit configured to, as prefetching is enabled, determine a number of blocks to be prefetched based on the information stored in the at least one control and status register; and a request signal generation unit configured to determine a bus burst value corresponding to the determined number of blocks and transmit the bus burst value as a data request signal through a bus interface. ### Claim 15 (depends on 14) The prefetching device of claim 14, wherein the matrix operation discrimination unit is configured to receive a Program Counter (PC) value of a central processing unit of the artificial intelligence processor and adjust the program counter value so that a difference between the address value of the artificial intelligence program, read from the instruction cache, and the program counter value is not increased to a certain distance or more. ### Claim 16 (depends on 14) The prefetching device of claim 14, wherein the prefetching/non-prefetching determination unit determines that prefetching is enabled only when a first matrix operation instruction is extracted. ### Claim 17 (depends on 14) The prefetching device of claim 14, wherein the prefetch block number determination unit determines the number of blocks to be prefetched based on the address value and the total amount of matrix operation data stored in the control and status register. ### Claim 18 (depends on 14) The prefetching device of claim 14, wherein the request signal generation unit determines the bus burst value based on a size of one block of a data cache in the off-chip memory and a data bandwidth of a bus interface. ### Claim 19 A prefetching method for an artificial intelligence processor, comprising: prefetching data, stored in an external off-chip memory, into an internal on-chip memory in the artificial intelligence processor, and storing information including an address value and a total amount of matrix operation data in at least one control and status register; as a kernel program is executed, extracting a matrix operation instruction among instructions provided from an instruction cache of the off-chip memory; determining whether prefetching is enabled based on a result of extracting the matrix operation instruction; as prefetching is enabled, determining a number of blocks to be prefetched based on the information stored in the at least one control and status register; and determining a bus burst value corresponding to the determined number of blocks and transmitting the bus burst value as a data request signal through a bus interface. ### Claim 20 (depends on 19) The prefetching method of claim 19, wherein: determining the number of block to be prefetched comprises determining the number of blocks to be prefetched based on the address value and the total amount of the matrix operation data stored in the control and status register, and transmitting as the data request signal comprises determining the bus burst value based on a size of one block of a data cache of the off-chip memory and a data bandwidth of the bus interface.