# US 12,112,175 B1 · Method and apparatus for performing machine learning operations in parallel on machine learning hardware > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,175 B1 - Document ID: US12112175B1 - Status: Granted - Kind: B1 - Application number: 17/590,994 - Filed: 2022-02-02 - Granted: 2024-10-08 - Earliest claimed: 2018-02-08 - Assignees: Marvell Asia Pte Ltd - Applicants: Marvell Asia Pte Ltd - Inventors: Ulf Hanebutte; Avinash Sodani - Web page: https://patentgrep.com/patent/US12112175B1 ## CPC - G06F 8/45: {Exploiting coarse grain parallelism in compilation, i.e. parallelism between groups of instructions} - G06F 9/3879: {for non-native instruction execution, e.g. executing a command; for Java instruction set} - G06F 8/453: {Data distribution} - G06F 9/30021: {Compare instructions, e.g. Greater-Than, Equal-To, MINMAX} - G06F 9/30036: {Instructions to perform operations on packed data, e.g. vector, tile or matrix operations} - G06F 9/30174: {for non-native instruction set, e.g. Javabyte, legacy code} - G06F 9/3836: {Instruction issuing, e.g. dynamic instruction scheduling or out of order instruction execution} - G06F 9/3851: {from multiple instruction streams, e.g. multistreaming} - G06F 9/3877: {using a secondary processor, e.g. coprocessor (peripheral processor G06F13/12)} - G06F 9/3885: {using a plurality of independent parallel functional units} - G06F 9/38873: {Iterative single instructions for multiple data lanes [SIMD]} - G06F 15/7807: {System on chip, i.e. computer system on a single chip; System in package, i.e. computer system on one or more chips in a single package} - G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} - G06N 3/045: Combinations of networks - G06N 3/063: using electronic means - G06N 20/00: Machine learning - G06N 20/10: using kernel methods, e.g. support vector machines [SVM] - G06F 9/3001: {Arithmetic instructions} - G06F 15/7864: {on more than one IC chip} - G06F 15/8023: {Two dimensional arrays, e.g. mesh, torus} - G06F 2212/602: Details relating to cache prefetching - G06N 5/04: Inference or reasoning models - G06N 20/20: Ensemble learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method includes receiving a set of data. The set of data is divided into a plurality of data portions. The method includes transmitting the plurality of data portions to a plurality of processing tiles, wherein each data portion of the plurality of data portions is associated with a processing tile of a plurality of tiles. Each processing tile of the plurality of tiles performs at least one local operation on its respective data portion to form a local result. The method includes exchanging local results between the plurality of processing tiles. Moreover, the method includes calculating a global value based on the local results. The method further includes performing at least one local operation by each processing tile of the plurality of tiles on its respective data portion based on the global value to form a computed result. ## Claims ### Claim 1 A computer implemented method, comprising: receiving a set of data; dividing the set of data into a plurality of data portions; transmitting the plurality of data portions to a plurality of processing tiles, wherein each data portion of the plurality of data portions is associated with a processing tile of a plurality of tiles; performing by each processing tile of the plurality of tiles at least one local operation on its respective data portion to form a local maxima; exchanging local maximas between the plurality of processing tiles; calculating a global maximum value based on the local maximas; performing by each processing tile of the plurality of tiles a subtraction operation of the global maximum value from each data input its respective data portion to form a subtraction result; performing by each processing tile of the plurality of tiles an exponential operation on the subtraction result to form exponential results; forming a sum of the exponential results from the plurality of tiles; and inverting the sum to form a scaled value. ### Claim 2 (depends on 1) The computer implemented method of claim 1, wherein the at least one local operation that forms the final result is an elementwise multiplication of the scaled value and the exponential results. ### Claim 3 (depends on 1) The computer implemented method of claim 1, wherein the plurality of processing tiles are within an inference engine of a machine learning hardware. ### Claim 4 (depends on 1) The computer implemented method of claim 1, wherein the plurality of processing tiles operate simultaneously when performing the at least one local operation. ### Claim 5 (depends on 1) The computer implemented method of claim 1, wherein the plurality of processing tiles operate simultaneously when calculating the global maximum value. ### Claim 6 (depends on 1) The computer implemented method of claim 1, wherein the plurality of processing tiles operate simultaneously when performing the at least one local operation on its respective data portion based on the global maximum value to form the final result. ### Claim 7 A computer implemented method, comprising: receiving a set of data; dividing the set of data into a plurality of data portions; transmitting the plurality of data portions to a plurality of processing tiles, wherein each data portion of the plurality of data portions is associated with a processing tile of a plurality of tiles; each processing tile of the plurality of tiles performing at least one operation on its respective data portion to form a computed result without exchanging data related to the at least one operation with other processing tiles of the plurality of tiles, wherein the at least one operation is finding a maximum value associated with the respective data portion; performing by each processing tile of the plurality of tiles a subtraction operation of the maximum value from each data input its respective data portion to form a subtraction result; performing an exponential operation on the subtraction result to form exponential results on each processing tile of the plurality of tiles; forming a sum of the exponential results on each processing tile of the plurality of tiles; and inverting the sum of exponential results to form a scaled value on each processing tile of the plurality of tiles. ### Claim 8 (depends on 7) The computer implemented method of claim 7, wherein the at least one operation that forms the final result is an elementwise multiplication of the scaled value and the exponential results. ### Claim 9 (depends on 7) The computer implemented method of claim 7, wherein the plurality of processing tiles are within an inference engine of a machine learning hardware. ### Claim 10 (depends on 7) The computer implemented method of claim 7, wherein the plurality of processing tiles operate simultaneously when performing the at least one operation. ### Claim 11 A system comprising: a means for receiving a set of data; a means for dividing the set of data into a plurality of data portions; a means for transmitting the plurality of data portions to a plurality of processing tiles, wherein each data portion of the plurality of data portions is associated with a processing tile of a plurality of tiles; a means for performing at least one local operation by each processing tile of the plurality of tiles on its respective data portion to form a local maxima; a means for exchanging local maximas between the plurality of processing tiles; a means for calculating a global maximum value based on the local maximas; a means for performing by each processing tile of the plurality of tiles a subtraction operation of the global maximum value from each data input its respective data portion to form a subtraction result; a means for performing by each processing tile of the plurality of tiles an exponential operation on the subtraction result to form exponential results; a means for forming a sum of the exponential results from the plurality of tiles; and a means for inverting the sum to form a scaled value. ### Claim 12 A system, comprising: a plurality of processing tiles within an inference engine, wherein a set of received data is divided into a plurality of data portions, and wherein the plurality of data portions is transmitted to the plurality of processing tiles, wherein each data portion of the plurality of data portions is associated with a processing tile of a plurality of tiles, and wherein each processing tile of the plurality of processing tiles is configured to: perform at least one local operation on its respective data portion to form a local maxima; exchange local maximas with processing tiles within the plurality of processing tiles; calculate a global maximum value based on the local maximas; perform a subtraction operation of the global maximum value from each data input its respective data portion to form a subtraction result; performing an exponential operation on the subtraction result to form exponential results; form a sum of the exponential results from the plurality of tiles; and invert the sum to form a scaled value. ### Claim 13 A system, comprising: a first processing tile and a first on-chip memory (OCM); and a second processing tile and a second OCM; wherein a set of received data is divided into a first data portion and a second data portion, wherein the first data portion is transmitted to the first processing tile and the first OCM and wherein the second data portion is transmitted to the second processing and the second OCM, wherein the first processing tile is configured to perform a local operation on the first data portion to form a first local maxima, and wherein the second processing tile is configured to perform a local operation on the second data portion to form a second local maxima, and wherein the first processing tile and the second processing tile are configured to exchange their respective local maximas with one another and to calculate a global maximum value based on the first and the second local maximas, and wherein the first processing tile and the second processing tile are configured to perform a subtraction operation of the global maximum value from each data input its respective data portion to form a subtraction result, and wherein the first processing tile and the second processing tile are configured to perform an exponential operation on the subtraction result to form exponential results, and wherein the first processing tile and the second processing tile are configured to form a sum of the exponential results from the plurality of tiles; and and wherein the first processing tile and the second processing tile are configured to invert the sum to form a scaled value. ### Claim 14 A computer implemented method, comprising: receiving a set of data; dividing the set of data into a plurality of data portions; transmitting the plurality of data portions to a plurality of processing tiles, wherein each data portion of the plurality of data portions is associated with a processing tile of a plurality of tiles; performing by each processing tile of the plurality of tiles at least one local operation on its respective data portion to form a local maxima; exchanging local maximas between the plurality of processing tiles; calculating a global maximum value based on the local maximas; negating the global maximum value to form a negative global maximum value; performing by each processing tile of the plurality of tiles an addition operation of the negative global maximum value and each data input its respective data portion to form an addition result; performing by each processing tile of the plurality of tiles an exponential operation on the addition result to form exponential results; forming a sum of the exponential results from the plurality of tiles; and inverting the sum to form a scaled value. ### Claim 15 A computer implemented method, comprising: receiving a set of data; dividing the set of data into a plurality of data portions; transmitting the plurality of data portions to a plurality of processing tiles, wherein each data portion of the plurality of data portions is associated with a processing tile of a plurality of tiles; performing by each processing tile of the plurality of tiles at least one local operation on its respective data portion to form a local maxima; exchanging local maximas between the plurality of processing tiles; calculating a global maximum value based on the local maximas; performing by each processing tile of the plurality of tiles a subtraction operation of the global maximum value from each data input its respective data portion to form a subtraction result; performing by each processing tile of the plurality of tiles an exponential operation on the subtraction result to form exponential results; forming a sum of the exponential results from the plurality of tiles; and divide each exponential result of the exponential results by the sum of the exponential results.