# US 12,007,974 B2 · Memory processing optimisation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,007,974 B2 - Document ID: US12007974B2 - Status: Granted - Kind: B2 - Application number: 17/909,174 - Filed: 2021-03-05 - Published: 2023-03-23 - Granted: 2024-06-11 - Earliest claimed: 2020-03-06 - Assignees: Myrtle Software Limited - Applicants: Myrtle Software Limited - Inventors: Graham Hazel - Web page: https://patentgrep.com/patent/US12007974B2 ## CPC - G06F 9/54: Interprogram communication - G06F 16/2282: {Tablespace storage structures; Management thereof} - G06F 3/0611: {in relation to response time} - G06F 3/0659: {Command handling arrangements, e.g. command buffers, queues, command scheduling} - G06F 3/0673: {Single storage device} - G06F 9/5038: {considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration (scheduling strategies G06F9/4881 and subgroups)} - G06F 9/505: {considering the load} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Table lookup from hardware data storage is managed as part of processing one or more computer programs operating on a data processing system. At least one request is received from the one or more computer programs for a table lookup, at a software component that is operating on the data processing system separately or segregated from the one or more computer programs. The software component retrieves the table lookup data corresponding with the at least one request from the hardware data storage. The retrieved table lookup data is returned to the one or more computer programs. ## Claims ### Claim 1 A method for managing embedding table lookup from hardware data storage as part of processing one or more computer programs operating on a data processing system, the method comprising: receiving at least one request from the one or more computer programs for an embedding table lookup by a segregated system software component that centralizes said embedded table lookups and operates on the data processing system separately or segregated from processing by the one or more computer programs, said system software component functions to manage access to said hardware data storage for said embedding table lookups and to schedule lookup operations to said hardware data storage, whereby execution time and latency of said data processing system is reduced by overlapping all non-lookup computations with lookup operations by said segregated system software component; retrieving, by the segregated system software component, the embedding table lookup data corresponding with the at least one request from the hardware data storage, the hardware data storage operative to store an embedding table corresponding to the at least one request; and returning the retrieved embedding table lookup data to the one or more computer programs. ### Claim 2 (depends on 1) The method of claim 1, further comprising: allocating resources of the data processing system to the software component. ### Claim 3 (depends on 2) The method of claim 2, wherein the allocated resources are fixed or semi-static. ### Claim 4 (depends on 2) The method of claim 2, wherein the resources are allocated on the basis of the received at least one request. ### Claim 5 (depends on 2) The method of claim 2, wherein the allocated resources comprise one or more of: a number of processing cores of the data processing system; one or more specific processing cores of the data processing system; one or more threads for execution; and a percentage of a capacity of each processing core of the data processing system. ### Claim 6 (depends on 1) The method of claim 1, further comprising: scheduling a retrieval for each request; and wherein the step of retrieving is performed in accordance with the scheduling. ### Claim 7 (depends on 6) The method of claim 6, wherein the step of scheduling uses first-come-first-served (FCFS) scheduling. ### Claim 8 (depends on 1) The method of claim 1, wherein the data processing system comprises: one or more main processors; and one or more hardware accelerators separate from the one or more main processors, the one or more main processors having associated memory for data storage and the one or more hardware accelerators each comprising processing logic and dedicated memory that is separate from the memory associated with the main processor and wherein the step of retrieving comprises communicating with the processing logic of the one or more hardware accelerators to retrieve the embedding table lookup data corresponding with the at least one request from the dedicated memory. ### Claim 9 (depends on 8) The method of claim 8, wherein the one or more computer programs and/or the software component are operating on the one or more main processors of the data processing system. ### Claim 10 (depends on 8) The method of claim 8, wherein the dedicated memory of each hardware accelerator comprises a plurality of Dynamic Random Access Memory (DRAM) modules, each DRAM module comprising a distinct packaged device; wherein each hardware accelerator further comprises a plurality of memory controllers, each memory controller being configured to address a subset of the plurality of DRAM modules; and wherein the processing logic of each hardware controller is arranged to control the respective plurality of memory controllers so as to read data from more than one of the plurality of DRAM modules in parallel. ### Claim 11 (depends on 1) The method of claim 1, wherein the software component forms part of an operating system or middleware operating on the data processing system. ### Claim 12 (depends on 1) The method of claim 1, wherein the data processing system comprises a plurality of processing cores, the plurality of processing cores contending for common memory resources of the data processing system. ### Claim 13 (depends on 1) The method of claim 1, wherein each request identifies at least one location within a respective embedding table that is stored in the hardware data storage, the embedding table lookup data for each request being based on at least one element of the embedding table corresponding with the at least one location. ### Claim 14 (depends on 13) The method of claim 13, wherein each request identifies a plurality of locations within an embedding table stored in the hardware data storage, the embedding table lookup data for the request being based on a plurality of vectors based on elements of the embedding table corresponding with the plurality of locations. ### Claim 15 (depends on 14) The method of claim 14, wherein the request further identifies a set of one or more weights, the embedding table lookup data for the request being based on a multiplication of each vector of the plurality of vectors by a corresponding weight of the set of one or more weights. ### Claim 16 (depends on 14) The method of claim 14, wherein the embedding table lookup data for the request comprises the result of an arithmetic operation on the plurality of vectors or the plurality of vectors modified by one or more weights. ### Claim 17 (depends on 16) The method of claim 16, wherein the arithmetic operation comprises one of: a summation; an averaging; and an extremum. ### Claim 18 (depends on 1) The method of claim 1, wherein the embedding table indicates indices of non-zero elements in one or more sparse vectors. ### Claim 19 A non-transitory computer readable medium storing a computer program, comprising instructions that, when the program is executed by a computer, cause the computer to carry out a method comprising: receiving at least one request from one or more computer programs for an embedding table lookup by a segregated system software component that centralizes embedded table lookups and operates on a data processing system separately or segregated from processing by the one or more computer programs, said system software component functions to manage access to said hardware data storage for said embedding table lookups and to schedule lookup operations to said hardware data storage whereby execution time and latency of said data processing system is reduced by overlapping all non-lookup computations with lookup operations by said segregated system software component; retrieving, by the segregated system software component, the embedding table lookup data corresponding with the at least one request from hardware data storage, the hardware data storage operative to store an embedding table corresponding with the at least one request; and returning the retrieved embedding table lookup data to the one or more computer programs. ### Claim 20 A data processing system, comprising: hardware data storage operative to store an embedding table therein; at least one processor, in communication with the hardware data storage, and operating one or more computer programs and operating a software component separately or segregated from the one or more computer programs, the software component being configured to: receive at least one request from the one or more computer programs for an embedding table lookup from the stored embedding table by a segregated system software component that centralizes said embedded table lookups and operates on the data processing system separately or segregated from processing by the one or more computer programs, said system software component functions to manage access to said hardware data storage for said embedding table lookups and to schedule lookup operations to said hardware data storage, whereby execution time and latency of said data processing system is reduced by overlapping all non-lookup computations with lookup operations by said segregated system software component; retrieve the embedding table lookup data corresponding with the at least one request from the hardware data storage, the hardware data storage operative to store an embedding table corresponding to the at least one request; and return the retrieved embedding table lookup data to the one or more computer programs. ### Claim 21 (depends on 1) The method of claim 1, wherein the embedding table stored on the hardware data storage comprises at least two gigabytes of data.