# US 2021/0279178 A1 · Bandwidth efficient access to persistent storage in a distributed storage system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0279178 A1 - Document ID: US20210279178A1 - Status: Published application - Kind: A1 - Application number: 16/813,012 - Filed: 2020-03-09 - Published: 2021-09-09 - Earliest claimed: 2020-03-09 - Assignees: EMC IP Holding Company LLC - Applicants: EMC IP Holding Company LLC - Inventors: Alex Soukhman; Lior Kamran - Web page: https://patentgrep.com/patent/US20210279178A1 ## CPC - G06F 12/0868: Data transfer between cache memory and other subsystems, e.g. storage devices or host systems - G06F 3/061: {Improving I/O performance} - G06F 3/0635: {by changing the path, e.g. traffic rerouting, path reconfiguration} - G06F 3/064: {Management of blocks} - G06F 3/067: {Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]} - G06F 13/4027: {using bus bridges (G06F13/4022 takes precedence)} - G06F 15/17331: {Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]} - G06F 16/2471: {Distributed queries} - H04L 9/0643: {Hash functions, e.g. MD5, SHA, HMAC or f9 MAC} - H04L 9/0894: {Escrow, recovery or storing of secret information, e.g. secret key escrow or cryptographic key storage} - H04L 9/3236: {using cryptographic hash functions} - H04L 67/1097: for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] - G06F 2212/284: being distributed - G06F 2212/608: Details relating to cache mapping - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract An apparatus comprises at least one processing device configured, in conjunction with processing of an input-output (IO) operation by a first storage node of a distributed storage system, where processing of the IO operation requires access to at least one persistent storage device remote from the first storage node but local to a second storage node of the distributed storage system, to send a request from the first storage node to the second storage node identifying data associated with the operation. The processing device determines a buffer of the second storage node that is allocated for use by the second storage node for temporary storage of the data, and utilizes the buffer to provide the data to the second storage node to allow the second storage node to write the data, and/or to obtain the data from the second storage node after the second storage node has read the data. ## Claims ### Claim 1 An apparatus comprising: at least one processing device comprising a processor coupled to a memory; said at least one processing device being configured: in conjunction with processing of an input-output operation by a first storage node of a distributed storage system where processing of the input-output operation requires access to at least one persistent storage device remote from the first storage node but local to a second storage node of the distributed storage system: to send a request from the first storage node to the second storage node identifying data associated with the operation; to determine a buffer of the second storage node that is allocated for use by the second storage node for temporary storage of the data; and to utilize the buffer for at least one of (i) providing the data to the second storage node to allow the second storage node to write the data to the persistent storage device, and (ii) obtaining the data from the second storage node after the second storage node has read the data from the persistent storage device. ### Claim 2 (depends on 1) The apparatus of claim 1 wherein said at least one processing device comprises at least a portion of a distributed storage controller of the distributed storage system. ### Claim 3 (depends on 1) The apparatus of claim 1 wherein the storage nodes of the distributed storage system are interconnected in a mesh network. ### Claim 4 (depends on 1) The apparatus of claim 1 wherein each of at least a subset of the storage nodes of the distributed storage system comprises a set of processing modules configured to communicate with corresponding sets of processing modules on other ones of the storage nodes, a given such set of processing modules comprising at least a routing module, a control module and a data module, the sets of processing modules of the storage nodes of the distributed storage system collectively comprising at least a portion of a distributed storage controller of the distributed storage system. ### Claim 5 (depends on 1) The apparatus of claim 1 wherein sending a request from the first storage node to the second storage node identifying data associated with the operation comprises sending a write request identifying write data to be written by the second storage node to the persistent storage device local to the second storage node. ### Claim 6 (depends on 5) The apparatus of claim 5 wherein the first storage node writes the write data into the buffer using a remote direct memory access connection between the first storage node and the second storage node. ### Claim 7 (depends on 1) The apparatus of claim 1 wherein sending a request from the first storage node to the second storage node identifying data associated with the operation comprises sending a read request identifying read data to be read by the second storage node from the persistent storage device local to the second storage node. ### Claim 8 (depends on 7) The apparatus of claim 7 wherein the first storage node reads the read data from the buffer using a remote direct memory access connection between the first storage node and the second storage node. ### Claim 9 (depends on 1) The apparatus of claim 1 wherein the second storage node writes the data to or reads the data from the persistent storage device using a local network connection that has a substantially higher bandwidth than that of a remote direct memory access connection between the first storage node and the persistent storage device. ### Claim 10 (depends on 1) The apparatus of claim 1 wherein each of a plurality of persistent storage devices of the distributed storage system is accessible to a corresponding one of the storage nodes via a local interface and is also accessible to one or more other ones of the storage nodes via a remote direct memory access interface. ### Claim 11 (depends on 1) The apparatus of claim 1 wherein sending the request, determining the buffer and utilizing the buffer are performed for each of a plurality of different operations of a particular type responsive to those operations each having an associated data size above a specified threshold. ### Claim 12 (depends on 11) The apparatus of claim 11 wherein the operations of the particular type comprise metadata update journal operations. ### Claim 13 (depends on 1) The apparatus of claim 1 wherein the buffer is dynamically allocated for use by the second storage node for temporary storage of the data responsive to receipt of the request from the first storage node, and further wherein the second storage node identifies the dynamically allocated buffer to the first storage node in a response to the request so as to thereby allow the first storage node to access the buffer. ### Claim 14 (depends on 1) The apparatus of claim 1 wherein the buffer is pre-allocated for use by the second storage node, and further wherein the second storage node identifies the pre-allocated buffer to the first storage node prior to receipt of the request from the first storage node. ### Claim 15 A method comprising: in conjunction with processing of an input-output operation by a first storage node of a distributed storage system where processing of the input-output operation requires access to at least one persistent storage device remote from the first storage node but local to a second storage node of the distributed storage system: sending a request from the first storage node to the second storage node identifying data associated with the operation; determining a buffer of the second storage node that is allocated for use by the second storage node for temporary storage of the data; and utilizing the buffer for at least one of (i) providing the data to the second storage node to allow the second storage node to write the data to the persistent storage device, and (ii) obtaining the data from the second storage node after the second storage node has read the data from the persistent storage device; wherein the method is performed by at least one processing device comprising a processor coupled to a memory. ### Claim 16 (depends on 15) The method of claim 15 wherein sending a request from the first storage node to the second storage node identifying data associated with the operation comprises sending a write request identifying write data to be written by the second storage node to the persistent storage device local to the second storage node, and further wherein the first storage node writes the write data into the buffer using a remote direct memory access connection between the first storage node and the second storage node. ### Claim 17 (depends on 15) The method of claim 15 wherein sending a request from the first storage node to the second storage node identifying data associated with the operation comprises sending a read request identifying read data to be read by the second storage node from the persistent storage device local to the second storage node, and further wherein the first storage node reads the read data from the buffer using a remote direct memory access connection between the first storage node and the second storage node. ### Claim 18 A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device: in conjunction with processing of an input-output operation by a first storage node of a distributed storage system where processing of the input-output operation requires access to at least one persistent storage device remote from the first storage node but local to a second storage node of the distributed storage system: to send a request from the first storage node to the second storage node identifying data associated with the operation; to determine a buffer of the second storage node that is allocated for use by the second storage node for temporary storage of the data; and to utilize the buffer for at least one of (i) providing the data to the second storage node to allow the second storage node to write the data to the persistent storage device, and (ii) obtaining the data from the second storage node after the second storage node has read the data from the persistent storage device. ### Claim 19 (depends on 18) The computer program product of claim 18 wherein sending a request from the first storage node to the second storage node identifying data associated with the operation comprises sending a write request identifying write data to be written by the second storage node to the persistent storage device local to the second storage node, and further wherein the first storage node writes the write data into the buffer using a remote direct memory access connection between the first storage node and the second storage node. ### Claim 20 (depends on 18) The computer program product of claim 18 wherein sending a request from the first storage node to the second storage node identifying data associated with the operation comprises sending a read request identifying read data to be read by the second storage node from the persistent storage device local to the second storage node, and further wherein the first storage node reads the read data from the buffer using a remote direct memory access connection between the first storage node and the second storage node.