# US 2021/0279187 A1 · Management of shared resources in a software-defined storage environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279187 A1
- Document ID: US20210279187A1
- Status: Published application
- Kind: A1
- Application number: 16/807,709
- Filed: 2020-03-03
- Published: 2021-09-09
- Earliest claimed: 2020-03-03
- Assignees: EMC IP Holding Company LLC
- Applicants: EMC IP Holding Company LLC
- Inventors: Avi Puder; Itay Keller; Galina Tcharny; Dvir Koren; Jonathan Sahar; Benjamin Grimberg
- Web page: https://patentgrep.com/patent/US20210279187A1

## CPC

- G06F 3/061: {Improving I/O performance}
- G06F 13/1668: {Details of memory controller}
- G06F 3/0659: {Command handling arrangements, e.g. command buffers, queues, command scheduling}
- G06F 3/067: {Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]}
- G06F 9/5011: {the resources being hardware resources other than CPUs, Servers and Terminals}
- G06F 9/5055: {considering software capabilities, i.e. software resources associated or available to the machine}
- H04L 67/1097: for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A storage control system receives an I/O request from a client for accessing storage resources that are logically divided into device groups, and determines a resource token request value associated with the I/O request and a target device group to which the I/O request is directed. The storage control system determines a number of allowed resource tokens to allocate to the client as a function of (i) the resource token request value, (ii) a sum total of resource tokens requested by other clients for accessing the target device group, and (iii) a total amount of resource tokens currently allocated to the target device group to which the I/O request is directed. The storage control system sends the determined number of allowed resource tokens to the client to thereby enable the client to limit a number of inflight I/O requests that the client issues to the storage control system.

## Claims

### Claim 1

A method, comprising:
receiving, by a storage control system, an input/output (I/O) request from a client application for accessing data storage resources controlled by the storage control system, wherein the data storage resources comprise storage devices which are logically divided by the storage control system into a plurality of device groups;
determining, by the storage control system, a resource token request value associated with the I/O request and a target device group to which the I/O request is directed;
determining, by the storage control system, a number of allowed resource tokens to allocate to the client application, wherein the number of allowed resource tokens is determined as a function of (i) the resource token request value, (ii) a sum total of resource tokens requested by other client applications for accessing the target device group, and (iii) a total amount of resource tokens currently allocated to the target device group to which the I/O request is directed; and
sending to the client application, by the storage control system, the determined number of allowed resource tokens allocated to the client application to thereby enable the client application to limit a number of inflight I/O requests that the client application issues to the storage control system based on the number of allowed resource tokens allocated to the client application for accessing the target device group.

### Claim 2 (depends on 1)

The method of claim 1, wherein the number of allowed resource tokens allocated to the client application comprises a number of allowed I/O tokens, wherein the number of allowed I/O tokens is configured to limit an amount of I/O-related resources that are utilized to process I/O requests of the client application which are directed to the target device group.

### Claim 3 (depends on 1)

The method of claim 1, wherein the number of allowed resource tokens allocated to the client application comprises a number of allowed throughput tokens, wherein the number of allowed throughput tokens is configured to limit an amount of storage-related resources that are utilized to process I/O requests of the client application which are directed to the target device group.

### Claim 4 (depends on 3)

The method of claim 3, wherein one throughput token represents a predefined amount of data.

### Claim 5 (depends on 1)

The method of claim 1, wherein:
the resource token request value associated with the I/O request comprises an I/O token request value, wherein the I/O token request value represents a total amount of current inflight I/O requests and accumulated I/O inflight requests of the client application which have not yet been sent; and
the number of allowed resource tokens allocated to the client application comprises a number of allowed I/O tokens, wherein the number of allowed I/O tokens represents a total number of inflight I/O requests the given client application can concurrently have in a next resource allocation cycle, wherein the total number of inflight requests comprises a total amount of (i) new inflight I/O requests that client application can send in the next resource allocation cycle plus (ii) currently pending inflight I/O requests.

### Claim 6 (depends on 1)

The method of claim 1, wherein:
the resource token request value associated with the I/O request comprises a throughput token request value, wherein the throughput token request value represents a total amount of data associated with all inflight I/O requests of the client application; and
the number of allowed resource tokens allocated to the client application comprises a number of allowed throughput tokens, wherein the number of allowed throughput tokens represents a maximum amount of data that can be associated with all concurrent inflight I/O requests that are allocated to the given client application for a next resource allocation cycle.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
determining, by the storage control system, a current rank value of the client application;
adjusting, by the storage control system, the resource token request value associated with the I/O request of the client application based on the determined current rank value of the client application; and
determining, by the storage control system, the number of allowed resource tokens to allocate to the client application based on the adjusted resource token request value.

### Claim 8 (depends on 7)

The method of claim 7, wherein adjusting the resource token request value received from the client application comprises reducing the resource token request value in proportion to a ratio of (i) the current rank value to (ii) the predefined maximum rank value.

### Claim 9 (depends on 1)

The method of claim 1, wherein the plurality of device groups comprises at least a first device group comprising storage devices of a first device type, and a second device group comprising storage devices of a second device type, which is different from the first device type; and wherein the method further comprises:
determining, by the storage control system, a respective maximum device group token value for each device group of the plurality of device groups, wherein the maximum device group token value for a given device group represents a maximum number of resource tokens that are allocated to the given device group, out of a total amount of available resource tokens of the data storage system, based on a maximum amount of concurrent I/O operations that the given device group can perform based on the type of storage devices of the given device group and a number of storage devices within the given device group.

### Claim 10

An article of manufacture comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code is executable by one or more processors to implement a method comprising:
receiving, by a storage control system, an input/output (I/O) request from a client application for accessing data storage resources controlled by the storage control system, wherein the data storage resources comprise storage devices which are logically divided by the storage control system into a plurality of device groups;
determining, by the storage control system, a resource token request value associated with the I/O request and a target device group to which the I/O request is directed;
determining, by the storage control system, a number of allowed resource tokens to allocate to the client application, wherein the number of allowed resource tokens is determined as a function of (i) the resource token request value, (ii) a sum total of resource tokens requested by other client applications for accessing the target device group, and (iii) a total amount of resource tokens currently allocated to the target device group to which the I/O request is directed; and
sending to the client application, by the storage control system, the determined number of allowed resource tokens allocated to the client application to thereby enable the client application to limit a number of inflight I/O requests that the client application issues to the storage control system based on the number of allowed resource tokens allocated to the client application for accessing the target device group.

### Claim 11 (depends on 10)

The article of manufacture of claim 10, wherein the number of allowed resource tokens allocated to the client application comprises a number of allowed I/O tokens, wherein the number of allowed I/O tokens is configured to limit an amount of I/O-related resources that are utilized to process I/O requests of the client application which are directed to the target device group.

### Claim 12 (depends on 10)

The article of manufacture of claim 10, wherein the number of allowed resource tokens allocated to the client application comprises a number of allowed throughput tokens, wherein the number of allowed throughput tokens is configured to limit an amount of storage-related resources that are utilized to process I/O requests of the client application which are directed to the target device group, wherein one throughput token represents a predefined amount of data.

### Claim 13 (depends on 10)

The article of manufacture of claim 10, wherein:
the resource token request value associated with the I/O request comprises an I/O token request value, wherein the I/O token request value represents a total amount of current inflight I/O requests and accumulated I/O inflight requests of the client application which have not yet been sent; and
the number of allowed resource tokens allocated to the client application comprises a number of allowed I/O tokens, wherein the number of allowed I/O tokens represents a total number of inflight I/O requests the given client application can concurrently have in a next resource allocation cycle, wherein the total number of inflight requests comprises a total amount of (i) new inflight I/O requests that client application can send in the next resource allocation cycle plus (ii) currently pending inflight I/O requests.

### Claim 14 (depends on 10)

The article of manufacture of claim 10, wherein:
the resource token request value associated with the I/O request comprises a throughput token request value, wherein the throughput token request value represents a total amount of data associated with all inflight I/O requests of the client application; and
the number of allowed resource tokens allocated to the client application comprises a number of allowed throughput tokens, wherein the number of allowed throughput tokens represents a maximum amount of data that can be associated with all concurrent inflight I/O requests that are allocated to the given client application for a next resource allocation cycle.

### Claim 15 (depends on 10)

The article of manufacture of claim 10, further comprising:
determining, by the storage control system, a current rank value of the client application;
adjusting, by the storage control system, the resource token request value associated with the I/O request of the client application based on the determined current rank value of the client application, wherein adjusting the resource token request value received from the client application comprises reducing the resource token request value in proportion to a ratio of (i) the current rank value to (ii) the predefined maximum rank value; and
determining, by the storage control system, the number of allowed resource tokens to allocate to the client application based on the adjusted resource token request value.

### Claim 16 (depends on 10)

The article of manufacture of claim 10, wherein the plurality of device groups comprises at least a first device group comprising storage devices of a first device type, and a second device group comprising storage devices of a second device type, which is different from the first device type; and wherein the method further comprises:
determining, by the storage control system, a respective maximum device group token value for each device group of the plurality of device groups, wherein the maximum device group token value for a given device group represents a maximum number of resource tokens that are allocated to the given device group, out of a total amount of available resource tokens of the data storage system, based on a maximum amount of concurrent I/O operations that the given device group can perform based on the type of storage devices of the given device group and a number of storage devices within the given device group.

### Claim 17

A server node, comprising:
at least one processor; and
system memory configured to store program code, wherein the program code is executable by the at least one processor to implement a storage control system which is configured to perform a process comprising:
receiving an input/output (I/O) request from a client application for accessing data storage resources controlled by the storage control system, wherein the data storage resources comprise storage devices which are logically divided by the storage control system into a plurality of device groups;
determining a resource token request value associated with the I/O request and a target device group to which the I/O request is directed;
determining a number of allowed resource tokens to allocate to the client application, wherein the number of allowed resource tokens is determined as a function of (i) the resource token request value, (ii) a sum total of resource tokens requested by other client applications for accessing the target device group, and (iii) a total amount of resource tokens currently allocated to the target device group to which the I/O request is directed; and
sending to the client application, the determined number of allowed resource tokens allocated to the client application to thereby enable the client application to limit a number of inflight I/O requests that the client application issues to the storage control system based on the number of allowed resource tokens allocated to the client application for accessing the target device group.

### Claim 18 (depends on 17)

The server node of claim 17, wherein the number of allowed resource tokens allocated to the client application comprises a number of allowed I/O tokens, wherein the number of allowed I/O tokens is configured to limit an amount of I/O-related resources that are utilized to process I/O requests of the client application which are directed to the target device group.

### Claim 19 (depends on 17)

The server node of claim 17, wherein the number of allowed resource tokens allocated to the client application comprises a number of allowed throughput tokens, wherein the number of allowed throughput tokens is configured to limit an amount of storage-related resources that are utilized to process I/O requests of the client application which are directed to the target device group.

### Claim 20 (depends on 17)

The server node of claim 17, wherein the storage control system comprises a software-defined storage control system.
