# US 2020/0320035 A1 · Temporal difference learning, reinforcement learning approach to determine optimal number of threads to use for file copying

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0320035 A1
- Document ID: US20200320035A1
- Status: Published application
- Kind: A1
- Application number: 16/837,209
- Filed: 2020-04-01
- Published: 2020-10-08
- Earliest claimed: 2019-04-02
- Assignees: Micro Focus Software Inc.
- Applicants: Micro Focus Software Inc.
- Inventors: ARSHAD JAVEED
- Web page: https://patentgrep.com/patent/US20200320035A1

## CPC

- G06F 9/543: {User-generated data transfer, e.g. clipboards, dynamic data exchange [DDE], object linking and embedding [OLE]}
- G06F 16/119: {Details of migration of file systems (migration mechanisms in storage systems G06F3/0647)}
- G06F 9/4843: {by program, e.g. task dispatcher, supervisor, operating system}
- G06F 16/11: File system administration, e.g. details of archiving or snapshots (error detection or correction of the data by redundancy in operations G06F11/14)
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

For a given file type, an optimal number of threads to use to copy files of each of a number of different discrete file sizes is determined, using a temporal difference learning, reinforcement learning approach in which file copy time is used as feedback reward reinforcement. A continuous function corresponding to the given file type and outputting the number of threads to use to copy files having this given file type and that are of any input file size is fitted onto the optimal numbers of threads determined for the discrete file sizes.

## Claims

### Claim 1

A method comprising:
for each discrete file size of a plurality of discrete file sizes, determining an optimal number of threads to use to copy files of the discrete file size and having a given file type, using a temporal difference learning, reinforcement learning approach in which file copy time is used as feedback reward reinforcement;
fitting a continuous function corresponding to the given file type, the continuous function outputting the number of threads to use to copy files of any input file size and that have the given file type onto the optimal numbers of threads determined for the discrete file sizes; and
copying particular files having the given file type using a number of threads output by the continuous function for a file size of the particular files.

### Claim 2 (depends on 1)

The method of claim 1, wherein a plurality of states corresponds to a plurality of different numbers of threads to use to copy the files,
wherein the optimal number of threads is determined by, for each discrete file size, performing a plurality of training iterations,
wherein in each training iteration, an action is taken to transition from a current state to a next state, and the files of the discrete file size copied using the number of threads of the next state,
and wherein the action is selected from incrementing the number of threads of the current state, decrementing the number of threads of the current state, or maintaining the number of threads of the current state.

### Claim 3 (depends on 2)

The method of claim 2, wherein the temporal difference learning, reinforcement learning machine approach is a Q-learning approach,
wherein for each discrete file size, a Q-learning table is maintained of cumulative values for a plurality of state-action pairs,
wherein in each training iteration, the file copy time to copy the files of the discrete file size using the number of threads of the next state transitioned to from the current state by taking the selected action is monitored,
and wherein after each training iteration the cumulative value for the state-action pair corresponding to the current state and the selected action is updated.

### Claim 4 (depends on 3)

The method of claim 3, wherein in each training iteration, the action to transition from the current state to the next state is probabilistically selected based on the state of the state-action pair having a highest cumulative value of any state-action pair including the current state.

### Claim 5 (depends on 3)

The method of claim 3, wherein after each training iteration, the cumulative value for the state-action pair corresponding to the current state and the selected action are updated based on a reward value taking into account the monitored file copy time to copy the files of the discrete file size.

### Claim 6 (depends on 2)

The method of claim 2, wherein for each discrete file size, after each training iteration, another training iteration is performed responsive to determining that a stable state has not yet been reached,
wherein responsive to reaching the stable state, the optimal number of threads for each discrete file size is set to the number of threads of the stable state.

### Claim 7 (depends on 1)

The method of claim 1, wherein the continuous function is a polynomial function.

### Claim 8 (depends on 1)

The method of claim 1, wherein the optimal number of threads is determined and the continuous function is fitted at a training system, and the particular files are copied using the optimal number of threads output by the continuous function corresponding to the given file type for the file size of the particular files at a production system

### Claim 9 (depends on 1)

The method of claim 1, wherein the particular file copied is one of a plurality of particular files of varying file sizes that are copied, and as the particular files are each copied using the number of threads output by the continuous function, storage device usage variance and mean are monitored.

### Claim 10 (depends on 9)

The method of claim 9, wherein a set of data points corresponds to the given file type and comprises the optimal numbers of threads determined for the discrete file sizes,
wherein periodically a plurality of additional files of the given file type is copied using a selected number of threads different than the number of threads output by the continuous function for a file size of the additional files,
wherein responsive to the storage device usage variance decreasing after copying the additional files using the selected number of threads different than the number of threads output by the continuous function corresponding to the given file type, the selected number of threads and the file size of the given files are added as a new data point to the set of data points corresponding to the given file type.

### Claim 11 (depends on 10)

The method of claim 10, wherein the new data point is one of a plurality of new data points added to the set of data points,
and wherein periodically the continuous function corresponding to the given file type is updated by refitting the continuous function onto the set of data points including the new data points.

### Claim 12 (depends on 1)

The method of claim 1, wherein, for each discrete file size, the optimal number of threads to use to copy files having the given file type and of the discrete file size is determined using just the file copy time and the discrete file size as input parameters to the temporal different learning, reinforcement learning approach.

### Claim 13

A non-transitory computer-readable data storage medium storing program code executable by a processor to perform processing comprising:
receiving a continuous function corresponding to a file type, the continuous function outputting the number of threads to use to copy files having the file type and of any input file size, the continuous function fitted onto optimal numbers of threads used to copy files having the file type and of discrete file sizes; and
copying a plurality of particular files having the file type using a number of threads output by the continuous function for a file size of the particular files,
wherein the optimal numbers of threads for the discrete file sizes correspond to the file type and have been determined using a temporal difference learning, reinforcement learning approach in which file copy time is used as feedback reward reinforcement.

### Claim 14 (depends on 13)

The non-transitory computer-readable data storage medium of claim 13, wherein the optimal numbers of threads for the discrete file sizes, corresponding to the file type, and onto which the continuous function corresponding to the file type has been fitted are received as a set of data points for the file type.

### Claim 15 (depends on 14)

The non-transitory computer-readable data storage medium of claim 14, wherein the processing further comprising:
as the particular files are copied, monitoring storage device usage mean and variance.

### Claim 16 (depends on 15)

The non-transitory computer-readable data storage medium of claim 15, wherein the processing further comprises:
periodically copying a plurality of given files using a selected number of threads different than the number of threads output by the continuous function for a file size of the given files, the given files having the file type to which the continuous function corresponds; and
responsive to the storage device usage variance decreasing after copying the given files using the selected number of threads different than the number of threads output by the continuous function, adding the selected number of threads for the file size of the given files as a new data point to the set of data points for the file type.

### Claim 17 (depends on 16)

The non-transitory computing-readable data storage medium of claim 16, wherein the new data point is one of a plurality of new data points added to the set of data points for the file type,
and wherein each new data point comprises a number of threads for a file size different than the number of threads output by the continuous function for the file size and that resulted in a decrease in the storage device usage variance after copying files of the file size and having the file type, the processing further comprising:
periodically updating the continuous function corresponding to the file type by fitting the continuous function onto the set of data points for the file type including the new data points.

### Claim 18 (depends on 14)

The non-transitory computer-readable data storage medium of claim 14, wherein receiving the continuous function comprises generating the continuous function by fitting the continuous function onto the received optimal numbers of threads for the discrete sizes.

### Claim 19 (depends on 13)

The non-transitory computer-readable data storage medium of claim 13, wherein the processor is part of a production computing system, and wherein the optimal numbers of threads for the discrete file sizes were determined at a training computing system different than the production computing system.

### Claim 20

A computing system comprising:
a memory storing a Q-learning table of cumulative values for a plurality of state-action pairs, each state-action pair comprising one of a plurality of states and one of a plurality of actions, the states corresponding to different numbers of threads, the actions comprising incrementing the number of threads, the Q-learning table corresponding to a given file type, decrementing the number of threads, and maintaining the number of threads; and
a processor to:
for each of a plurality of discrete file sizes, determine an optimal number of threads to use to copy files of the discrete file size, the files having the given file type, by iteratively performing, until a stable state has been reached:
probabilistically selecting an action to transition from a current state to a next state using the Q-learning table, the action probabilistically selected based on the action of the state-action pair including the current state and that has a highest cumulative value within the table;
copying the files of the discrete file size using the number of threads of the next state and monitoring file transfer times to copy the files;
updating, within the Q-learning table, the cumulative value for the state-action pair corresponding to the current state and the selected action based on a reward value taking into account the monitored file transfer times;
for each discrete file size, set the optimal number of threads for the discrete file size to the number of threads corresponding to the stable state that has been reached for the discrete file size; and
fit a continuous function corresponding to the given file type and outputting the number of threads to use to copy files having the given file type and of any input file size onto the optimal numbers of threads determined for the discrete file sizes.
