# US 2022/0044113 A1 · Asynchronous Neural Network Systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044113 A1
- Document ID: US20220044113A1
- Status: Published application
- Kind: A1
- Application number: 17/178,809
- Filed: 2021-02-18
- Published: 2022-02-10
- Earliest claimed: 2020-08-10
- Assignees: Western Digital Technologies, Inc.
- Applicants: Western Digital Technologies, Inc.
- Inventors: Haoyu Wu; Qian Zhong; Toshiki Hirano
- Web page: https://patentgrep.com/patent/US20220044113A1

## CPC

- G06N 3/08: Learning methods
- G06V 10/82: using neural networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06N 3/048: Activation functions
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A device configured for processing time-series data within an asynchronous neural network may include a processor configured to execute the neural network. The device may further include a multi-step convolution pathway wherein the output of at least one step includes one or more feature maps. Additionally, a multi-step upsampling pathway with steps having corresponding convolution step inputs is included. The device further utilizes feature map data from at least one step of the multi-step convolution process as input data in at least one corresponding step of the upsampling process. The device also includes an inference frequency controller to receive input data and transmit a processing frequency signal to the neural network. The neural network can then generate feature maps at a reduced frequency within the multi-step convolution pathway, and utilize previously processed feature maps as input data within the multi-step upsampling pathway until a subsequent feature map is generated.

## Claims

### Claim 1

A device comprising:
a processor configured to execute a neural network, the neural network being configured to receive a set of time-series data for processing and further comprising:
a multi-step convolution pathway comprising a plurality of steps, wherein the output of at least one step of the plurality of steps comprises one or more feature maps; and
a multi-step upsampling pathway wherein a plurality of steps have a corresponding convolution step input;
wherein, in response to receiving a set of time-series data, feature map data from at least one step of the multi-step convolution pathway is utilized as input data in at least one corresponding step of the multi-step upsampling pathway; and
an inference frequency controller configured to:
receive input data; and
transmit an output signal based on the received input data to the neural network;
wherein the neural network is further configured to, in response to receiving the output signal from the inference frequency controller, generate feature maps at fewer than every step within the multi-step convolution pathway, and utilize previously processed feature maps as input data within at least one step within the multi-step upsampling pathway until a subsequent feature map is generated.

### Claim 2 (depends on 1)

The device of claim 1, wherein the transmitted output signal of the inference frequency controller is generated based on the received input data.

### Claim 3 (depends on 1)

The device of claim 1, wherein the device further comprises a data cache configured to store feature map data.

### Claim 4 (depends on 3)

The device of claim 3, wherein the data cache is further configured to provide the stored feature map data to the neural network for processing as an alternative to generating new feature map data.

### Claim 5 (depends on 4)

The device of claim 4, wherein the neural network is further configured to additionally output generated feature map data to the data cache, the data cache storing the feature map data until requested by the neural network or replaced by subsequently generated feature map data.

### Claim 6

A device, comprising:
a processor configured to execute a neural network, the neural network being configured to process a series of images, and further comprising:
a first multi-step processing pathway; and
a second multi-step processing pathway wherein a plurality of steps within the second multi-step processing pathway comprises at least:
an input from a previous step within the second multi-step processing pathway;
an input from the first multi-step processing pathway; and
an output configured to generate inferences; and
an inference frequency controller configured to modulate the neural network processing in at least one step within the first multi-step processing pathway.

### Claim 7 (depends on 6)

The device of claim 6, wherein the first multi-step processing pathway generates output data that is passed as in input into a corresponding step within the second multi-step processing pathway.

### Claim 8 (depends on 7)

The device of claim 7, wherein each step within the first multi-step processing pathway and the corresponding step from within the second multi-step processing pathway are grouped as a stage.

### Claim 9 (depends on 8)

The device of claim 8, wherein the modulation includes reducing the processing in at least one stage of the neural network.

### Claim 10 (depends on 6)

The device of claim 6, wherein the second multi-step processing pathway is a upsampling pathway.

### Claim 11 (depends on 10)

The device of claim 10, wherein the output of the upsampling pathway comprises a plurality of inferences.

### Claim 12 (depends on 6)

The device of claim 6, wherein the first multi-step processing pathway is a convolution pathway.

### Claim 13 (depends on 12)

The device of claim 12, wherein the output of the convolution pathway is feature map data.

### Claim 14 (depends on 13)

The device of claim 13 wherein the inference frequency controller is further configured to direct the neural network to generate less feature map data per frame by skipping one or more steps within the convolution pathway.

### Claim 15 (depends on 14)

The device of claim 14, wherein, when directed to generate less feature map data, the neural network is further configured to utilize previously generated feature map data associated with a similar step within the convolution pathway.

### Claim 16 (depends on 15)

The device of claim 15, wherein the previously generated feature map data is retrieved from a feature map data cache within the device.

### Claim 17 (depends on 16)

The device of claim 16, wherein the retrieved feature map data is utilized for a number of processes specified by the inference frequency controller.

### Claim 18 (depends on 16)

The device of claim 16, wherein the inference frequency controller is further configured to direct multiple stages within the neural network to operate at different frequencies.

### Claim 19 (depends on 16)

The device of claim 16, wherein the inference frequency controller is further configured to receive computing resources data as input data.

### Claim 20 (depends on 16)

The device of claim 16, wherein the inference frequency controller is further configured to receive environmental variables data as input data.

### Claim 21 (depends on 20)

The device of claim 20, wherein the environmental variables received by the inference frequency controller include local thermal data.

### Claim 22 (depends on 21)

The device of claim 21, wherein the inference frequency controller is further configured to modulate the neural network processing based on received local thermal data exceeding a preconfigured threshold.

### Claim 23

A method, comprising:
configuring a neural network to receive a series of images to generate prediction data;
establishing a multi-step convolution pathway within the neural network;
establishing a multi-step upsampling pathway within the neural network wherein a plurality of upsampling steps comprise an input to receive output data from a corresponding convolution pathway step;
wherein, in response to receiving image for processing, feature map output data is generated at a plurality of steps within the convolution pathway, and at least one step of the upsampling pathway utilizes at least the received feature map data to generate prediction data;
configuring an inference frequency controller to provide an output signal to the neural network; and
configuring the neural network to, in response to receiving the output signal from the inference frequency controller, generate feature map data at fewer than every step within the multi-step convolution pathway and previously processed feature map data is utilized as input data within the multi-step upsampling pathway until a subsequent feature map input is received.

### Claim 24

The method of 23, wherein, based on received time-series input data, the inference frequency controller is further configured to format the output signal to indicate which neural network type from a plurality of neural network types will be suitable for processing subsequent input data within the time-series.

### Claim 25

A method comprising:
configuring an inference frequency controller to receive input data from a plurality of inputs;
processing the received input data;
determining a processing frequency for a neural network configured to process time-series data; and
transmitting a signal associated with the determined frequency to the neural network;
wherein the signal is configured to change the frequency of processing time-series data within the neural network.
