# US 11,030,131 B2 · Data processing performance enhancement for neural networks using a virtualized data iterator

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,030,131 B2
- Document ID: US11030131B2
- Status: Granted
- Kind: B2
- Application number: 16/942,728
- Filed: 2020-07-29
- Published: 2020-11-12
- Granted: 2021-06-08
- Earliest claimed: 2017-04-17
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Chad Balling McBride; George Petre; Amol Ashok Ambardekar; Kent D. Cedola; Larry Marvin Wall; Boris Bobrov
- Web page: https://patentgrep.com/patent/US11030131B2

## CPC

- H03M 7/3059: {Digital compression and data reduction techniques where the original information is represented by a subset or similar information, e.g. lossy compression}
- G06F 1/3206: Monitoring of events, devices or parameters that trigger a change in power modality
- G06F 1/324: by lowering clock frequency
- G06F 1/3275: {Power saving in memory, e.g. RAM, cache}
- G06F 1/3287: by switching off individual functional units in the computer system
- G06F 3/0604: {Improving or facilitating administration, e.g. storage management}
- G06F 3/0631: {by allocating resources to storage systems}
- G06F 3/067: {Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]}
- G06F 9/30087: {Synchronisation or serialisation instructions}
- G06F 9/3836: {Instruction issuing, e.g. dynamic instruction scheduling or out of order instruction execution}
- G06F 9/3858: {Result writeback, i.e. updating the architectural state or memory}
- G06F 9/3887: {controlled by a single instruction for multiple data lanes [SIMD]}
- G06F 9/46: Multiprogramming arrangements
- G06F 9/4881: {Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues}
- G06F 9/5016: {the resource being the memory}
- G06F 9/5033: {considering data affinity}
- G06F 9/5061: {Partitioning or combining of resources}
- G06F 9/5077: {Logical partitioning of resources; Management or configuration of virtualized resources (specific details on emulation or internal functioning of virtual machines G06F9/455)}
- G06F 12/0207: {with multidimensional access, e.g. row/column, matrix}
- G06F 12/0238: {Memory management in non-volatile memory, e.g. resistive RAM or ferroelectric memory}
- G06F 12/08: in hierarchically structured memory systems, e.g. virtual memory systems
- G06F 12/0862: with prefetch
- G06F 12/10: Address translation
- G06F 13/1673: {using buffers}
- G06F 13/1689: {Synchronisation and timing concerns (synchronisation on a memory bus G06F13/4234)}
- G06F 13/28: using burst mode transfer, e.g. direct memory access {DMA}, cycle steal (G06F13/32 takes precedence)
- G06F 15/8007: {single instruction multiple data [SIMD] multiprocessors}
- G06F 17/15: Correlation function computation {including computation of convolution operations (arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00)}
- G06N 3/045: Combinations of networks
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/06: Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N 3/063: using electronic means
- G06N 3/065: Analogue means
- G06N 3/10: Interfaces, programming languages or software development kits, e.g. for simulating neural networks
- H03M 7/3066: {by means of a mask or a bit-map}
- H03M 7/6005: {Decoder aspects}
- H03M 7/6011: {Encoder aspects}
- H03M 7/6058: {Saving memory space in the encoder or decoder}
- H03M 7/70: {Type of the data to be coded, other than image and sound}
- H04L 45/04: {Interdomain routing, e.g. hierarchical routing}
- H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L 67/1001: for accessing one among a plurality of replicated servers
- G06F 9/467: {Transactional memory (G06F9/528 takes precedence)}
- G06F 2209/484: Precedence
- G06F 2209/485: Resource constraint
- G06F 2212/1016: Performance improvement
- G06F 2212/6026: Prefetching based on access pattern detection, e.g. stride based prefetch
- G06F 2212/657: Virtual address space management
- H03M 7/46: Conversion to or from run-length codes, i.e. by representing the number of consecutive digits, or groups of digits, of the same kind by a code word and a digit indicative of that kind
- H04L 45/50: using label swapping, e.g. multi-protocol label switch [MPLS]
- Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management
- Y02D 30/50: in wire-line communication networks, e.g. low power modes or reduced link rate
- H03M: CODING; DECODING; CODE CONVERSION IN GENERAL (using fluidic means F15C4/00; optical analogue/digital converters G02F7/00; coding, decoding or code conversion, specially adapted for particular applications, see the relevant subclasses, e.g. G01D, G01R, G06F, G06T, G09G, G10L, G11B, G11C, H04B, H04L, H04M, H04N; ciphering or deciphering for cryptography or other purposes involving the need for secrecy G09C)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE

## Abstract

The performance of a neural network (NN) and/or deep neural network (DNN) can limited by the number of operations being performed as well as management of data among the various memory components of the NN/DNN. Using virtualized hardware iterators, data for processing by the NN/DNN can be traversed and configured to optimize the number of operations as well as memory utilization to enhance the overall performance of a NN/DNN. Operatively, an iterator controller can generate instructions for execution by the NN/DNN representative of one more desired iterator operation types and to perform one or more iterator operations. Data can be iterated according to a selected iterator operation and communicated to one or more neuron processors of the NN/DD for processing and output to a destination memory. The iterator operations can be applied to various volumes of data (e.g., blobs) in parallel or multiple slices of the same volume.

## Claims

### Claim 1

A system comprising:
at least one processor;
at least one hardware iterator configured to execute multiple instances for parallel processing of one or more data iteration functions; and
at least one memory in communication with the at least one processor, the at least one memory having computer-readable instructions stored thereupon that, when executed by the at least one processor, cause the system to:
receive one or more initialization parameters comprising data representative of dimensions of data to be processed by a neural network environment;
load the data from a memory component of the neural network environment;
receive one or more instructions to traverse the loaded data according to one or more iterator operation types and the one or more initialization parameters and using one or more selected traversal patterns;
process the one or more instructions to select one or more portions of the loaded data according to one or more iterator operation types and the one or more initialization parameters; and
communicate the one or more portions of the loaded data to one or more processing components of the neural network environment;
wherein the hardware iterator is virtualized using logical mapping to the data in the memory component, the logical mapping determined using parameters indicative of the data's physical memory addressing, dimension, and volume.

### Claim 2 (depends on 1)

The system of claim 1, wherein the one or more portions of the loaded data are equal portions.

### Claim 3 (depends on 1)

The system of claim 1, wherein the computer-readable instructions further cause the system to traverse the data according to a select data volume defined by the received one or more initialization parameters, the parameters comprising data comprising data height, data width, number of channels, number of kernels, and one or more other data descriptor.

### Claim 4 (depends on 3)

The system of claim 3, wherein the computer-readable instructions further cause the system to traverse the data utilizing one or more sliding windows, the sliding windows operative to select one or more data elements of the data volume as the one or more portions communicated to the one or more processing components.

### Claim 5 (depends on 4)

The system of claim 4, wherein the computer-readable instructions further cause the system to traverse the loaded data using one or more sliding windows that straddle a data dimensional boundary of the loaded data.

### Claim 6 (depends on 1)

The system of claim 1, wherein the computer-readable instructions further cause the system to insert one or more data paddings to the loaded data.

### Claim 7 (depends on 1)

The system of claim 1, wherein the computer-readable instructions further cause the system to generate an output volume of data representative of data processed by one or more neural processors of the neural network environment, the output volume generated according to one or more instructions received by a controller component of the neural network environment.

### Claim 8

A computer-implemented method for enhanced data processing using an iterator in a computing device, the iterator configured to execute multiple instances for parallel processing of one or more data iteration functions, the method comprising:
loading data from a memory component of the computing device;
receiving one or more initialization parameters indicative of dimensions of the data as a data volume comprising any of height, width, number of channels, number of kernels, or one or more other data descriptors;
receiving one or more instructions to traverse the loaded data according to one or more iterator operation types and the one or more initialization parameters and using one or more selected traversal patterns;
processing the one or more instructions to select one or more portions of the loaded data according to one or more iterator operation types and the one or more initialization parameters; and
communicating the one or more portions of the loaded data to one or more processing components of the computing device;
wherein the iterator is virtualized using logical mapping to the data in the memory component, the logical mapping determined using parameters indicative of the data's physical memory addressing, dimension, and volume.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein the one or more portions of the loaded data are equal portions.

### Claim 10 (depends on 8)

The computer-implemented method of claim 8, wherein the traversing of the loading data utilizes one or more sliding windows operative to straddle a data dimensional boundary of the data.

### Claim 11 (depends on 8)

The computer-implemented method of claim 8, further comprising:
inserting a padding sub-volume into the loaded data.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11, wherein the padding sub-volume is defined by the received one or more instructions and by the received one or more initialization parameters.

### Claim 13 (depends on 8)

The computer-implemented method of claim 8, further comprising:
generating an output data volume for data processed by the one or more processing components, the output data volume defined by the received one or more instructions.

### Claim 14 (depends on 13)

The computer-implemented method of claim 13, further comprising:
discarding one or more portions of the generated output data volume.

### Claim 15 (depends on 8)

The computer-implemented method of claim 8, further comprising storing a traverse position of the loaded data as internal state data in a memory component of the computing device.

### Claim 16

A computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by one or more processors of a computing device, cause the computing device to:
instantiate an iterator configured to execute multiple instances for parallel processing of one or more data iteration functions;
receive one or more initialization parameters comprising data representative of dimensions of data to be processed, the dimensions of the data comprising data representative of a volume of the data, the one or more initialization parameters comprising a physical memory addressing of the data to be processed;
load the data utilizing the one or more initialization parameters;
receive one or more instructions to traverse the loaded data according to one or more iterator operation types and the one or more initialization parameters and using one or more selected traversal patterns;
process the one or more instructions to select one or more portions of the loaded data according to the one or more iterator operation types and the one or more initialization parameters; and
communicate the one or more portions of the loaded data to one or more processing components;
wherein the iterator is virtualized using logical mapping to the data, the logical mapping determined using parameters indicative of the data's physical memory addressing, dimension, and volume.

### Claim 17 (depends on 16)

The computer-readable storage medium of claim 16, wherein the one or more instructions comprise instructions to store data representative of an internal state of the traversing of the loaded data, wherein the computer-executable instructions further cause the computing device to:
store the internal state data in a memory component of the computing device.

### Claim 18 (depends on 17)

The computer-readable storage medium of claim 17, wherein the computer-executable instructions further cause the computing device to:
insert an additional data volume to the loaded data.

### Claim 19 (depends on 16)

The computer-readable storage medium of claim 16, wherein the computer-executable instructions further cause the computing device to:
generate an output data volume representative of data processed by the one or more processing components.

### Claim 20 (depends on 16)

The computer-readable storage medium of claim 16, wherein the computer-executable instructions further cause the computing device to:
traverse the loaded data utilizing a logical data mapping of the loaded data, the traversing of the loaded data comprising applying one or more sliding windows to the logical data mapping to associate a portion of the loaded data to one or more physical memory addresses.