Systems and methods for generating code for parallel processing units

A method comprising:

• for a source program including a format for sequential execution,
• generating code, by one or more processors, for executing the source program in a data processing environment that includes a parallel processing device coupled to a sequential processing host, the generating including:
• generating one or more in-memory intermediate representations (IRs) for the source program;
• partitioning, by the one or more processors, the one or more in-memory IRs for the source program into serial code segments and parallel code segments identified as suitable for parallel execution;
• determining, by the one or more processors, a number of thread blocks and a number of threads per thread block for executing one or more of the parallel code segments, the determining based on an analysis of the one or more of the parallel code segments;
• and
• converting, by the one or more processors, the one or more of the parallel code segments into one or more kernels, where the converting includes:
• applying one or more optimizations to the one or more in-memory IRs, where the one or more optimizations includes:
• detecting a first serial code segment of the serial code segments as being disposed between a first parallel code segment of the parallel code segments and a second parallel code segment of the parallel code segments;
• converting the first serial code segment of the serial code segments into a new kernel where the new kernel includes a single thread in a single thread block;
• and
• adding a kernel launch directive that includes the number of thread blocks and the number of threads per thread block, wherein the kernel launch directive conforms to a predefined heterogeneous programming model.

A method comprising:

• for a source program conforming to a programming language and including a format for sequential execution,
• providing one or more data structures that map sequential function calls to parallel function calls that are predefined for execution on a parallel processing device and implemented through an extension to the programming language of the source program or through a different language than the programming language of the source program, wherein the sequential function calls perform at least one of array-based operations or matrix-based operations;
• and
• generating code, by one or more processors, for executing the source program in a data processing environment that includes a parallel processing device coupled to a sequential processing host, the generating including:
• detecting an occurrence of a given sequential function call of the sequential function calls in the source program;
• identifying a predefined parallel function call implementing equivalent functionality as the given sequential function call detected in the source program, where the identifying includes performing a look up on the one or more data structures;
• and
• including the predefined parallel function call identified as implementing equivalent functionality in the code generated for the source program, where the predefined parallel function call replaces the occurrence of the given sequential function call wherein the predefined parallel function call that replaces the occurrence of the given sequential function call is selected from at least one of:
• a first library of Fast Fourier Transform (FFT) functions optimized for the parallel processing device;
• a second library of Deep Neural Network (DNN) functions optimized for the parallel processing device;
• a third library of Basic Linear Algebra Subprograms (BLAS) functions optimized for the parallel processing device;
• or
• a fourth library of dense and sparse direct solvers for use with Computer Vision, Computational Fluid Dynamics, Computational Chemistry, and Linear Optimization (SOLVER) functions optimized for the parallel processing device.

One or more non-transitory computer-readable media having stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations comprising:

• for a source program including a format for sequential execution,
• generating code for executing the source program in a data processing environment that includes a parallel processing device coupled to a sequential processing host, the generating including:
• generating one or more in-memory intermediate representations (IRs) for the source program;
• identifying a nested loop structure of for-loops in the one or more in-memory IRs, wherein the for-loops included in the nested loop structure include loop bounds;
• applying a parallel loop analysis check to the for-loops included in the nested loop structure to identify one or more sets of the for-loops suitable for parallel execution by the parallel processing device;
• determining a number of thread blocks and a number of threads per thread block for parallel execution of the one or more sets of the for-loops by the parallel processing device, wherein the number of thread blocks and the number of threads per thread block are determined based on one or more of the loop bounds;
• and
• converting the one or more sets of the for-loops into a kernel for parallel execution by the parallel processing device, wherein the converting includes:
• adding a kernel launch directive that includes the number of thread blocks and the number of threads per thread block,
• wherein the kernel launch directive conform to a predefined heterogeneous programming model.

An apparatus comprising:

• a memory storing a source program including a format for sequential execution;
• and
• one or more processors coupled to the memory, the one or more processors configured to:
• generate code for executing the source program in a data processing environment that includes a parallel processing device coupled to a sequential processing host, the generating including:
• generate one or more in-memory intermediate representations (IRs) for the source program;
• identify a nested loop structure of for-loops in the one or more in-memory IRs, wherein the for-loops included in the nested structure include loop bounds;
• apply a parallel loop analysis check to the for-loops included in the nested loop structure to identify one or more sets of the for-loops suitable for parallel execution by the parallel processing device;
• determine a number of thread blocks and a number of threads per thread block for parallel execution of the one or more sets of the for-loops by the parallel processing device, wherein the number of thread blocks and the number of threads per thread block are determined based on one or more of the loop bounds;
• and
• convert the one or more sets of the for-loops into a kernel for parallel execution by the parallel processing device, wherein the convert includes:
• adding a kernel launch directive that includes the number of thread blocks and the number of threads per thread block,
• wherein the kernel launch directive conform to a predefined heterogeneous programming model.

One or more non-transitory computer-readable media having stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations comprising:

• for a source program including a format for sequential execution,
• generating code for executing the source program in a data processing environment that includes a parallel processing device coupled to a sequential processing host, the generating including:
• generating one or more in-memory intermediate representations (IRs) for the source program;
• partitioning the one or more in-memory IRs for the source program into serial code segments and parallel code segments identified as suitable for parallel execution;
• determining a number of thread blocks and a number of threads per thread block for executing one or more of the parallel code segments, the determining based on an analysis of the one or more of the parallel code segments;
• and
• converting the one or more of the parallel code segments into one or more kernels, where the converting includes:
• applying one or more optimizations to the one or more in-memory IRs, where the one or more optimizations includes:
• detecting a first serial code segment of the serial code segments as being disposed between a first parallel code segment of the parallel code segments and a second parallel code segment of the parallel code segments;
• converting the first serial code segment of the serial code segments into a new kernel where the new kernel includes a single thread in a single thread block;
• and
• adding a kernel launch directive that includes the number of thread blocks and the number of threads per thread block,
• wherein the kernel launch directive conforms to a predefined heterogeneous programming model.

An apparatus comprising:

• a memory storing a source program including a format for sequential execution;
• and
• one or more processors configured to:
• generate code for executing the source program in a data processing environment that includes a parallel processing device coupled to a sequential processing host, the generating including:
• generate one or more in-memory intermediate representations (IRs) for the source program;
• partition the one or more in-memory IRs for the source program into serial code segments and parallel code segments identified as suitable for parallel execution;
• determine a number of thread blocks and a number of threads per thread block for executing one or more of the parallel code segments, the determining based on an analysis of the one or more of the parallel code segments;
• and
• convert the one or more of the parallel code segments into one or more kernels, where the converting includes:
• applying one or more optimizations to the one or more in-memory IRs, where the one or more optimizations includes:
• detecting a first serial code segment of the serial code segments as being disposed between a first parallel code segment of the parallel code segments and a second parallel code segment of the parallel code segments;
• converting the first serial code segment of the serial code segments into a new kernel where the new kernel includes a single thread in a single thread block;
• and
• adding a kernel launch directive that includes the number of thread blocks and the number of threads per thread block,
• wherein the kernel launch directive conforms to a predefined heterogeneous programming model.

One or more non-transitory computer-readable media having stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations comprising:

• for a source program conforming to a programming language and including a format for sequential execution,
• providing one or more data structures that map sequential function calls to parallel function calls that are predefined for execution on a parallel processing device and implemented through an extension to the programming language of the source program or through a different language than the programming language of the source program, wherein the sequential function calls perform at least one of array-based operations or matrix-based operations;
• and
• generating code for executing the source program in a data processing environment that includes a parallel processing device coupled to a sequential processing host, the generating including:
• detecting an occurrence of a given sequential function call of the sequential function calls in the source program;
• identifying a predefined parallel function call implementing equivalent functionality as the given sequential function call detected in the source program, where the identifying includes performing a look up on the one or more data structures;
• and
• including the predefined parallel function call identified as implementing equivalent functionality in the code generated for the source program, where the predefined parallel function call replaces the occurrence of the given sequential function call,
• wherein the predefined parallel function call that replaces the occurrence of the given sequential function call is selected from at least one of:
• a first library of Fast Fourier Transform (FFT) functions optimized for the parallel processing device;
• a second library of Deep Neural Network (DNN) functions optimized for the parallel processing device;
• a third library of Basic Linear Algebra Subprograms (BLAS) functions optimized for the parallel processing device;
• or
• a fourth library of dense and sparse direct solvers for use with Computer Vision, Computational Fluid Dynamics, Computational Chemistry, and Linear Optimization (SOLVER) functions optimized for the parallel processing device.

An apparatus comprising:

• a memory storing a source program conforming to a programming language and including a format for sequential execution;
• and
• one or more processors configured to:
• provide one or more data structures that map sequential function calls to parallel function calls that are predefined for execution on a parallel processing device and implemented through an extension to the programming language of the source program or through a different language than the programming language of the source program, wherein the sequential function calls perform at least one of array-based operations or matrix-based operations;
• and
• generate code for executing the source program in a data processing environment that includes a parallel processing device coupled to a sequential processing host, the generating including:
• detect an occurrence of a given sequential function call of the sequential function calls in the source program;
• identify a predefined parallel function call implementing equivalent functionality as the given sequential function call detected in the source program, where the identifying includes performing a look up on the one or more data structures;
• and
• include the predefined parallel function call identified as implementing equivalent functionality in the code generated for the source program, where the predefined parallel function call replaces the occurrence of the given sequential function call,
• wherein the predefined parallel function call that replaces the occurrence of the given sequential function call is selected from at least one of:
• a first library of Fast Fourier Transform (FFT) functions optimized for the parallel processing device;
• a second library of Deep Neural Network (DNN) functions optimized for the parallel processing device;
• a third library of Basic Linear Algebra Subprograms (BLAS) functions optimized for the parallel processing device;
• or
• a fourth library of dense and sparse direct solvers for use with Computer Vision, Computational Fluid Dynamics, Computational Chemistry, and Linear Optimization (SOLVER) functions optimized for the parallel processing device.