Optimized sensor fusion in deep learning accelerator with integrated random access memory

A device, comprising:

• random access memory;
• at least one interface configured to receive a plurality of inputs from a plurality of sensors respectively;
• and
• at least one processing unit coupled with the random access memory and the at least one interface and configured to execute the instructions having matrix operands to implement the matrix computations of an artificial neural network, the artificial neural network having a plurality of first portions and a second portion, the plurality of the first portions configured to process the plurality of inputs from the plurality of sensors respectively to generate first outputs corresponding to the plurality of sensors respectively, and the second portion configured to process a combination of the plurality inputs to generate a second output, wherein the second portion of the artificial neural network is configured to reduce overlapping processing with the first portions of the artificial neural network.

An apparatus, comprising:

• memory configured to store first data representative of parameters of an artificial neural network, second data representative of instructions executable to implement matrix computations of the artificial neural network using at least the first data stored in the memory, and third data representative of a plurality of inputs from a plurality of sensors respectively;
• at least one interface configured to receive the third data representative of the plurality of inputs from the plurality of sensors respectively and store the third data into the memory;
• a connection to the memory;
• and
• a Field-Programmable Gate Array (FPGA) or Application Specific Integrated circuit (ASIC) having:
• a memory interface configure to access the memory via the connection;
• and
• at least one processing unit configured to execute the instructions having matrix operands to implement the matrix computations of the artificial neural network, the artificial neural network having a plurality of first portions and a second portion, the plurality of the first portions configured to process the plurality of inputs from the plurality of sensors respectively to generate first outputs corresponding to the plurality of sensors respectively, and the second portion configured to process a combination of the plurality inputs to generate a second output.