Apparatus and system for processing fusion of heterogeneous data including synchronized infrared images

A neural processing unit (NPU) for autonomous driving perception, comprising:

• an on-chip memory configured to store at least a portion of weight parameters of a fusion neural network model;
• and
• at least one processing element configured to execute the fusion neural network model;
• wherein the NPU is configured to receive heterogeneous sensor data comprising:
• a first data from a camera sensor;
• and
• a second data from an infrared sensor synchronized with a PWM-driven infrared light source;
• wherein the at least one processing element is configured to fuse the first data and the second data using the fusion neural network model to detect an object, and
• wherein the second data includes image information captured while the infrared light source is activated in synchronization with an exposure of the infrared sensor.

A system for processing sensor fusion data, comprising:

• a first sensor configured to capture a visible light image;
• a second sensor system comprising an infrared light source and an infrared camera, configured to capture an infrared image by synchronizing a pulse width modulation (PWM) driving of the infrared light source and an exposure timing of the infrared camera;
• and
• a neural processing unit (NPU) configured to:
• receive the visible light image and the infrared image;
• process a fusion artificial neural network (ANN) using the visible light image and the infrared image;
• and
• output an inference result indicating a presence of an object based on the visible light image and the infrared image.