Sensor compensation using backpropagation

A computer program product comprising one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by a processor to cause the processor to perform operations comprising:

• training a first convolutional neural network (CNN) using a plurality of training images;
• generating, upon determining that the first CNN has been trained, a first trained CNN and a second trained CNN, wherein the first trained CNN and the second trained CNN are based on the first CNN;
• adding an interface layer to the second trained CNN;
• processing a first image in a sequence of successive images using the first trained CNN to generate a first result vector, wherein the first image receives a full-image analysis;
• processing a second image that is subsequent to the first image in the sequence of images using the first trained CNN to generate a second result vector;
• processing the second image using the second trained CNN and sensor data input to the interface layer to generate a third result vector, wherein the sensor data is collected after the first image is captured and prior to a capture of the second image, wherein the processing of the second image using the second trained CNN includes modifying the sensor data using a compensation value, and wherein, based on the modified sensor data, a cached first result vector from the first trained CNN is reused to label objects in subsequent images from the sequence of successive images based on sensor-detected movement and pixel shift;
• comparing the third result vector to the second result vector to generate an error value;
• calculating a modified compensation value using the error value via a backpropagation algorithm to reduce the error value;
• generating a sensor-compensated trained CNN based on the second trained CNN with the modified compensation value, wherein a sensor interface layer of the sensor-compensated trained CNN is updated with the modified compensation value and sensor data input to the interface layer of the sensor-compensated trained CNN is modified using the modified compensation value;
• and
• processing a third image using the sensor-compensated trained CNN and determining the third image comprises one or more objects depicted in the first image and the second image, tracked using the sensor data and pixel shift.

A computer system comprising a processor and one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by the processor to cause the processor to perform operations comprising:

• training a first convolutional neural network (CNN) using a plurality of training images;
• generating, upon determining that the first CNN has been trained, a first trained CNN and a second trained CNN, wherein the first trained CNN and the second trained CNN are based on the first CNN;
• adding an interface layer to the second trained CNN;
• processing a first image in a sequence of successive images using the first trained CNN to generate a first result vector, wherein the first image receives a full-image analysis;
• processing a second image that is subsequent to the first image in the sequence of images using the first trained CNN to generate a second result vector;
• processing the second image using the second trained CNN and sensor data input to the interface layer to generate a third result vector, wherein the sensor data is collected after the first image is captured and prior to a capture of the second image, wherein the processing of the second image using the second trained CNN includes modifying the sensor data using a compensation value, and wherein, based on the modified sensor data, a cached first result vector from the first trained CNN is reused to label objects in subsequent images from the sequence of successive images based on sensor-detected movement and pixel shift;
• comparing the third result vector to the second result vector to generate an error value;
• calculating a modified compensation value using the error value via a backpropagation algorithm to reduce the error value;
• generating a sensor-compensated trained CNN based on the second trained CNN with the modified compensation value, wherein a sensor interface layer of the sensor-compensated trained CNN is updated with the modified compensation value and sensor data input to the interface layer of the sensor-compensated trained CNN is modified using the modified compensation value;
• and
• processing a third image using the sensor-compensated trained CNN and determining the third image comprises one or more objects depicted in the first image and the second image, tracked using the sensor data and pixel shift.