Mechanisms for generating augmented sensor data

A computer system for training a generative model to insert an object in spatial sensor data, the computer system comprising:

• at least one memory storing computer-readable instructions;
• and
• at least one processor coupled to the at least one memory and configured to execute the computer-readable instructions, which upon execution cause the at least one processor to:
• receive a first training sample of spatial sensor data of a first sensor modality, and a second training sample of spatial sensor data of a second sensor modality, the first training sample and the second training sample capturing a common object;
• remove from the first training sample a first portion of sensor data corresponding to the object, resulting a cropped training sample;
• extract from the second training sample a second portion of spatial sensor data corresponding to the common object;
• and
• train the generative model to reconstruct the first training sample from the cropped training sample by:
• providing to the generative model:
• the cropped training sample as a target input, and
• the second portion of spatial sensor data as a reference input,
• resulting in a generated output sample of spatial sensor data, and
• tuning parameters of the generative model to reduce a reconstruction error between the first training sample and the generated output sample,
• resulting in a trained generative model configured to insert at inference, in a first set of spatial sensor data of the first modality received as a target input, an object indicated in a second set of spatial sensor data of the second modality received as a reference input.

A non-transitory computer readable medium embodying computer program instructions, the computer program instructions configured so as, when executed on one or more hardware processors, to implement operations comprising:

• receiving a first training sample of spatial sensor data of a first sensor modality, and a second training sample of spatial sensor data of a second sensor modality, the first training sample and the second training sample capturing a common object;
• removing from the first training sample a first portion of sensor data corresponding to the object, resulting a cropped training sample;
• extracting from the second training sample a second portion of spatial sensor data corresponding to the common object;
• and
• training a generative model to reconstruct the first training sample from the cropped training sample by:
• providing to the generative model:
• the cropped training sample as a target input, and
• the second portion of spatial sensor data as a reference input,
• resulting in a generated output sample of spatial sensor data, and
• tuning parameters of the generative model to reduce a reconstruction error between the first training sample and the generated output sample,
• resulting in a trained generative model configured to insert at inference, in a first set of spatial sensor data of the first modality received as a target input, an object indicated in a second set of spatial sensor data of the second modality received as a reference input.