System and method for multimodal, motion-aware radar imaging

A method for reconstructing a radar reflectivity image of a scene including an object moving with the scene, wherein the method uses a processor operatively connected to an optical sensor and an electromagnetic sensor, wherein the processor is coupled with stored instructions implementing the method, wherein the instructions, when executed by the processor carry out steps of the method, comprising:

• tracking, using the optical sensor, the object over a period of time including multiple time steps to produce, for each of the multiple time steps, a deformation of a nominal shape of the object, wherein the nominal shape of the object is a shape of the object arranged in a prototypical pose;
• acquiring, using the electromagnetic sensor, snapshots of the scene over the multiple time steps to produce a set of radar reflectivity images of the object with deformed shapes defined by the corresponding deformations of the nominal shape of the object, wherein at least two different radar reflectivity images include the object with different deformed shapes;
• determining, for each of the multiple time steps using the deformation determined for the corresponding time step, a transformation between the radar reflectivity image of the object acquired by the electromagnetic sensor at the corresponding time step and a radar reflectivity image of the object in the prototypical pose;
• and
• combining the radar reflectivity images of the object with deformed shapes transformed with the corresponding transformations to produce the radar reflectivity image of the object in the prototypical pose.

A non-transitory computer readable storage medium embodied thereon a program executable by a processor for performing a method, wherein the processor is operatively connected to an optical sensor and an electromagnetic sensor, the method comprising:

• tracking, using the optical sensor, the object over a period of time including multiple time steps to produce, for each of the multiple time steps, a deformation of a nominal shape of the object, wherein the nominal shape of the object is a shape of the object arranged in a prototypical pose;
• acquiring, using the electromagnetic sensor, snapshots of the scene over the multiple time steps to produce a set of radar reflectivity image of the object with deformed shapes defined by the corresponding deformations of the nominal shape of the object, wherein at least two different radar reflectivity images include the object with different deformed shapes;
• determining, for each of the multiple time steps using the deformation determined for the corresponding time step, a transformation between the radar reflectivity image of the object acquired by the electromagnetic sensor at the corresponding time step and a radar reflectivity image of the object in the prototypical pose;
• and
• combining the radar reflectivity images of the object with deformed shapes transformed with the corresponding transformations to produce the radar reflectivity image of the object in the prototypical pose.