Upsampling radar via multimodal neural fields

A system for training a multimodal network for multimodal scene reconstruction, the system comprising:

• a memory configured to store synchronized camera images and radar images captured from a scene;
• and
• one or more computing devices configured to:
• generate a shared spatial representation of the scene by encoding spatial features into a spatial hash table of a shared geometry encoder,
• decode the shared spatial representation using a geometry decoder to produce camera occupancy and radar occupancy values,
• apply a color multilayer perceptron (MLP) to the decoded camera occupancy to determine camera radiance and apply a radar MLP to the decoded radar occupancy to determine radar reflectance,
• generate camera density and radar density by applying a density decoding function to the spatial representation,
• render predicted camera images from the camera density and camera radiance via camera volumetric rendering,
• render predicted radar images from the radar density and radar reflectance via radar volumetric rendering, and
• optimize the shared geometry encoder based on a multimodal loss function, wherein the multimodal loss function comprises a first reconstruction loss term for the predicted camera images, a second reconstruction loss term for the predicted radar images, a proposal loss term to enforce consistency across multimodal ray samplings, and a sparsity constraint to encourage compact geometry representations, and
• output the trained multimodal network for use in high resolution radar simulation in response to optimizing the multimodal loss function.

A non-transitory computer-readable medium comprising instructions for training a multimodal scene reconstruction that, when executed by one or more computing devices, cause the one or more computing devices to perform operations including to:

• generate a shared spatial representation of the scene by encoding spatial features into a spatial hash table of a shared geometry encoder;
• decode the shared spatial representation using a geometry decoder to produce camera occupancy and radar occupancy values;
• apply a color multilayer perceptron (MLP) to the decoded camera occupancy to determine camera radiance and apply a radar MLP to the decoded radar occupancy to determine radar reflectance;
• generate camera density and radar density by applying a density decoding function to the spatial representation;
• render predicted camera images from the camera density and camera radiance via camera volumetric rendering;
• render predicted radar images from the radar density and radar reflectance via radar volumetric rendering;
• optimize the shared geometry encoder based on a multimodal loss function, wherein the multimodal loss function comprises a first reconstruction loss term for the predicted camera images, a second reconstruction loss term for the predicted radar images, a proposal loss term to enforce consistency across multimodal ray samplings, and a sparsity constraint to encourage compact geometry representations;
• and
• output the trained multimodal network for use in high resolution radar simulation in response to optimizing the multimodal loss function.