Systems and methods for camera-to-radar knowledge distillation
AssigneeRobert Bosch GmbH
InventorsSirajum Munir, Wenpeng Wang
Life of this patent
Estimating the term…
Abstract
A workflow is described herein for training a radar data processing model with a reduced requirement for annotated training data. The training workflow leverages an image sensor, such as a camera, that is synchronized with a radar sensor to capture synchronous image and radar point cloud data. The training workflow utilizes a self-supervised knowledge distillation process to pre-train the radar data processing model on the unannotated synchronous image and radar point cloud data, using a pre-trained image model. Subsequently, the radar data processing model is fine-tuned with a limited set of annotated radar point cloud data, thereby greatly reducing the human annotation burden.
Claims20
A method for training a first neural network to perform a radar data processing task, the method comprising:
- receiving, with a processor, a plurality of training data pairs, each respective training data pair in the plurality of training data pairs including a respective image and a respective radar point cloud, which were captured synchronously with one another of a same scene;
- training, with the processor, in a first phase based on the plurality of training data pairs, the first neural network to extract features from radar point clouds using a second neural network that is pre-trained to extract features from images;
- receiving, with the processor, a plurality of annotated radar point clouds, the annotated radar point clouds having labels corresponding to a radar data processing task;
- and
- further training, with the processor, in a second phase based on the plurality of annotated radar point clouds, the first neural network to perform the radar data processing task.
A non-transitory computer-readable medium for training a first neural network to perform a radar data processing task, the non-transitory computer-readable medium storing program instructions that, when executed by a processor, cause the processor to:
- receive a plurality of training data pairs, each respective training data pair in the plurality of training data pairs including a respective image and a respective radar point cloud, which were captured synchronously with one another of a same scene;
- train, in a first phase based on the plurality of training data pairs, the first neural network to extract features from radar point clouds using a second neural network that is pre-trained to extract features from images;
- receive a plurality of annotated radar point clouds, the annotated radar point clouds having labels corresponding to a radar data processing task;
- and
- further train, in a second phase based on the plurality of annotated radar point clouds, the first neural network to perform the radar data processing task.
Dependencies are inferred from the claim text.