Cross-modal dense semantic label generation by diffusion and iterative inpainting
AssigneeQUALCOMM Incorporated
InventorsVarun Ravi Kumar, Balaji Shankar Balachandran, Senthil Kumar Yogamani
Life of this patent
Estimating the term…
Abstract
Generating dense semantic labels for objects in a camera image may be accomplished by constructing an image graph where nodes of the image graph represent pixels of a camera image; performing a first diffusion of labels on the image graph using sparse labels from a point cloud sensor to generate propagated labels; applying inpainting to one or more regions of the camera image to generate inpainted labels; performing a second diffusion of labels on the image graph to update the propagated labels; and fusing the propagated labels and the inpainted labels.
Claims14
A method comprising:
- constructing, by a computing system, an image graph where nodes of the image graph represent pixels of a camera image;
- performing a first diffusion of labels on the image graph using sparse labels from a point cloud sensor to generate propagated labels;
- applying inpainting to one or more regions of the camera image to generate inpainted labels;
- performing a second diffusion of labels on the image graph to update the propagated labels;
- fusing the propagated labels and the inpainted labels to generate dense semantic labels for the camera image, wherein fusing the propagated labels and the inpainted labels comprises updating segmentation masks for the pixels using the propagated labels and the inpainted labels;
- and
- repeating the inpainting and the second diffusion until a convergence of labels is achieved, wherein the convergence is determined based at least in part on a similarity between consecutive segmentation masks.
An apparatus comprising:
- at least one memory;
- and
- processing circuitry in communication with the at least one memory, the processing circuitry configured to:
- construct an image graph where nodes of the image graph represent pixels of a camera image;
- perform a first diffusion of labels on the image graph using sparse labels from a point cloud sensor to generate propagated labels;
- apply inpainting to one or more regions of the camera image to generate inpainted labels;
- perform a second diffusion of labels on the image graph to update the propagated labels;
- fuse the propagated labels and the inpainted labels to generate dense semantic labels for the camera image, wherein the processing circuitry configured to fuse the propagated labels and the inpainted labels is further configured to update segmentation masks for the pixels using the propagated labels and the inpainted labels;
- and
- repeat the inpainting and the second diffusion until a convergence of labels is achieved, wherein the processing circuitry configured to determine convergence is further configured to determine convergence based at least in part on a similarity between consecutive segmentation masks.
A non-transitory computer-readable medium storing instructions, that when executed by processing circuitry, cause the processing circuitry to:
- construct an image graph where nodes of the image graph represent pixels of a camera image;
- perform a first diffusion of labels on the image graph using sparse labels from a point cloud sensor to generate propagated labels;
- apply inpainting to one or more regions of the camera image to generate inpainted labels;
- perform a second diffusion of labels on the image graph to update the propagated labels;
- fuse the propagated labels and the inpainted labels to generate dense semantic labels for the camera image, wherein the instructions to fuse the propagated labels and the inpainted labels further cause the processing circuitry to update segmentation masks for the pixels using the propagated labels and the inpainted labels;
- and
- repeat the inpainting and the second diffusion until a convergence of labels is achieved, wherein the instructions to determine convergence further cause the processing circuitry to determine convergence based at least in part on a similarity between consecutive segmentation masks.
Dependencies are inferred from the claim text.