Back to results
US 12,586,199 B2Granted
Diffusion-based open-vocabulary segmentation
AssigneeNVIDIA Corporation
InventorsJiarui Xu, Shalini De Mello, Sifei Liu, Arash Vahdat, Wonmin Byeon
Life of this patent
Estimating the term…
Loading patent…
Loading patent…
AssigneeNVIDIA Corporation
InventorsJiarui Xu, Shalini De Mello, Sifei Liu, Arash Vahdat, Wonmin Byeon
Estimating the term…
An open-vocabulary diffusion-based panoptic segmentation system is not limited to perform segmentation using only object categories seen during training, and instead can also successfully perform segmentation of object categories not seen during training and only seen during testing and inferencing. In contrast with conventional techniques, a text-conditioned diffusion (generative) model is used to perform the segmentation. The text-conditioned diffusion model is pre-trained to generate images from text captions, including computing internal representations that provide spatially well-differentiated object features. The internal representations computed within the diffusion model comprise object masks and a semantic visual representation of the object. The semantic visual representation may be extracted from the diffusion model and used in conjunction with a text representation of a category label to classify the object. Objects are classified by associating the text representations of category labels with the object masks and their semantic visual representations to produce panoptic segmentation data.
A method of generating segmentation data, comprising:
A system, comprising:
A non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause the one or more processors to generate segmentation data by performing the steps of:
Dependencies are inferred from the claim text.