# US 12,183,010 B2 · Systems and methods for instance segmentation based on semantic segmentation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,010 B2 - Document ID: US12183010B2 - Status: Granted - Kind: B2 - Application number: 17/651,243 - Filed: 2022-02-16 - Published: 2022-06-02 - Granted: 2024-12-31 - Earliest claimed: 2019-10-11 - Assignees: Beijing Didi Infinity Technology and Development Co., Ltd. - Applicants: Beijing Didi Infinity Technology and Development Co., Ltd. - Inventors: Jian Tang; Chengxiang Yin; Kun Wu; Zhengping Che - Web page: https://patentgrep.com/patent/US12183010B2 ## CPC - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215) - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06V 10/764: using classification, e.g. of video objects - G06T 2207/20084: Artificial neural networks [ANN] - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The present disclosure relates to a system and a method for performing instance segmentation based on semantic segmentation that is capable of (1) processing HD images in real-time given semantic segmentation; 2) delivering comparable performance with Mask R-CNN in terms of accuracy when combined with a widely-used semantic segmentation method (such as DPC), while consistently outperforms a state-of-the-art real-time solution; (3) working flexibly with any semantic segmentation model for instance segmentation; (4) outperforming Mask R-CNN if the given semantic segmentation is sufficiently good; and (5) being easily extended to panoptic segmentation. ## Claims ### Claim 1 A system for obtaining an instance segmentation or panoptic segmentation of an image based on semantic segmentation, the system comprising: a storage medium storing a set of instructions; and a processor in communication with the storage medium to execute the set of instructions to: perform semantic segmentation on an input image to obtain a semantic label map having specific set of classes, using a trained semantic segmentation model; generate a boundary map, using a trained generator, based on the obtained semantic label map concatenated with the input image; and process the boundary map, using a post-processing step, to differentiate objects of the specific set of classes to obtain the instance segmentation or panoptic segmentation of the input image, wherein the post-processing step comprises performing Breadth-First-Search for each enclosed area of the semantic label map to get a mask for each enclosed area, a class of the mask being determined based on its semantic label map. ### Claim 2 (depends on 1) The system of claim 1, wherein the trained semantic segmentation model is DeepLabv3+. ### Claim 3 (depends on 1) The system of claim 1, wherein the trained semantic segmentation model is dense prediction cell (DPC). ### Claim 4 (depends on 1) The system of claim 1, wherein the trained generator comprises a conditional Generative Adversarial Networks (GANs) coupled with deep supervision as well as a weighted fusion layer. ### Claim 5 (depends on 1) The system of claim 1, wherein the system is able to obtain instance segmentation or panoptic segmentation in real time. ### Claim 6 (depends on 1) The system of claim 1, wherein the set of instructions further instructs the processor to generate masks for at least one of thing classes and stuff classes. ### Claim 7 (depends on 1) The system of claim 1, further comprising a discriminator that engages in a minimax game with a generator to form the trained generator, wherein the discriminator distinguishes between a boundary map generated by the trained generator and a corresponding boundary map of ground truth. ### Claim 8 A method for obtaining an instance segmentation or panoptic segmentation of an image based on semantic segmentation, on a computing device including a storage medium storing a set of instructions, and a processor in communication with the storage medium to execute the set of instructions, the method comprising: performing semantic segmentation on an input image to obtain a semantic label map having specific set of classes, using a trained semantic segmentation model; generating a boundary map, using a trained generator, based on the obtained semantic label map concatenated with the input image; and processing the boundary map, using a post-processing step, to differentiate objects of the specific set of classes to obtain the instance segmentation or panoptic segmentation of the input image, wherein the post-processing step comprises performing Breadth-First-Search for each enclosed area of the semantic label map to get a mask for each enclosed area, a class of the mask being determined based on its semantic label map. ### Claim 9 (depends on 8) The method of claim 8, wherein the trained semantic segmentation model is DPC. ### Claim 10 (depends on 8) The method of claim 8, wherein the trained generator comprises a conditional Generative Adversarial Networks (GANs) coupled with deep supervision as well as a weighted fusion layer. ### Claim 11 (depends on 8) The method of claim 8, wherein the instance segmentation or panoptic segmentation is obtained in real time. ### Claim 12 (depends on 8) The method of claim 8, further comprising generating masks for at least one of thing classes and stuff classes. ### Claim 13 (depends on 8) The method of claim 8, further comprising using a discriminator to distinguish between a boundary map generated by the trained generator and a corresponding boundary map of ground truth to engage in a minimax game with a generator to form the trained generator. ### Claim 14 A computer non-transitory readable medium, storing a set of instructions for obtaining an instance segmentation or panoptic segmentation of an image based on semantic segmentation, wherein when the set of instructions is executed by a processor of an electrical device, the electrical device performs a method comprising: performing semantic segmentation on an input image to obtain a semantic label map having specific set of classes, using a trained semantic segmentation model; generating a boundary map, using a trained generator, based on the obtained semantic label map concatenated with the input image; and processing the boundary map, using a post-processing step, to differentiate objects of the specific set of classes to obtain the instance segmentation or panoptic segmentation of the input image, wherein the instance segmentation or panoptic segmentation is obtained in real time, wherein the post-processing step comprises performing Breadth-First-Search for each enclosed area of the semantic label map to get a mask for each enclosed area, a class of the mask being determined based on its semantic label map. ### Claim 15 (depends on 14) The computer non-transitory readable medium of claim 14, wherein the trained semantic segmentation model is DPC and the trained generator comprises a conditional Generative Adversarial Networks (GANs) coupled with deep supervision as well as a weighted fusion layer. ### Claim 16 (depends on 14) The computer non-transitory readable medium of claim 14, the method further comprising generating masks for at least one of thing classes and stuff classes. ### Claim 17 (depends on 14) The computer non-transitory readable medium of claim 14, the method further comprising using a discriminator to distinguish between a boundary map generated by the trained generator and a corresponding boundary map of ground truth to engage in a minimax game with a generator to form the trained generator.