# US 12,217,431 B2 · Systems and methods for panoptic segmentation of images for autonomous driving > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,431 B2 - Document ID: US12217431B2 - Status: Granted - Kind: B2 - Application number: 17/649,960 - Filed: 2022-02-04 - Published: 2023-08-10 - Granted: 2025-02-04 - Earliest claimed: 2022-02-04 - Assignees: Ford Global Technologies, LLC - Applicants: Ford Global Technologies, LLC - Inventors: Guy Hotson; Nicolas Cebron; John Ryan Peterson; Marius Seritan; Craig Bryan - Web page: https://patentgrep.com/patent/US12217431B2 ## CPC - G06T 7/11: Region-based segmentation - G06V 10/147: Details of sensors, e.g. sensor lenses (fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19) - G06V 10/225: {based on a marking or identifier characterising the area} - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/7747: {Organisation of the process, e.g. bagging or boosting} - G06V 10/776: Validation; Performance evaluation - G06V 10/82: using neural networks - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30242: Counting objects in image - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06T 2210/12: Bounding box - G06T 2219/004: Annotating, labelling - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Systems and methods for generating a panoptic segmentation mask for an input image. The methods include receiving the input image comprising a plurality of pixels, generating a semantic mask and an instance mask from the input image, and combining the semantic mask and the instance mask to generate a panoptic mask for the input image. The semantic mask includes a single-channel mask that associates each pixel in the input image with a corresponding one of a plurality of labels. The instance mask includes a plurality of masks, where each of the plurality of masks identifies an instance of a countable object in the input image, and is associated with an indication of whether that instance of the countable object is hidden behind another object in the input image. ## Claims ### Claim 1 A method for panoptic image segmentation, the method comprising: receiving an input image comprising a plurality of pixels; generating, from the input image, a semantic mask and an instance mask, wherein: the semantic mask comprises a single-channel mask that associates each pixel in the input image with a corresponding one of a plurality of labels, and the instance mask comprises a plurality of masks, wherein each of the plurality of masks: identifies an instance of a countable object in the input image, and is associated with an indication of whether that instance of the countable object is hidden behind another object in the input image; and combining the semantic mask and the instance mask to generate a panoptic mask for the input image. ### Claim 2 (depends on 1) The method of claim 1, wherein each of the plurality of labels includes a semantic indication of a type of a countable object or amorphous region for each pixel in the input image. ### Claim 3 (depends on 1) The method of claim 1, further comprising using the panoptic mask to control operations of a mobile platform. ### Claim 4 (depends on 1) The method of claim 1, wherein the instance mask further comprises a plurality of bounding boxes, each of the plurality of bounding boxes being associated with an instance of one of a plurality of countable objects in the input image, the method further comprising: determine the instance mask as invalid responsive to an overlap between a first bounding box and a second bounding box is less than a threshold indicative of a first countable object associated with the first bounding box is hidden behind a second countable object associated with the second countable object, wherein the second countable object is transparent. ### Claim 5 (depends on 1) The method of claim 1, further comprising: generating the instance mask using an instance segmentation neural network; and generating the semantic mask with a semantic segmentation neural network. ### Claim 6 (depends on 5) The method of claim 5, further comprising training the semantic segmentation neural network and the instance segmentation neural network using a ground truth data set comprising a collection of ground truth examples, each ground truth example comprising: a training image; a ground truth panoptic mask corresponding to the training image; and a plurality of instance labels, each of the plurality of instance labels comprising a modal bounding box for one of a plurality of countable objects in the training image, a class label, and pixel id associations with the ground truth panoptic mask. ### Claim 7 (depends on 6) The method of claim 6, further comprising generating a ground truth semantic mask from the ground truth panoptic mask for training the semantic segmentation neural network. ### Claim 8 (depends on 6) The method of claim 6, further comprising generating, from the ground truth panoptic mask and the plurality of instance labels, a ground truth instance mask for training the instance segmentation neural network, the ground truth instance mask comprising: a mask for each modal bounding box and an indication of mask validity for that modal bounding box. ### Claim 9 (depends on 8) The method of claim 8, wherein: the indication of mask validity for that modal bounding box provides information relating to whether or not that modal bounding box includes an instance of a countable object of the training image that is not hidden behind other objects in the training image; and the indication of mask validity is determined by computing, for each instance in the ground truth panoptic mask, an overlap between an outline encompassing that instance and a corresponding modal bounding box. ### Claim 10 (depends on 9) The method of claim 9, further comprising training the instance segmentation neural network by conditioning loss propagation during training upon the indication of mask validity such that no loss is propagated for predicting, by the instance segmentation neural network, a bounding box for an instance of a countable object that is hidden behind other objects in the training image. ### Claim 11 (depends on 9) The method of claim 9, further comprising determining that a mask is not valid when the overlap is less than a threshold. ### Claim 12 A system for panoptic image segmentation, the system comprising: a processor; and a non-transitory computer readable medium comprising programming instructions that when executed by the processor, will cause the processor to: receive an input image comprising a plurality of pixels; generate, from the input image, a semantic mask and an instance mask, wherein: the semantic mask comprises a single-channel mask that associates each pixel in the input image with a corresponding one of a plurality of labels, and the instance mask comprises a plurality of masks, wherein each of the plurality of masks: identifies an instance of a countable object in the input image, and is associated with an indication of whether that instance of the countable object is hidden behind another object in the input image; and combine the semantic mask and the instance mask to generate a panoptic mask for the input image, wherein the instance mask further comprises a plurality of bounding boxes, each of the plurality of bounding boxes being associated with an instance of one of a plurality of countable objects in the input image, the non-transitory computer readable medium further comprising programming instructions that when executed by the processor, will cause the processor to: determine the instance mask as invalid responsive to an overlap between a first bounding box and a second bounding box is less than a threshold indicative of a first countable object associated with the first bounding box is hidden behind a transparent amorphous region. ### Claim 13 (depends on 12) The system of claim 12, further comprising programming instructions that when executed by the processor, will cause the processor to use the panoptic mask to control operations of a mobile platform. ### Claim 14 (depends on 12) The system of claim 12, further comprising programming instructions that when executed by the processor, will cause the processor to: generate the instance mask using an instance segmentation neural network; and generate the semantic mask with a semantic segmentation neural network. ### Claim 15 (depends on 14) The system of claim 14, further comprising programming instructions that when executed by the processor, will cause the processor to train the semantic segmentation neural network and the instance segmentation neural network using a ground truth data set comprising a collection of ground truth examples, each ground truth example comprising: a training image; a ground truth panoptic mask corresponding to the training image; and a plurality of instance labels, each of the plurality of instance labels comprising a modal bounding box for one of a plurality of countable objects in the training image, a class label, and pixel id associations with the ground truth panoptic mask. ### Claim 16 (depends on 15) The system of claim 15, further comprising programming instructions that when executed by the processor, will cause the processor to generate a ground truth semantic mask from the ground truth panoptic mask for training the semantic segmentation neural network. ### Claim 17 (depends on 15) The system of claim 15, further comprising programming instructions that when executed by the processor, will cause the processor to generate, from the ground truth panoptic mask and the plurality of instance labels, a ground truth instance mask for training the instance segmentation neural network, the ground truth instance mask comprising: a mask for each modal bounding box and an indication of mask validity for that modal bounding box. ### Claim 18 (depends on 17) The system of claim 17, wherein: the indication of mask validity for that modal bounding box provides information relating to whether or not that modal bounding box includes an instance of a countable object of the training image that is not hidden behind other objects in the training image; and the indication of mask validity is determined by computing, for each instance in the ground truth panoptic mask, an overlap between an outline encompassing that instance and a corresponding modal bounding box. ### Claim 19 (depends on 18) The system of claim 18, further comprising programming instructions that when executed by the processor, will cause the processor to train the instance segmentation neural network by conditioning loss propagation during training upon the indication of mask validity such that no loss is propagated for predicting, by the instance segmentation neural network, a bounding box for an instance of a countable object that is hidden behind other objects in the training image. ### Claim 20 A computer program product comprising a memory and programming instructions that are configured to cause a processor to: receive an input image comprising a plurality of pixels; generate, from the input image, a semantic mask and an instance mask, wherein: the semantic mask comprises a single-channel mask that associates each pixel in the input image with a corresponding one of a plurality of labels, and the instance mask comprises a plurality of masks, wherein each of the plurality of masks: identifies an instance of a countable object in the input image, and is associated with an indication of whether that instance of the countable object is hidden behind another object in the input image; and combine the semantic mask and the instance mask to generate a panoptic mask for the input image.