# US 12,112,525 B2 · System and method for segment-aware semantic segmentation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,525 B2 - Document ID: US12112525B2 - Status: Granted - Kind: B2 - Application number: 18/383,493 - Filed: 2023-10-25 - Published: 2024-04-25 - Granted: 2024-10-08 - Earliest claimed: 2022-10-25 - Assignees: Geox Gis Innovations Ltd. - Applicants: Geox Gis Innovations Ltd. - Inventors: Lior Shabtay; Alon Wander - Web page: https://patentgrep.com/patent/US12112525B2 ## CPC - G06V 10/776: Validation; Performance evaluation - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06V 10/764: using classification, e.g. of video objects - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/82: using neural networks - G06V 20/17: taken from planes or by drones - G06V 20/176: {Urban or other man-made structures} - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Updating a machine learning model for semantic segmentation by receiving an image and a segment ID mask, the image having a plurality of pixels, and the segment ID mask containing for each pixel of the image one of a plurality of segment identifiers identifying a segment to which the pixel belongs, feeding the image to a machine learning model and computing an outcome, computing a per-segment value for each one of the plurality of segment identifiers from the outcome and from pixels associated with the one of the plurality of segment identifiers, aggregating a plurality of per-segment values to create a segment-aware value and using the segment-aware value for training, validating and testing the machine learning model. ## Claims ### Claim 1 A computer-implemented method comprising: receiving an image, a ground truth annotation mask and a segment ID mask, wherein the image comprises a plurality of pixels, the ground truth annotation mask comprises for each pixel a label of a correct class, and the segment ID mask comprises for each pixel a segment identifier identifying a segment to which the pixel belongs; feeding the image to a machine learning model for semantic segmentation and computing an outcome; computing a per-segment value for each one of the plurality of segments from the information in the outcome and in the ground truth annotation mask related to pixels associated with the one of the plurality of segments in the segment ID mask; calculating a segment-aware value from a plurality of per-segment values; using the segment-aware value for training the machine learning model for semantic segmentation; and updating a plurality of parameters of the semantic segmentation machine learning model using the segment-aware value, wherein the step of computing the per-segment value further comprises computing, for each one of the plurality of pixels, a per-pixel loss value using the ground truth annotation mask and the outcome, wherein the outcome is a per class probability map containing for each pixel a probability of belonging to each class, and computing a per-segment value for each one of the plurality of segments using a plurality of per-pixel loss values of pixels associated with the one of the plurality of segments, thereby creating a per-segment loss, and wherein the step of calculating a segment-aware value further comprises computing a segment aware-loss value as an aggregation of the per-segment loss values. ### Claim 2 (depends on 1) The method of claim 1 wherein the machine learning model for semantic segmentation is a deep convolution network. ### Claim 3 (depends on 1) The method of claim 1 wherein the step of computing a per-segment loss value further comprises, for each segment, selecting K pixels from a plurality of pixels associated with the segment having a smallest per-pixel loss value, and selecting a highest value from the K selected pixels as the per-segment loss value, and the step of calculating a segment-aware loss value further comprises aggregating the per-segment loss values of all segments, thereby reflecting the ability of the model to correctly classify at least K pixels of each segment. ### Claim 4 (depends on 1) The method of claim 1 wherein the step of computing a per-segment value further comprises multiplying each per-segment value by a known per-segment weight thereby providing a weighted per-segment value signifying a different importance to different segments. ### Claim 5 A computer-implemented system comprising: at least one memory; at least one processor communicatively coupled to the memory; and a segment-aware semantic segmentation unit, operated by the at least one processor, configured to: receive an image, a ground truth annotation mask and a segment ID mask, wherein the image comprises a plurality of pixels, the ground truth annotation mask comprises for each pixel a label of a correct class and the segment ID mask comprises, for each pixel, a segment identifier identifying a segment to which the pixel belongs, compute a segment-aware value using the image, the ground truth annotation mask and the segment ID mask, and use the segment-aware value to perform any of: train a machine learning model for semantic segmentation, validate the machine learning model for semantic segmentation and test the machine learning model for semantic segmentation, wherein the segment-aware semantic segmentation unit further comprises a segment-aware function unit configured to compute a segment-aware value, a machine learning model for semantic segmentation, a training unit configured to train the machine learning model for semantic segmentation using the segment-aware value, a validation unit configured to provide a metric indicative of the quality of the machine learning model for semantic segmentation using the segment-aware value, and an inference unit configured to use the machine learning model for semantic segmentation and provide a segment-aware segmentation result containing for each pixel a predicted class, and wherein the segment-aware function unit further comprises a segment-aware loss module configured to use the segment ID map to compute a segment-aware loss value indicative of an ability of the machine learning model for semantic segmentation to correctly classify all segments in the image and a segment-aware metric module configured to use the segment ID map to compute a segment-aware metric value indicative of a quality of the machine learning model for semantic segmentation, and wherein the segment-aware loss module is configured to compute for each one of the plurality of pixels a per-pixel loss value between the ground truth annotation mask, and a per class probability map containing for each pixel a probability of belonging to each class, and compute a per-segment loss value for each one of the plurality of segments in the segment ID map using a plurality of per-pixel loss values of pixels associated with the one of the plurality of segments. ### Claim 6 (depends on 5) The system of claim 5 wherein the machine learning model for semantic segmentation is a deep convolution network. ### Claim 7 (depends on 5) The system of claim 5 wherein the segment-aware loss module is further configured to: select from a plurality of pixels associated with the segment in the segment ID mask K pixels having smallest per-pixel loss values; select a highest per-pixel loss value from the K selected pixels as the per-segment loss value; and aggregate the per-segment loss values of all segments to create a segment-aware loss value that reflects the ability of the model to correctly classify at least K pixels of each segment in the image. ### Claim 8 (depends on 5) The system of claim 5 wherein the segment-aware loss module is further configured to multiply each per-segment value by a known per-segment weight thereby providing a weighted per-segment value to each segment. ### Claim 9 (depends on 5) The system of claim 5 wherein the segment-aware metric module is further configured to: compute for each one of the plurality of pixels a per-pixel metric value using the ground truth annotation mask and the segmentation result; and compute a per-segment metric value for each one of the plurality of segments in the segment ID mask using a plurality of per-pixel metric values of pixels associated with the one of the plurality of segments. ### Claim 10 (depends on 9) The system of claim 9 wherein the segment-aware metric module is further configured to: count for each segment a number of correctly classified pixels wherein a correctly classified pixel is a pixel having a label of a predicted class in the segmentation result identical to the label in the ground truth annotation mask, and compute the segment-aware metric as a ratio between a number of segments for which at least a predetermined number of pixels were classified correctly, and a total number of segments present in the ground truth annotation mask. ### Claim 11 (depends on 9) The system of claim 9 wherein the segment-aware metric module is further configured to replace the segment ID mask with a containing segment ID mask, wherein each segment in the containing segment ID mask consists of a plurality of original pixels from a corresponding segment in the segment ID mask and at least one additional pixel located within a specified distance from one of the original pixels of the segment. ### Claim 12 (depends on 9) The system of claim 9 wherein the segment-aware metric module is further configured to compute a per-segment intersection over union (IoU) metric for each segment in the segment ID mask based on a plurality of pixels associated with the segment in the ground truth annotation mask and a plurality of pixels associated with the segment in the containing ID mask and computing the segment-aware value provides a segment-aware IoU metric computed as an average of the per-segment IoU metrics of the plurality of segments. ### Claim 13 A computer-implemented method comprising: receiving an image, a ground truth annotation mask and a segment ID mask, wherein the image comprises a plurality of pixels, the ground truth annotation mask comprises for each pixel a label of a correct class, and the segment ID mask comprises for each pixel a segment identifier identifying a segment to which the pixel belongs; feeding the image to a machine learning model for semantic segmentation and computing an outcome; computing a per-segment value for each one of the plurality of segments from the information in the outcome and in the ground truth annotation mask related to pixels associated with the one of the plurality of segments in the segment ID mask; calculating a segment-aware value from a plurality of per-segment values; and using the segment-aware value for validating the machine learning model for semantic segmentation and testing the machine learning model for semantic segmentation, wherein the step of computing a per-segment value further comprises computing for each one of the plurality of pixels a per-pixel metric value using the ground truth annotation mask and the outcome, wherein the outcome is a segmentation result containing for each pixel a label of a predicted class, and computing a per-segment value for each one of the plurality of segments in the segment ID mask using a plurality of per-pixel metric values of pixels associated with the one of the plurality of segments thereby creating a per-segment metric, wherein computing a per-segment metric further comprises counting for each segment a number of correctly classified pixels, wherein a correctly classified pixel is a pixel having a label of a predicted class in the segmentation result identical to the label in the ground truth annotation mask, and wherein the computing a segment-aware metric further comprises computing a ratio between the number of segments for which at least a predetermined number of pixels were classified correctly, and a total number of segments present in the ground truth annotation mask. ### Claim 14 (depends on 13) The method of claim 13 further comprising: creating a containing segment ID mask from the segment ID mask; and replacing the segment ID mask with the containing segment ID mask in the step of computing a per-segment value, wherein the creating a containing segment ID mask comprises, for each one segment, assigning an identifier of the one segment in the segment ID mask to a plurality of original pixels within a corresponding segment in the containing segment ID mask, and assigning the identifier to at least one additional pixel located within a certain distance from one of the original pixels of the segment in the containing segment ID mask thereby associating the at least one of the additional pixels with the segment in the containing segment ID mask. ### Claim 15 (depends on 14) The method of claim 14 wherein wherein the step of computing a per-segment metric further comprises, for each one segment, computing a per-segment intersection over union (IoU) between a plurality of pixels associated with the one segment in the ground truth annotation mask and a plurality of pixels associated with the one segment in the containing segment ID mask, and wherein the step of calculating a segment-aware IoU comprises computing an average between the plurality of per-segment IoU.