# US 12,118,788 B2 · Learning semantic segmentation models in the absence of a portion of class labels > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,788 B2 - Document ID: US12118788B2 - Status: Granted - Kind: B2 - Application number: 17/592,378 - Filed: 2022-02-03 - Published: 2023-08-03 - Granted: 2024-10-15 - Earliest claimed: 2022-02-03 - Assignees: Robert Bosch GmbH - Applicants: Robert Bosch GmbH - Inventors: S Alireza Golestaneh; João D. Semedo; Filipe J. Cabrita Condessa; Wan-Yi Lin; Stefan Gehrer - Web page: https://patentgrep.com/patent/US12118788B2 ## CPC - G06V 10/82: using neural networks - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 10/764: using classification, e.g. of video objects - G06V 10/765: {using rules for classification or partitioning the feature space} - G06V 10/7753: {Incorporation of unlabelled data, e.g. multiple instance learning [MIL]} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Performing semantic segmentation in an absence of labels for one or more semantic classes is provided. One or more weak predictors are utilized to obtain label proposals of novel classes for an original dataset for which at least a subset of sematic classes are unlabeled classes. The label proposals are merged with ground truth of the original dataset to generate a merged dataset, the ground truth defining labeled classes of portions of the original dataset. A machine learning model is trained using the merged dataset. The machine learning model is utilized for performing semantic segmentation on image data. ## Claims ### Claim 1 A method for performing semantic segmentation in an absence of labels for one or more semantic classes, comprising: utilizing one or more weak predictors to obtain label proposals of novel classes for an original dataset for which at least a subset of sematic classes are unlabeled classes; merging the label proposals with ground truth of the original dataset to generate a merged dataset, the ground truth defining labeled classes of portions of the original dataset; training a machine learning model using the merged dataset; and utilizing the machine learning model for performing semantic segmentation on image data. ### Claim 2 (depends on 1) The method of claim 1, wherein the one or more weak predictors include a feature-based predictor configured to generate the label proposals by detecting edges within a region of interest in the original dataset. ### Claim 3 (depends on 1) The method of claim 1, wherein the one or more weak predictors include a transfer-based predictor configured to generate the label proposals by using a transfer model trained to predict classes of interest on a transfer information dataset separate from the original dataset. ### Claim 4 (depends on 1) The method of claim 1, wherein the merging of the label proposals includes assigning pixels of the original dataset to the unlabeled classes if so indicated by the label proposals; and assigning remaining pixels of the original dataset to the labeled classes specified by the ground truth of the original dataset. ### Claim 5 (depends on 1) The method of claim 1, wherein the training of the machine learning model utilizes a standard loss function that measures mismatch between predictions of the machine learning model and the ground truth. ### Claim 6 (depends on 1) The method of claim 1, wherein the training of the machine learning model utilizes a modified loss function that includes a standard loss function that measures mismatch between predictions of the machine learning model and the ground truth plus additional components that target losses specific to one or more of the novel classes. ### Claim 7 (depends on 1) The method of claim 1, wherein the semantic segmentation of the image data includes assigning of pixels of the image data to classes indicated in the ground truth as well as to the novel classes indicated in the label proposals. ### Claim 8 A system for performing semantic segmentation in an absence of labels for one or more semantic classes, comprising: one or more processors programmed to: utilize one or more weak predictors to obtain label proposals of novel classes for an original dataset for which at least a subset of sematic classes are unlabeled classes; merge the label proposals with ground truth of the original dataset to generate a merged dataset, the ground truth defining labeled classes of portions of the original dataset; train a machine learning model using the merged dataset; and utilize the machine learning model for performing semantic segmentation on image data. ### Claim 9 (depends on 8) The system of claim 8, wherein the one or more weak predictors include a feature-based predictor configured to generate the label proposals by detecting edges within a region of interest in the original dataset. ### Claim 10 (depends on 8) The system of claim 8, wherein the one or more weak predictors include a transfer-based predictor configured to generate the label proposals by using a transfer model trained to predict classes of interest on a transfer information dataset separate from the original dataset. ### Claim 11 (depends on 8) The system of claim 8, wherein to merge the label proposals includes to: assign pixels of the original dataset to the unlabeled classes if so indicated by the label proposals; and assign remaining pixels of the original dataset to the labeled classes specified by the ground truth of the original dataset. ### Claim 12 (depends on 8) The system of claim 8, wherein to train the machine learning model utilizes a standard loss function that measures mismatch between predictions of the machine learning model and the ground truth. ### Claim 13 (depends on 8) The system of claim 8, wherein to train the machine learning model utilizes a modified loss function that includes a standard loss function that measures mismatch between predictions of the machine learning model and the ground truth plus additional components that target losses specific to one or more of the novel classes. ### Claim 14 (depends on 8) The system of claim 8, wherein the semantic segmentation of the image data includes to assign of pixels of the image data to classes indicated in the ground truth as well as to the novel classes indicated in the label proposals. ### Claim 15 A non-transitory computer-readable medium comprising instructions for performing semantic segmentation in an absence of labels for one or more semantic classes, that, when executed by one or more processors case the one or more processors to perform operations including to: utilize one or more weak predictors to obtain label proposals of novel classes for an original dataset for which at least a subset of sematic classes are unlabeled classes; merge the label proposals with ground truth of the original dataset to generate a merged dataset, the ground truth defining labeled classes of portions of the original dataset; train a machine learning model using the merged dataset; and utilize the machine learning model for performing semantic segmentation on image data. ### Claim 16 (depends on 15) The medium of claim 15, wherein the one or more weak predictors include a feature-based predictor configured to generate the label proposals by detecting edges within a region of interest in the original dataset. ### Claim 17 (depends on 15) The medium of claim 15, wherein the one or more weak predictors include a transfer-based predictor configured to generate the label proposals by using a transfer model trained to predict classes of interest on a transfer information dataset separate from the original dataset. ### Claim 18 (depends on 15) The medium of claim 15, wherein to merge the label proposals includes to: assign pixels of the original dataset to the unlabeled classes if so indicated by the label proposals; and assign remaining pixels of the original dataset to the labeled classes specified by the ground truth of the original dataset. ### Claim 19 (depends on 15) The medium of claim 15, wherein to train the machine learning model utilizes a standard loss function that measures mismatch between predictions of the machine learning model and the ground truth. ### Claim 20 (depends on 15) The medium of claim 15, wherein to train the machine learning model utilizes a modified loss function that includes a standard loss function that measures mismatch between predictions of the machine learning model and the ground truth plus additional components that target losses specific to one or more of the novel classes. ### Claim 21 (depends on 15) The medium of claim 15, wherein the semantic segmentation of the image data includes to assign of pixels of the image data to classes indicated in the ground truth as well as to the novel classes indicated in the label proposals.