Methods and systems for the automated quality assurance of annotated images
AssigneeVolvo Car Corporation
InventorSohini Roy Chowdhury
Life of this patent
Estimating the term…
Abstract
A framework in which annotated images can be analyzed in small batches to learn and distinguish between higher-quality annotations and lower-quality annotations, especially in the case of manual annotations for which quality assurance is desired. This framework is extremely generalizable and can be used for indoor images, outdoor images, medical images, etc., without limitation. An echo state network (ESN) is provided as a special case of semantic segmentation model that can be trained using as few as tens of annotated images to predict semantic regions and provide metrics that can be used to distinguish between higher-quality annotations and lower-quality annotations.
Claims20
A method, comprising:
- training a plurality of parallel semantic segmentation models on an initial annotated dataset;
- generating a plurality of annotation regional proposals for a batch of images;
- computing a confidence metric indicating a degree of agreement between the plurality of annotation regional proposals;
- providing the batch of images to a first manual annotator and a second manual annotator to generate a first manual annotation set and a second manual annotation set and determining a first confidence score associated with the first manual annotator related to the first manual annotation set and a second confidence score associated with the second manual annotator related to the second manual annotation set;
- and
- assessing a preferred of the first manual annotator and the second manual annotator by comparing the first confidence score and the second confidence score;
- wherein the confidence metric is computed with the plurality of annotation regional proposals as inputs by computing scores for regional proposal pairs and computing a confidence comprising a mean over a variance.
A non-transitory computer readable medium stored in a memory and executed by a processor to execute the steps, comprising:
- training a plurality of parallel semantic segmentation models on an initial annotated dataset;
- using the trained plurality of parallel semantic segmentation models, generating a plurality of annotation regional proposals for a batch of images;
- using the plurality of annotation regional proposals, computing a confidence metric indicating a degree of agreement between the plurality of annotation regional proposals;
- providing the batch of images to a first manual annotator and a second manual annotator to generate a first manual annotation set and a second manual annotation set and determining a first confidence score associated with the first manual annotator related to the first manual annotation set and a second confidence score associated with the second manual annotator related to the second manual annotation set;
- and
- assessing a preferred of the first manual annotator and the second manual annotator by comparing the first confidence score and the second confidence score;
- wherein the confidence metric is computed with the plurality of annotation regional proposals as inputs by computing scores for regional proposal pairs and computing a confidence comprising a mean over a variance.
A system, comprising:
- a processor executing an algorithm stored in a memory operable for:
- training a plurality of parallel semantic segmentation models on an initial annotated dataset;
- using the trained plurality of parallel semantic segmentation models, generating a plurality of annotation regional proposals for a batch of images;
- using the plurality of annotation regional proposals, computing a confidence metric indicating a degree of agreement between the plurality of annotation regional proposals;
- providing the batch of images to a first manual annotator and a second manual annotator to generate a first manual annotation set and a second manual annotation set and determining a first confidence score associated with the first manual annotator related to the first manual annotation set and a second confidence score associated with the second manual annotator related to the second manual annotation set;
- and
- assessing a preferred of the first manual annotator and the second manual annotator by comparing the first confidence score and the second confidence score;
- wherein the confidence metric is computed with the plurality of annotation regional proposals as inputs by computing scores for regional proposal pairs and computing a confidence comprising a mean over a variance.
Dependencies are inferred from the claim text.