Computer-implemented method for testing conformance between real and synthetic images for machine learning

A computer-implemented method for producing a synthetic image generator tested using a computer vision model, comprising the following steps:

• testing conformance of the synthetic image generator between images generated by a synthetic image generator and images obtained from authentic visual data, the testing including:
• obtaining a computer vision model in an initial training state configured to perform a computer vision function,
• obtaining a visual parameter set including a plurality of visual parameters,
• generating a synthetic visual data set including synthetic visual data and corresponding groundtruth data according to the visual parameter set, and sampling an authentic visual data set including authentic images and corresponding groundtruth data according to the visual parameter set;
• applying the computer vision model to the synthetic visual data set and the authentic visual data set, to obtain a predicted synthetic visual data set and a predicted authentic visual data set, respectively, and generating a plurality of synthetic and authentic performance scores over the plurality of visual parameters of the visual parameter set, wherein each of the plurality of synthetic and authentic performance scores is a comparison of an item of predicted synthetic visual data or predicted authentic visual data with a corresponding item of groundtruth data,
• (i) generating a first sensitivity measure of the plurality of synthetic performance scores over the plurality of visual parameters, and generating a second sensitivity measure of the plurality of authentic performance scores over the plurality of visual parameters, or (ii) generating a combined sensitivity measure based on a difference between corresponding synthetic and authentic performance scores over the plurality of visual parameters, and
• generating a conformance result defining the conformance between images generated by the synthetic image generator and images obtained from the authentic visual data at a same or similar visual parameter of the visual parameter set by (i) comparing the first sensitivity measure and the second sensitivity measure, or (ii) generating the conformance result based on the combined sensitivity measure;
• based on the conformance result being negative, adjusting the synthetic image generator and/or amending one or more of the plurality of visual parameters of the visual parameter set, and repeating the testing step, until the conformance result is positive;
• and
• outputting the adjusted synthetic image generator.

An apparatus for testing a conformance between images generated by a synthetic image generator and images obtained from authentic visual data, comprising:

• an input interface;
• a processor;
• a memory;
• and
• an output interface;
• wherein the input interface is configured to obtain a computer vision model in an initial training state configured to perform a computer vision function, and to obtain a visual parameter set comprising a plurality of visual parameters;
• wherein the processor is configured to generate a synthetic visual data set including synthetic visual data and corresponding groundtruth data according to the visual parameter set, and to sample an authentic visual data set comprising authentic images and corresponding groundtruth data according to the visual parameter set;
• wherein the processor is configured to apply the computer vision model to the synthetic visual data set and the authentic visual data set, to obtain a predicted synthetic visual data set and a predicted authentic visual data set, respectively, and to generate a plurality of synthetic and authentic performance scores over the plurality of visual parameters of the visual parameter set, wherein each of the synthetic and authentic performance scores is a comparison of an item of predicted synthetic visual data or predicted authentic visual data with a corresponding item of groundtruth data;
• wherein the processor is configured:
• to (i) generate a first sensitivity measure of the plurality of synthetic performance scores over the plurality of visual parameters, and to generate a second sensitivity measure of the plurality of authentic performance scores over the plurality of visual parameters, or (ii) generate a combined sensitivity measure based on a difference between corresponding synthetic and authentic performance scores over the plurality of visual parameters;
• and
• wherein the processor is configured to generate a conformance result defining the conformance between images generated by the synthetic image generator and images obtained from the authentic visual data at the same or similar visual parameter of the visual parameter set by (i) comparing the first sensitivity measure and the second sensitivity measure, or (ii) generating the conformance result based on the combined sensitivity measure.

A non-transitory computer readable medium on which is stored a computer program for testing a conformance between images generated by a synthetic image generator and images obtained from authentic visual data, the computer program, when executed by a processor, causing the processor to perform the following steps:

• obtaining a computer vision model in an initial training state configured to perform a computer vision function;
• obtaining a visual parameter set including a plurality of visual parameters;
• generating a synthetic visual data set including synthetic visual data and corresponding groundtruth data according to the visual parameter set, and sampling an authentic visual data set including authentic images and corresponding groundtruth data according to the visual parameter set;
• applying the computer vision model to the synthetic visual data set and the authentic visual data set, to obtain a predicted synthetic visual data set and a predicted authentic visual data set, respectively, and generating a plurality of synthetic and authentic performance scores over the plurality of visual parameters of the visual parameter set, wherein each of the plurality of synthetic and authentic performance scores is a comparison of an item of predicted synthetic visual data or predicted authentic visual data with a corresponding item of groundtruth data;
• (i) generating a first sensitivity measure of the plurality of synthetic performance scores over the plurality of visual parameters, and generating a second sensitivity measure of the plurality of authentic performance scores over the plurality of visual parameters, or (ii) generating a combined sensitivity measure based on a difference between corresponding synthetic and authentic performance scores over the plurality of visual parameters;
• and
• generating a conformance result defining the conformance between images generated by the synthetic image generator and images obtained from the authentic visual data at a same or similar visual parameter of the visual parameter set by (i) comparing the first sensitivity measure and the second sensitivity measure, or (ii) generating the conformance result based on the combined sensitivity measure.