# US 11,908,178 B2 · Verification of computer vision models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,178 B2 - Document ID: US11908178B2 - Status: Granted - Kind: B2 - Application number: 17/646,911 - Filed: 2022-01-04 - Published: 2022-07-21 - Granted: 2024-02-20 - Earliest claimed: 2021-01-15 - Assignees: Robert Bosch GmbH - Applicants: Robert Bosch GmbH - Inventors: Christoph Gladisch; Christian Heinzemann; Matthias Woehrle - Web page: https://patentgrep.com/patent/US11908178B2 ## CPC - G06V 10/776: Validation; Performance evaluation - G06V 10/762: using clustering, e.g. of similar faces in social networks - G06V 10/764: using classification, e.g. of video objects - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Reducing the number of parameters in a visual parameter set based on a sensitivity analysis of how a given visual parameter affects the performance of a computer vision model to provide a verification parameter set having a reduced size. ## Claims ### Claim 1 A computer-implemented method for verifying a computer vision model, comprising the following steps: obtaining a computer vision model configured to perform a computer vision function of characterising elements of observed scenes; obtaining a visual parameter specification including at least one visual parameter set, wherein a classification or regression performance of the computer vision model is affected by an item of visual data provided based on the at least one visual parameter set; providing a first visual data set including a subset of items of visual data compliant with the visual parameter specification, and a corresponding subset of items of groundtruth data; performing a sensitivity analysis of a plurality of performance scores output from the computer vision model over a domain of the visual parameter specification when a plurality of items of visual data are applied to the visual data set; generating a verification parameter specification based on the visual parameter specification and the sensitivity analysis, wherein, compared to the visual parameter specification, the verification parameter specification includes at least one fewer visual parameter sets or visual parameters, and/or wherein the verification parameter specification includes at least one visual parameter set having a reduced range; and verifying the computer vision model by: providing a second visual data set including a second subset of items of visual data compliant with the verification parameter specification, and a corresponding second subset of items of groundtruth data, wherein the second visual data set is provided using the verification parameter specification, and applying at least one verification test to the computer vision model to obtain at least one verification test result. ### Claim 2 (depends on 1) The computer-implemented method according to claim 1, wherein the plurality of performance scores are provided by: generating, using the computer vision model, a plurality of predictions of elements of observed scenes in the subset of items of visual data, wherein the plurality of predictions include at least one prediction of a classification label and/or at least one regression value of at least one item in the subset of visual data; and comparing the plurality of predictions of elements in the subset of items of visual data with the corresponding subset of groundtruth data, to thus obtain the plurality of performance scores. ### Claim 3 (depends on 1) The computer-implemented method according to claim 1, further comprising: applying a plurality of verification tests to the computer vision model, wherein the at least one verification test result includes an average, or a weighted average, or worst-case test result, from the plurality of test results. ### Claim 4 (depends on 1) The computer-implemented method according to claim 1, wherein the providing of the second visual data set includes sampling the at least one visual parameter set using one of combinatorial testing, or search-based testing, or Latin hypercube sampling. ### Claim 5 (depends on 1) The computer-implemented method according to claim 1, wherein the plurality of performance scores includes or depends on, any one or any combination of a confusion matrix, or precision, or recall, or F1 score, or intersection of union, or mean average. ### Claim 6 (depends on 1) The computer-implemented method according to claim 1, further comprising: removing one or more visual parameters from the verification parameter specification, or adding one or more visual parameters to the verification parameter specification, to form an amended verification parameter specification; providing a third visual data set including a third subset of items of visual data compliant with the amended verification parameter specification, and a corresponding third subset of items of groundtruth data, wherein the third visual data set is provided using the amended verification parameter specification; applying a further verification test to the computer vision model using the third visual data set to obtain a further verification test result. ### Claim 7 (depends on 1) The computer-implemented method according to claim 1, wherein the first visual data set is provided based on a sampling of the visual parameters of the visual parameter specification, and the second visual data set is provided based on a sampling of the visual parameters of the verification parameter specification, wherein a sampling technique applied to the visual parameters of the verification parameter specification is different to a sampling technique applied to the sampling of the visual parameters of the visual parameter specification. ### Claim 8 (depends on 1) The computer-implemented method according to claim 1, further comprising: for each verification test applied to the computer vision model, storing in a verification coverage record at least one visual parameter describing each item of visual data from the second visual data set used to perform the each verification test; stopping the verification test when the verification coverage record reaches a predefined state. ### Claim 9 (depends on 1) The computer-implemented method according to claim 1, wherein the performing of the sensitivity analysis further includes: computing a plurality of variances of the plurality of performance scores respect to the visual parameters of the visual parameter specification; and ranking the visual parameters of the visual parameter specification based on the computed plurality of variances. ### Claim 10 (depends on 1) The computer-implemented method according to claim 1, wherein the generating of the verification visual parameter specification includes modifying the at least one visual parameter set by dividing at least one visual parameter set of the visual parameter specification into at least a first and a second modified visual parameter set, or combining a first and a second visual parameter set of the visual parameter specification into a cluster. ### Claim 11 (depends on 1) The computer-implemented method according to claim 1, wherein the verification parameter specification includes at least one fewer visual parameters compared to the visual parameter specification, and/or at least one visual parameter of the verification parameter specification has a reduced numerical scope as compared to the corresponding at least one visual parameter of the visual parameter specification, thus providing a reduced verification parameter space. ### Claim 12 (depends on 1) The computer-implemented method according to claim 1, further comprising: displaying the at least one verification test result via a graphical user interface. ### Claim 13 An apparatus for verifying a computer vision model, comprising: an input interface; a processor; a memory; and an output interface; wherein the input interface is configured to obtain a computer vision model, which is configured to perform a computer vision function of characterising elements of observed scenes, and to obtain a visual parameter specification including at least one visual parameter set, wherein a classification or regression performance of the computer vision model is affected by an item of visual data provided based on the at least one visual parameter set; wherein the input interface is configured to provide a first visual data set including a subset of items of visual data compliant with the visual parameter specification, and a corresponding subset of items of groundtruth data; wherein the processor is configured to perform a sensitivity analysis of a plurality of performance scores output from the computer vision model over a domain of the visual parameter specification when a plurality of items of visual data are applied to the visual data set, to generate a verification parameter specification based on the visual parameter specification and the sensitivity analysis, wherein, compared to the visual parameter specification, the verification parameter specification includes at least one fewer visual parameter sets or visual parameters, and/or wherein the verification parameter specification comprises at least one visual parameter set having a reduced range; and wherein the processor is configured to verify the computer vision model by: providing a second visual data set including a second subset of items of visual data compliant with the verification parameter specification, and a corresponding second subset of items of groundtruth data, wherein the second visual data set is provided using the verification parameter specification; and applying at least one verification test to the computer vision model to obtain at least one verification test result. ### Claim 14 A non-transitory computer readable medium on which is stored a computer program for verifying a computer vision model, the computer program, when executed by a processor, causing the processor to perform the following steps: obtaining a computer vision model configured to perform a computer vision function of characterising elements of observed scenes; obtaining a visual parameter specification including at least one visual parameter set, wherein a classification or regression performance of the computer vision model is affected by an item of visual data provided based on the at least one visual parameter set; providing a first visual data set including a subset of items of visual data compliant with the visual parameter specification, and a corresponding subset of items of groundtruth data; performing a sensitivity analysis of a plurality of performance scores output from the computer vision model over a domain of the visual parameter specification when a plurality of items of visual data are applied to the visual data set; generating a verification parameter specification based on the visual parameter specification and the sensitivity analysis, wherein, compared to the visual parameter specification, the verification parameter specification includes at least one fewer visual parameter sets or visual parameters, and/or wherein the verification parameter specification includes at least one visual parameter set having a reduced range; and verifying the computer vision model by: providing a second visual data set including a second subset of items of visual data compliant with the verification parameter specification, and a corresponding second subset of items of groundtruth data, wherein the second visual data set is provided using the verification parameter specification, and applying at least one verification test to the computer vision model to obtain at least one verification test result.