US 2026/0010786 A1Published application

Slice-based methods for edge case detection in machine learning models

AssigneeRobert Bosch GmbH

InventorsJorge Henrique Piazentin Ono, Wenbin He, Arvind Kumar Shekar, Liang Gou, Liu Ren

Life of this patent

Estimating the term…

Loading patent…

Abstract

Methods for a machine-learning network that provide efficient, scalable, and granular analyses during validation of a machine learning model are disclosed. Validation of models depends upon many factors, including the real-world application of the model, the type of model being trained, and the types of data samples it is being trained on. In order to provide relevant edge case information to users that pertains to their specific model, data slice finding techniques may be used to identify subsets of the dataset that are particularly problematic. By limiting a length of the slice description that the algorithm searches and by configuring the algorithm to target specific types of errors, users are provided with a more granular analysis that then allows them to determine how or if they need to retrain the model.

Claims20

Dependent claims

A computer-implemented method for a machine learning network, comprising:

providing a validation dataset to a machine learning model, wherein the validation dataset comprises:
data samples;
attributes that correspond to respective ones of the data samples;
and
ground truth labels;
executing the machine learning model to generate predictions associated with the data samples of the validation dataset;
and
executing a slice finding model, wherein the executing comprises:
identifying slices associated with the validation dataset, wherein the identifying the slices comprises:
receiving an indication of a length constraint associated with the attributes;
and
for each of the slice identifications,
determining a frequency of a number of the attributes that are common across two or more of the data samples, wherein the number of the attributes does not exceed the length constraint associated with the attributes;
and
defining a given slice based, at least in part, on the attributes that are common across the two or more of the data samples and on the frequency of the number of common attributes;
determining, for each of the identified slices, a performance metric value based, at least in part, on the generated predictions and on the ground truth labels;
and
displaying, via a user interface, the identified slices and corresponding performance metric values to a user of the machine learning network.

15 more elements

A system, comprising:

one or more processors;
and
memory having program instructions that, when executed by the one or more processors, cause the one or more processors to:
provide a validation dataset to a machine learning model, wherein the validation dataset comprises:
data samples;
attributes that correspond to respective ones of the data samples;
and
ground truth labels;
execute the machine learning model to generate predictions associated with the data samples of the validation dataset;
and
execute a slice finding model, wherein the execution of the slice finding model further cause the one or more processors to:
identify slices associated with the validation dataset, wherein the identification of the slices comprises:
receive an indication of a length constraint associated with the attributes;
for each of the slice identifications,
determine a frequency of a number of the attributes that are common across two or more of the data samples, wherein the number of the attributes does not exceed the length constraint associated with the attributes;
and
define a given slice based, at least in part, on the attributes that are common across the two or more of the data samples and on the frequency of the number of common attributes;
determine, for each of the identified slices, a performance metric value based, at least in part, on the generated predictions and on the ground truth labels;
and
display, via a user interface, the identified slices and corresponding performance metric values to a user.

17 more elements

One or more non-transitory, computer-readable media storing program instructions that, when executed on or across one or more processors, cause the one or more processors to:

receive a combined dataframe, wherein the combined dataframe comprises:
data samples of a validation dataset;
attributes that correspond to respective ones of the data samples;
ground truth labels;
predictions, generated by a machine learning model;
receive an indication of an length constraint associated with the attributes to be used when identifying slices;
and
execute a slice finding model, wherein the execution of the slice finding model further cause the one or more processors to:
identify slices associated with the validation dataset, wherein, for each of the slice identifications, the program instructions cause the one or more processors to:
determine a frequency of a number of the attributes that are common across two or more of the data samples, wherein the number of the attributes does not exceed the length constraint associated with the attributes;
and
define a given slice based, at least in part, on the attributes that are common across the two or more of the data samples and on the frequency of the number of common attributes;
determine, for each of the identified slices, a performance metric value based, at least in part, on the generated predictions and on the ground truth labels;
and
display, via a user interface, the identified slices and corresponding performance metric values to a user.

12 more elements

Dependencies are inferred from the claim text.