Back to results
US 2024/0193432 A1Published application
Systems and methods for federated validation of models
AssigneeCapital One Services, LLC
InventorsKenny BEAN, Jeremy GOODSITT, Michael DAVIS, Taylor TURNER, Tyler FARNAN
Life of this patent
Estimating the term…
Abstract
Methods and systems described herein for validating machine learning models in federated machine learning model environments. More specifically, the methods and systems relate to unloading training and validation techniques to client devices using newly collected data to improve accuracy of federated machine learning models.
Claims20
A system for validating a machine learning model using federated machine learning process, the system comprising:
- one or more processors;
- and
- a non-transitory computer-readable storage medium storing instructions, which when executed by the one or more processors cause the one or more processors to perform operations comprising:
- receiving, from a plurality of client devices, a plurality of sets of machine learning model data, wherein the machine learning model data in each set comprises corresponding updated model parameters resulting from training each machine learning model at a corresponding client device of the plurality of client devices using a training dataset stored at each corresponding client device of the plurality of client devices;
- in response to receiving the plurality of sets of the machine learning model data, combining each set of the updated model parameters into an aggregated set of model parameters;
- transmitting to each client device a first request to update a corresponding machine learning model using the aggregated set of model parameters;
- in response to receiving a confirmation from each of the plurality of client devices that the corresponding machine learning model has been updated, transmitting a second request to each of the plurality of client devices to validate the machine learning model using a validation dataset, wherein each client device of the plurality of client devices validates the corresponding machine learning model using a corresponding validation dataset generated on each client device;
- in response to the second request, receiving a corresponding performance score from each of the plurality of client devices, wherein each performance score indicates an accuracy level of the corresponding machine learning model;
- and
- in response to determining that a first machine learning model corresponding to a first client device is associated with a performance score lower than a threshold, reverting a corresponding set of updated model parameters to an original set of model parameters.
7 more elements
A method for validating a machine learning model using federated machine learning process, the method comprising:
- receiving, from a plurality of client devices, a plurality of sets of machine learning model data, wherein the machine learning model data in each set comprises corresponding updated model parameters resulting from training each machine learning model at a corresponding client device of the plurality of client devices;
- aggregating each set of the updated model parameters into an aggregated set of model parameters;
- causing each client device to update a corresponding machine learning model using the aggregated set of model parameters;
- causing each of the plurality of client devices to validate the machine learning model using a validation dataset on each client device;
- receiving a corresponding performance score from each of the plurality of client devices, wherein each performance score indicates an accuracy level of the corresponding machine learning model;
- and
- in response to determining that a first machine learning model corresponding to a first client device is associated with a performance score lower than a threshold, generating a notification indicating the first client device.
4 more elements
A non-transitory, computer-readable storage medium storing instructions that when executed by one or more processors cause the one or more processors to perform operations comprising:
- receiving, from a client device, updated model parameters resulting from training a machine learning model at the client device;
- aggregating the updated model parameters with other updated model parameters from other client devices into an aggregated set of model parameters;
- causing the client device to update the machine learning model using the aggregated set of model parameters;
- causing the client device to validate the machine learning model using a validation dataset, wherein the validation dataset is stored on the client device, and wherein the validation dataset is a portion of a training dataset not used in training the machine learning model;
- receiving a performance score from the client device, wherein the performance score indicates an accuracy level of the machine learning model;
- and
- in response to determining that the performance score is lower than a threshold, generating a notification indicating the client device.
4 more elements
Dependencies are inferred from the claim text.