# US 12,602,582 B2 · Dynamic distributed training of machine learning models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,582 B2 - Document ID: US12602582B2 - Status: Granted - Kind: B2 - Application number: 17/226,399 - Filed: 2021-04-09 - Published: 2022-10-13 - Granted: 2026-04-14 - Earliest claimed: 2021-04-09 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Abdullah Kayi; Wei Zhang; Xiaodong Cui; Alper Buyuktosunoglu - Web page: https://patentgrep.com/patent/US12602582B2 ## CPC - G06N 3/08: Learning methods - G06F 18/251: {of input or preprocessed data} - G06N 3/063: using electronic means - G06N 3/098: Distributed learning, e.g. federated learning - H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G06N 3/047: Probabilistic or stochastic networks - G06V 10/95: {structured as a network, e.g. client-server architectures} - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Computer hardware and/or software that performs the following operations: (i) updating a machine learning model by synchronously applying, to the machine learning model, a first set of training results received from a set of trainers having respective training datasets; (ii) receiving, from one or more trainers of the set of trainers, a first set of metrics pertaining to at least some of the training results of the first set of training results; and (iii) based, at least in part, on the first set of metrics, determining to subsequently update the machine learning model via asynchronous application of subsequent training results received from respective trainers of the set of trainers. ## Claims ### Claim 1 A computer-implemented method comprising: updating, by one or more processors, a machine learning model by synchronously applying, to the machine learning model, a first set of training results received at a sub-system from a set of trainers having respective training datasets, as synchronous training; receiving at the sub-system, by one or more processors, from one or more trainers of the set of trainers, a first set of metrics pertaining to at least some of the training results of the first set of training results; based, at least in part, on the first set of metrics, determining, by one or more processors on a configuration computer communicating with the sub-system, to subsequently update the machine learning model via asynchronous application of subsequent training results received from respective trainers of the set of trainers; in response to the first set of metrics indicating that an amount of time taken by a first trainer of the set of trainers to produce respective training results differs from an amount of time taken by one or more other trainers to produce respective training results by at least a threshold amount of time, switching, using the configuration computer, to an asynchronous training mode, the switching to the asynchronous training mode being based on the first trainer being a slow trainer; and switching back to the synchronous training, using the configuration computer, when the slow trainer is no longer the slow trainer when all the one or more trainers of the set of trainers report similar training times within a threshold distance. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: updating, by one or more processors, the machine learning model by applying, to the machine learning model, at a first time, a second set of training results received from a first trainer of the set of trainers; and updating, by one or more processors, the machine learning model by applying, to the machine learning model, at a second time, a third set of training results received from a second trainer of the set of trainers. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, further comprising: generating, by one or more processors, a first updated set of machine learning model parameters based, at least in part, on the synchronous application, to the machine learning model, of the first set of training results; and communicating, by one or more processors, the first updated set of machine learning model parameters to the set of trainers, wherein the second set of training results and the third set of training results are based, at least in part, on the first updated set of machine learning model parameters. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, further comprising: receiving, by one or more processors, from one or more trainers of the set of trainers, a second set of metrics pertaining to at least some training results of a second set of training results; and based, at least in part, on the second set of metrics, determining, by one or more processors, to subsequently update the machine learning model via synchronous application of subsequent training results received from the respective trainers of the set of trainers. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, further comprising: receiving, by one or more processors, respective training results from each trainer of the set of trainers; combining, by one or more processors, the respective training results from each trainer of the set of trainers into a third set of training results; and updating, by one or more processors, the machine learning model by synchronously applying, to the machine learning model, the third set of training results. ### Claim 6 (depends on 4) The computer-implemented method of claim 4, wherein the first set of metrics indicates that an amount of time taken by a first trainer of the set of trainers to produce respective training results of the first set of training results differs from an amount of time taken by one or more other trainers of the set of trainers to produce respective training results of the first set of training results by at least a threshold amount of time. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein the second set of metrics indicates that an amount of time taken by the first trainer to produce respective training results of the second set of training results differs from an amount of time taken by the one or more other trainers to produce respective training results of the second set of training results by less than the threshold amount of time. ### Claim 8 (depends on 6) The computer-implemented method of claim 6, wherein the second set of metrics indicates that an amount of difference between a current measure of machine learning model convergence and a desired measure of machine learning model convergence exceeds a threshold amount of difference. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, further comprising: configuring, by one or more processors, the set of trainers for training by sending an initial copy of the machine learning model to each trainer of the set of trainers. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the first set of training results includes respective machine learning model gradients for each trainer of the set of trainers. ### Claim 11 A computer program product comprising one or more computer readable storage media and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by one or more processors to cause the one or more processors to perform a method comprising: updating a machine learning model by synchronously applying, to the machine learning model, a first set of training results received at a sub-system from a set of trainers having respective training datasets, as synchronous training; receiving, at the sub-system, from one or more trainers of the set of trainers, a first set of metrics pertaining to at least some of the training results of the first set of training results; based, at least in part, on the first set of metrics, determining on a configuration computer communicating with the sub-system to subsequently update the machine learning model via asynchronous application of subsequent training results received from respective trainers of the set of trainers; in response to the first set of metrics indicating that an amount of time taken by a first trainer of the set of trainers to produce respective training results differs from an amount of time taken by one or more other trainers to produce respective training results by at least a threshold amount of time, switching, using the configuration computer, to an asynchronous training mode, the switching to the asynchronous training mode being based on the first trainer being a slow trainer; and switching back to the synchronous training, using the configuration computer, when the slow trainer is no longer the slow trainer when all the one or more trainers of the set of trainers report similar training times within a threshold distance. ### Claim 12 (depends on 11) The computer program product of claim 11, the method further comprising: updating the machine learning model by applying, to the machine learning model, at a first time, a second set of training results received from a first trainer of the set of trainers; and updating the machine learning model by applying, to the machine learning model, at a second time, a third set of training results received from a second trainer of the set of trainers. ### Claim 13 (depends on 12) The computer program product of claim 12, the method further comprising: generating a first updated set of machine learning model parameters based, at least in part, on the synchronous application, to the machine learning model, of the first set of training results; and communicating the first updated set of machine learning model parameters to the set of trainers, wherein the second set of training results and the third set of training results are based, at least in part, on the first updated set of machine learning model parameters. ### Claim 14 (depends on 11) The computer program product of claim 11, the method further comprising: receiving, from one or more trainers of the set of trainers, a second set of metrics pertaining to at least some training results of a second set of training results; and based, at least in part, on the second set of metrics, determining to subsequently update the machine learning model via synchronous application of subsequent training results received from the respective trainers of the set of trainers. ### Claim 15 (depends on 14) The computer program product of claim 14, the method further comprising: receiving respective training results from each trainer of the set of trainers; combining the respective training results from each trainer of the set of trainers into a third set of training results; and updating the machine learning model by synchronously applying, to the machine learning model, the third set of training results. ### Claim 16 A computer system comprising: one or more processors; and one or more computer readable storage media; wherein: the one are more processors are structured, located, connected and/or programmed to execute program instructions collectively stored on the one or more computer readable storage media; and the program instructions, when executed by the one or more processors, cause the one or more processors to perform a method comprising: updating a machine learning model by synchronously applying, to the machine learning model, a first set of training results received at a sub-system from a set of trainers having respective training datasets, as synchronous training; receiving at the sub-system, from one or more trainers of the set of trainers, a first set of metrics pertaining to at least some of the training results of the first set of training results; based, at least in part, on the first set of metrics, determining on a configuration computer communicating with the sub-system, to subsequently update the machine learning model via asynchronous application of subsequent training results received from respective trainers of the set of trainers; in response to the first set of metrics indicating that an amount of time taken by a first trainer of the set of trainers to produce respective training results differs from an amount of time taken by one or more other trainers to produce respective training results by at least a threshold amount of time, switching, using the configuration computer, to an asynchronous training mode, the switching to the asynchronous training mode being based on the first trainer being a slow trainer; and switching back to the synchronous training, using the configuration computer, when the slow trainer is no longer the slow trainer when all the one or more trainers of the set of trainers report similar training times within a threshold distance. ### Claim 17 (depends on 16) The computer system of claim 16, the method further comprising: updating the machine learning model by applying, to the machine learning model, at a first time, a second set of training results received from a first trainer of the set of trainers; and updating the machine learning model by applying, to the machine learning model, at a second time, a third set of training results received from a second trainer of the set of trainers. ### Claim 18 (depends on 17) The computer system of claim 17, the method further comprising: generating a first updated set of machine learning model parameters based, at least in part, on the synchronous application, to the machine learning model, of the first set of training results; and communicating the first updated set of machine learning model parameters to the set of trainers, wherein the second set of training results and the third set of training results are based, at least in part, on the first updated set of machine learning model parameters. ### Claim 19 (depends on 16) The computer system of claim 16, the method further comprising: receiving, from one or more trainers of the set of trainers, a second set of metrics pertaining to at least some training results of a second set of training results; and based, at least in part, on the second set of metrics, determining to subsequently update the machine learning model via synchronous application of subsequent training results received from the respective trainers of the set of trainers. ### Claim 20 (depends on 19) The computer system of claim 19, the method further comprising: receiving respective training results from each trainer of the set of trainers; combining the respective training results from each trainer of the set of trainers into a third set of training results; and updating the machine learning model by synchronously applying, to the machine learning model, the third set of training results.