# US 11,907,825 B2 · Training neural networks using distributed batch normalization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,825 B2 - Document ID: US11907825B2 - Status: Granted - Kind: B2 - Application number: 16/659,543 - Filed: 2019-10-21 - Published: 2020-04-23 - Granted: 2024-02-20 - Earliest claimed: 2018-10-19 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Blake Alan Hechtman; Sameer Kumar - Web page: https://patentgrep.com/patent/US11907825B2 ## CPC - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06N 3/098: Distributed learning, e.g. federated learning - G06V 10/82: using neural networks - G06N 3/063: using electronic means - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Methods, systems, and apparatus, including instructions encoded on storage media, for performing reduction of gradient vectors for distributed training of a neural network. One of the methods includes receiving, at each of the plurality of devices, a respective batch; performing, by each device, a forward pass comprising, for each batch normalization layer: generating, by each of the devices, a respective output of the corresponding other layer for each training example in the batch, determining, by each of the devices, a per-replica mean and a per-replica variance; determining, for each sub-group, a distributed mean and a distributed variance from the per-replica means and the per-replica variances for the devices in the sub-group; and applying, by each device, batch normalization to the respective outputs of the corresponding other layer generated by the device using the distributed mean and the distributed variance for the sub-group to which the device belongs. ## Claims ### Claim 1 A method of distributing the training of a neural network across a plurality of devices, wherein the neural network has a plurality of network parameters, wherein the neural network has one or more batch normalization layers, wherein each batch normalization layer applies batch normalization to outputs of a corresponding other layer in the neural network, wherein each device maintains a respective replica of the neural network, and wherein the method comprises: maintaining data that partitions the plurality of devices into a plurality of sub-groups that each comprise two or more of the plurality of the devices, wherein each of the plurality of devices comprises one or more hardware accelerators for training neural networks; receiving, at each of the plurality of devices, a respective batch of training examples; training the neural network on the respective batches of training examples received by the devices, the training comprising: performing, by each device, a forward pass through the replica maintained by the device to determine a respective network output for each training example in the batch received by the device, comprising, for each batch normalization layer in the neural network: generating, by each of the plurality of devices and using the replica maintained by the device, a respective output of the corresponding other layer for each training example in the batch received by the device, determining, by each of the plurality of devices, a per-replica mean and a per-replica variance of the respective outputs of the corresponding other layer for the training examples in the batch received by the device; determining, for each sub-group, a distributed mean and a distributed variance from the per-replica means and the per-replica variances for the two or more devices in the sub-group and determined from the respective batches received by the two or more devices in the sub-group by performing, over a network that connects the two or more devices in the group using one or more communication links, a distributed reduction operation on the per-replica means and the per-replica variances for the two or more devices in the sub-group and determined from the respective batches received by the two or more devices in the sub-group; and applying, by each device, batch normalization to the respective outputs of the corresponding other layer generated by the device using the distributed mean and the distributed variance for the sub-group to which the device belongs; and performing, by each device, a backward pass through the replica maintained by the device to determine a respective update to the parameters of the neural network for each training example in the batch received by the device. ### Claim 2 (depends on 1) The method of claim 1, wherein the neural network is an image processing neural network and the training examples each include (i) one or more input images and (ii) a ground truth output for the one or more input images. ### Claim 3 (depends on 1) The method of claim 1, wherein each sub-group has the same number of devices and the number of devices in each sub-group is greater than one. ### Claim 4 (depends on 1) The method of claim 1, wherein each batch of training examples has the same number of training examples. ### Claim 5 (depends on 1) The method of claim 1, wherein the network is a dedicated reduction network. ### Claim 6 (depends on 1) The method of claim 1, wherein determining, for each sub-group, a distributed mean and a distributed variance from the per-replica means and the per-replica variances for the devices in the sub-group, comprises: generating the distributed mean by computing an average of the per-replica means for the devices in the sub-group; and generating the distributed variance by computing an average of the per-replica variances for the devices in the sub-group. ### Claim 7 (depends on 1) The method of claim 1, wherein generating, by each of the plurality of devices and using the replica maintained by the device, a respective output of the corresponding other layer for each training example in the batch received by the device comprises: for each training example, processing the training example through each of the layers up to and including the corresponding other layer in the replica of the neural network. ### Claim 8 A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for distributing the training of a neural network across a plurality of devices, wherein the neural network has a plurality of network parameters, wherein the neural network has one or more batch normalization layers, wherein each batch normalization layer applies batch normalization to outputs of a corresponding other layer in the neural network, wherein each device maintains a respective replica of the neural network, and wherein the operations comprise: maintaining data that partitions the plurality of devices into a plurality of sub-groups that each comprise two or more of the plurality of the devices, wherein each of the plurality of devices comprises one or more hardware accelerators for training neural networks; receiving, at each of the plurality of devices, a respective batch of training examples; training the neural network on the respective batches of training examples received by the devices, the training comprising: performing, by each device, a forward pass through the replica maintained by the device to determine a respective network output for each training example in the batch received by the device, comprising, for each batch normalization layer in the neural network: generating, by each of the plurality of devices and using the replica maintained by the device, a respective output of the corresponding other layer for each training example in the batch received by the device, determining, by each of the plurality of devices, a per-replica mean and a per-replica variance of the respective outputs of the corresponding other layer for the training examples in the batch received by the device; determining, for each sub-group, a distributed mean and a distributed variance from the per-replica means and the per-replica variances for the two or more devices in the sub-group and determined from the respective batches received by the two or more devices in the sub-group by performing, over a network that connects the two or more devices in the group using one or more communication links, a distributed reduction operation on the per-replica means and the per-replica variances for the two or more devices in the sub-group and determined from the respective batches received by the two or more devices in the sub-group; and applying, by each device, batch normalization to the respective outputs of the corresponding other layer generated by the device using the distributed mean and the distributed variance for the sub-group to which the device belongs; and performing, by each device, a backward pass through the replica maintained by the device to determine a respective update to the parameters of the neural network for each training example in the batch received by the device. ### Claim 9 (depends on 8) The system of claim 8, wherein the neural network is an image processing neural network and the training examples each include (i) one or more input images and (ii) a ground truth output for the one or more input images. ### Claim 10 (depends on 8) The system of claim 8, wherein each sub-group has the same number of devices and the number of devices in each sub-group is greater than one. ### Claim 11 (depends on 8) The system of claim 8, wherein each batch of training examples has the same number of training examples. ### Claim 12 (depends on 8) The system of claim 8, wherein the network is a dedicated reduction network. ### Claim 13 (depends on 8) The system of claim 8, wherein determining, for each sub-group, a distributed mean and a distributed variance from the per-replica means and the per-replica variances for the devices in the sub-group, comprises: generating the distributed mean by computing an average of the per-replica means for the devices in the sub-group; and generating the distributed variance by computing an average of the per-replica variances for the devices in the sub-group. ### Claim 14 (depends on 8) The system of claim 8, wherein generating, by each of the plurality of devices and using the replica maintained by the device, a respective output of the corresponding other layer for each training example in the batch received by the device comprises: for each training example, processing the training example through each of the layers up to and including the corresponding other layer in the replica of the neural network. ### Claim 15 One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for distributing the training of a neural network across a plurality of devices, wherein the neural network has a plurality of network parameters, wherein the neural network has one or more batch normalization layers, wherein each batch normalization layer applies batch normalization to outputs of a corresponding other layer in the neural network, wherein each device maintains a respective replica of the neural network, and wherein the operations comprise: maintaining data that partitions the plurality of devices into a plurality of sub-groups that each comprise two or more of the plurality of the devices, wherein each of the plurality of devices comprises one or more hardware accelerators for training neural networks; receiving, at each of the plurality of devices, a respective batch of training examples; training the neural network on the respective batches of training examples received by the devices, the training comprising: performing, by each device, a forward pass through the replica maintained by the device to determine a respective network output for each training example in the batch received by the device, comprising, for each batch normalization layer in the neural network: generating, by each of the plurality of devices and using the replica maintained by the device, a respective output of the corresponding other layer for each training example in the batch received by the device, determining, by each of the plurality of devices, a per-replica mean and a per-replica variance of the respective outputs of the corresponding other layer for the training examples in the batch received by the device; determining, for each sub-group, a distributed mean and a distributed variance from the per-replica means and the per-replica variances for the two or more devices in the sub-group and determined from the respective batches received by the two or more devices in the sub-group by performing, over a network that connects the two or more devices in the group using one or more communication links, a distributed reduction operation on the per-replica means and the per-replica variances for the two or more devices in the sub-group and determined from the respective batches received by the two or more devices in the sub-group; and applying, by each device, batch normalization to the respective outputs of the corresponding other layer generated by the device using the distributed mean and the distributed variance for the sub-group to which the device belongs; and performing, by each device, a backward pass through the replica maintained by the device to determine a respective update to the parameters of the neural network for each training example in the batch received by the device. ### Claim 16 (depends on 15) The computer-readable storage media of claim 15, wherein each sub-group has the same number of devices and the number of devices in each sub-group is greater than one. ### Claim 17 (depends on 15) The computer-readable storage media of claim 15, wherein determining, for each sub-group, a distributed mean and a distributed variance from the per-replica means and the per-replica variances for the devices in the sub-group, comprises: generating the distributed mean by computing an average of the per-replica means for the devices in the sub-group; and generating the distributed variance by computing an average of the per-replica variances for the devices in the sub-group. ### Claim 18 (depends on 15) The computer-readable storage media of claim 15, wherein the network is a dedicated reduction network. ### Claim 19 (depends on 15) The computer-readable storage media of claim 15, wherein the neural network is an image processing neural network and the training examples each include (i) one or more input images and (ii) a ground truth output for the one or more input images.