# US 12,770,000 B2 · Vertical federated learning with secure aggregation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,770,000 B2 - Document ID: US12770000B2 - Status: Granted - Kind: B2 - Application number: 17/838,445 - Filed: 2022-06-13 - Published: 2023-12-14 - Granted: 2026-10-20 - Earliest claimed: 2022-06-13 - Assignees: International Business Machines Corporation; Rensselaer Polytechnic Institute - Applicants: International Business Machines Corporation; Rensselaer Polytechnic Institute - Inventors: Shiqiang Wang; Timothy John Castiglia; Nathalie Baracaldo Angel; Stacy Elizabeth Patterson; Runhua Xu; Yi Zhou - Web page: https://patentgrep.com/patent/US12770000B2 ## CPC - G06N 3/08: Learning methods - G06N 3/04: Architecture, e.g. interconnection topology - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract The method provides for analyzing input and output connections of layers of a received neural network model configured for vertical federated learning. An undirected graph of nodes is generated in which a node having two or more child nodes includes an aggregation operation, based on the analysis of the model in which a model output corresponds to a node of the graph. A layer of the model is identified in which a sum of lower layer outputs are computed. The identified model layer is partitioned into a first part applied respectively to the multiple entities and a second part applied as an aggregator of the output of the first part. The aggregation operation is performed between pairs of lower layer outputs, and multiple forward and backward passes of the neural network model are performed that include secure aggregation and maintain model partitioning in forward and backward passes. ## Claims ### Claim 1 A method for training a neural network model using vertical federated learning, the method comprising: analyzing an architecture of input and output connections of layers of a neural network model that are received, wherein the neural network model is structured to receive input from vertically partitioned data across multiple entities; generating an undirected graph of nodes in which a node having two or more child nodes includes an aggregation operation, based on the analysis of the architecture of the neural network model in which an output of a layer of the neural network model corresponds to a node of the graph; identifying the layer of the neural network model in which a sum of lower layer outputs is computed; partitioning the identified neural network model layer into a first part applied respectively to the multiple entities and a second part applied to an aggregator; adding a noise term to the lower layer outputs of at least a pair of entity inputs of the multiple entities, which are configured as inputs to an aggregation operation performed by a first part partition of the neural network model, wherein generation of the noise term includes coordination between the at least two lower layer outputs through a secure communication channel such that the noise terms cancel out as a result of computing a sum of all noise terms; performing the aggregation operation between at least two of the lower layer outputs; and performing multiple forward and backward passes of the neural network model including application of secure aggregation in the multiple forward and backward passes, respectively. ### Claim 2 (depends on 1) The method of claim 1, further comprising: receiving the lower layer outputs of a neural network model partition applied to a first entity and a second entity as inputs to a first-level aggregation operation that results in a first-level output; receiving the first-level output and output of a third entity as input to a second-level aggregation operation that results in a second-level output; and receiving the second-level output and output of a fourth entity as input to a third-level aggregation operation. ### Claim 3 (depends on 1) The method of claim 1, wherein the first part partition of the partitioned neural network model performs a multiplication of respective outputs of the multiple entities with a weight matrix portion associated respectively with the multiple entities. ### Claim 4 (depends on 1) The method of claim 1, wherein a second part of the partitioned neural network model computes a sum of a respective entity output multiplication results followed by an activation function. ### Claim 5 (depends on 1) The method of claim 1, wherein the multiple forward and backward passes of the neural network model preserve the partitioning of the neural network model by computing the multiple forward and backward passes in a distributed manner across the multiple entities. ### Claim 6 (depends on 1) The method of claim 1, wherein the partitioning of the neural network model includes identifying a layer of the neural network model that performs a sum of outputs of a preceding layer. ### Claim 7 (depends on 1) The method of claim 1, where the partitioning of the neural network model identifies a node as a single building block that sums outputs of preceding building blocks, and initial preceding building blocks correspond to the multiple entities. ### Claim 8 (depends on 1) The method of claim 1, where the partitioning is performed automatically and includes decomposing an identified single building block so that the first part partition of a partition of the neural network model includes computing a partial matrix product on a preceding building block output of each entity of the multiple entities, and the second part of the partition of the neural network model includes computing a sum of a result from the first part followed by additional computation of an activation function. ### Claim 9 A computer system for training a neural network model using vertical federated learning, the computer system comprising: one or more computer processors; at least one computer-readable storage medium; program instructions stored on the at least one computer-readable storage medium, the program instructions comprising: program instructions to analyze an architecture of input and output connections of layers of a neural network model that are received, wherein the neural network model is structured to receive input from vertically partitioned data across multiple entities; program instructions to generate an undirected graph of nodes in which a node having two or more child nodes includes an aggregation operation, based on the analysis of the architecture of the neural network model in which an output of a layer of the neural network model corresponds to a node of the graph; program instructions to identify the layer of the neural network model in which a sum of lower layer outputs is computed; program instructions to partition the identified neural network model layer into a first part applied respectively to the multiple entities and a second part applied to an aggregator; program instructions to add a noise term to the lower layer outputs of at least a pair of entity inputs of the multiple entities, which are configured as inputs to an aggregation operation performed by a first part partition of the neural network model, wherein generation of the noise term includes coordination between the at least two lower layer outputs through a secure communication channel such that the noise terms cancel out as a result of computing a sum of all noise terms; program instructions to perform the aggregation operation between at least two of the lower layer outputs; and program instructions to perform multiple forward and backward passes of the neural network model including application of secure aggregation in the multiple forward and backward passes, respectively. ### Claim 10 (depends on 9) The computer system of claim 9, further comprising: program instructions to receive the lower layer outputs of a neural network model partition applied to a first entity and a second entity as inputs to a first-level aggregation operation that results in a first-level output; program instructions to receive the first-level output and output of a third entity as input to a second-level aggregation operation that results in a second-level output; and program instructions to receive the second-level output and output of a fourth entity as input to a third-level aggregation operation. ### Claim 11 (depends on 9) The computer system of claim 9, wherein program instructions for the first part partition of the partitioned neural network model perform a multiplication of respective outputs of the multiple entities with a weight matrix portion associated respectively with the multiple entities. ### Claim 12 (depends on 9) The computer system of claim 9, wherein the program instructions to perform multiple forward and backward passes of the neural network model preserve the partitioning of the neural network model by computing the multiple forward and backward passes in a distributed manner across the multiple entities. ### Claim 13 (depends on 9) The computer system of claim 9, wherein the program instructions to perform the partitioning of the neural network model identifies a node as a single building block that sums outputs of preceding building blocks, and initial preceding building blocks correspond to the multiple entities. ### Claim 14 (depends on 9) The computer system of claim 9, wherein the program instructions to perform the partitioning of the neural network model are performed automatically and include decomposing an identified single building block so that the first part partition of a partition of the neural network model includes computing a partial matrix product on a preceding building block output of each entity of the multiple entities, and the second part of the partition of the neural network model includes computing a sum of a result from the first part followed by additional computation of an activation function. ### Claim 15 A computer program product for training a neural network model using vertical federated learning, the computer system comprising: at least one computer-readable storage medium; and program instructions stored on the at least one computer-readable storage medium, the program instructions comprising: program instructions to analyze an architecture of input and output connections of layers of a neural network model that is received, wherein the neural network model is structured to receive input from vertically partitioned data across multiple entities; program instructions to generate an undirected graph of nodes in which a node having two or more child nodes includes an aggregation operation, based on the analysis of the architecture of the neural network model in which an output of a layer of the neural network model corresponds to a node of the graph; program instructions to identify the layer of the neural network model in which a sum of lower layer outputs is computed; program instructions to partition the identified neural network model layer into a first part applied respectively to the multiple entities and a second part applied to an aggregator; program instructions to add a noise term to the lower layer outputs of at least a pair of entity inputs of the multiple entities, which are configured as inputs to an aggregation operation performed by a first part partition of the neural network model, wherein generation of the noise term includes coordination between the at least two lower layer outputs through a secure communication channel such that the noise terms cancel out as a result of computing a sum of all noise terms; program instructions to perform the aggregation operation between at least two of the lower layer outputs; and program instructions to perform multiple forward and backward passes of the neural network model including application of secure aggregation in the multiple forward and backward passes, respectively. ### Claim 16 (depends on 15) The computer program product of claim 15, further comprising: program instructions to receive the lower layer outputs of a neural network model partition applied to a first entity and a second entity as inputs to a first-level aggregation operation that results in a first-level output; program instructions to receive the first-level output and output of a third entity as input to a second-level aggregation operation that results in a second-level output; and program instructions to receive the second-level output and output of a fourth entity as input to a third-level aggregation operation. ### Claim 17 (depends on 15) The computer program product of claim 15, wherein the program instructions to perform multiple forward and backward passes of the neural network model preserve the partitioning of the neural network model by computing the multiple forward and backward passes in a distributed manner across the multiple entities.