# US 12,112,129 B2 · Method and apparatus for decentralized supervised learning in NLP applications

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,129 B2
- Document ID: US12112129B2
- Status: Granted
- Kind: B2
- Application number: 17/527,167
- Filed: 2021-11-16
- Published: 2022-06-09
- Granted: 2024-10-08
- Earliest claimed: 2020-12-03
- Assignees: Fujitsu Limited
- Applicants: Fujitsu Limited
- Inventors: Nuria Garcia Santa; Kendrick Cetina
- Web page: https://patentgrep.com/patent/US12112129B2

## CPC

- G06F 40/226: Validation
- G06F 40/30: Semantic analysis
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 21/6254: {by anonymising data, e.g. decorrelating personal data from the owner's identification}
- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/098: Distributed learning, e.g. federated learning
- G10L 15/063: {Training}
- G10L 15/075: {supervised, i.e. under machine guidance}
- G10L 15/16: using artificial neural networks
- G10L 15/18: using natural language modelling
- G06F 40/279: Recognition of textual entities
- G06F 40/295: Named entity recognition
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 15/183: using context dependencies, e.g. language models
- G10L 2015/0635: {updating or merging of old and new templates; Mean values; Weighting}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A method of training a neural network as a natural language processing, NLP, model, comprises: inputting annotated training data to first architecture portions of the neural network, the first architecture portions being executed respectively in a plurality of distributed client computing devices in communication with a server computing device, the training data being derived from text data private to the client computing device in which the first architecture portion is executed, the server computing device having no access to any of the private text data; deriving from the training data, using the first architecture portions, weight matrices of numeric weights which are decoupled from the private text data; concatenating the weight matrices, in a second architecture portion of the neural network executed in the server computing device, to obtain a single concatenated weight matrix; and training, on the second architecture portion, the NLP model using the concatenated weight matrix.

## Claims

### Claim 1

A method of training a neural network as a natural language processing, NLP, model, the method comprising:
inputting respective sets of annotated training data to a plurality of first architecture portions of the neural network, which first architecture portions are executed in respective client computing devices of a plurality of distributed client computing devices in communication with a server computing device, wherein each set of training data is derived from a set of text data which is private to the client computing device in which the first architecture portion is executed, the server computing device having no access to any of the private text data sets, and all sets of training data share a common encoding;
deriving from the sets of annotated training data, using the first architecture portions, respective weight matrices of numeric weights which are decoupled from the private text data sets;
concatenating, in a second architecture portion of the neural network which is executed in the server computing device, the weight matrices received from the client computing devices to obtain a single concatenated weight matrix; and
training, on the second architecture portion, the NLP model using the concatenated weight matrix;
wherein the sets of training data in the common encoding are derived by pre-processing private sets of text data in respective client computing devices by:
carrying out on the set of text data in each client computing device a vocabulary codification process to ensure a common vocabulary codification amongst all the training data to be provided by the client computing devices, wherein in the vocabulary codification process a common alphanumeric character-level representation is established for the vocabulary that uses characters decided by the server computing device, and
using the predefined common character-level representations and predefined common setting parameters, carrying out in each client computing device a word embedding process in which the text data is mapped to vectors of real numbers.

### Claim 2

Apparatus for training a neural network as a natural language processing, NLP, model, the apparatus comprising:
a plurality of distributed client computing devices to execute respectively a plurality of first architecture portions of the neural network, wherein each first architecture portion receives a set of annotated training data derived from a set of text data which is private to the client computing device in which the first architecture portion is executed, all sets of training data sharing a common encoding; and
a server computing device in communication with each of the client computing devices of the plurality, the server computing device to execute a second architecture portion of the neural network, the server computing device having no access to any of the private text data sets;
wherein:
the first architecture portions derive, from the sets of annotated training data, respective weight matrices of numeric weights which are decoupled from the private text data sets, and
the weight matrices received from the client computing devices are concatenated in the second architecture portion to obtain a single concatenated weight matrix, the NLP model being trained on the second architecture portion using the concatenated weight matrix; and
wherein the client computing devices are operable to derive respective sets of training data in the common encoding by pre-processing respective private sets of text data by:
carrying out on the set of text data in each client computing device a vocabulary codification process to ensure a common vocabulary codification amongst all the training data to be provided by the client computing devices, wherein in the vocabulary codification process a common alphanumeric character-level representation is established for the vocabulary that uses characters decided by the server computing device, and
using the predefined common character-level representations and predefined common setting parameters, carrying out in each client computing device a word embedding process in which the text data is mapped to vectors of real numbers.