# US 2023/0410818 A1 · Method and system for personalising machine learning models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410818 A1
- Document ID: US20230410818A1
- Status: Published application
- Kind: A1
- Application number: 18/220,567
- Filed: 2023-07-11
- Published: 2023-12-21
- Earliest claimed: 2022-06-17
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Alberto Gil C. P. RAMOS; Abhinav MEHROTRA; Sourav BHATTACHARYA
- Web page: https://patentgrep.com/patent/US20230410818A1

## CPC

- G10L 17/18: Artificial neural networks; Connectionist approaches
- G10L 21/0264: characterised by the type of parameter measurement, e.g. correlation techniques, zero crossing techniques or predictive techniques
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G10L 17/04: Training, enrolment or model building
- G10L 21/0208: Noise filtering
- G10L 13/02: Methods for producing synthetic speech; Speech synthesisers
- G10L 13/033: Voice editing, e.g. manipulating the voice of the synthesiser
- G10L 15/063: {Training}
- G10L 15/075: {supervised, i.e. under machine guidance}
- G10L 15/16: using artificial neural networks
- G10L 17/02: Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Broadly speaking, embodiments of the present techniques provide a method and system for personalising machine learning models on resource-constrained devices by using conditional neural networks. In particular, the present techniques allow for resource-efficient use of a conditioning vector by incorporating the conditioning vector into weights learned during training. This reduces the computational resources required at inference time.

## Claims

### Claim 1

A computer-implemented method for training, on a server, a machine learning, ML, model having a first neural network and a second neural network, the method comprising:
obtaining a training dataset comprising a plurality of pairs of data items to be processed by the ML model, each pair of data items comprising a conditioning vector that represents an individual user and an input data item; and
training the ML model by jointly:
training the first neural network of the ML model to transform weights of at least one layer of the second neural network into personalized weights for individual users using the conditioning vector of the pairs of data items; and
training the second neural network of the ML model to process the input data item of the pairs of data items using the personalized weights for the at least one layer, and to generate output data.

### Claim 2 (depends on 1)

The method as claimed in claim 1, wherein training the ML model comprises training the first neural network to generate personalized weights that cause the second neural network to generate output data that best matches a ground truth.

### Claim 3 (depends on 1)

The method as claimed in claim 1, wherein weights of the first neural network and second neural network are randomly initialized.

### Claim 4 (depends on 1)

The method as claimed in claim 1, wherein training the first neural network comprises training the first neural network to learn a function that transforms the weights to personalized weights.

### Claim 5 (depends on 1)

The method as claimed in claim 1, further comprising receiving at least one constraint, and wherein training the ML model further comprises using the received at least one constraint to train the first and second neural networks to generate output data.

### Claim 6 (depends on 5)

The method as claimed in claim 5, wherein the at least one constraint is any one or more of: a hardware constraint, a computational capability, an inference latency, and an inference energy consumption.

### Claim 7 (depends on 5)

The method as claimed in claim 5, wherein using the received at least one constraint to train the ML model to generate output data comprises restricting values of the personalized weights to be within a predefined range.

### Claim 8 (depends on 1)

The method as claimed in claim 1, wherein:
the ML model is for sound enhancement and/or automatic speech recognition;
the input data item of each pair of data items comprises at least one audio noise sample and at least one clean audio sample that each contain speech of individual speakers; and
the conditioning vector is a speaker embedding vector for each individual speaker.

### Claim 9 (depends on 8)

The method as claimed in claim 8, wherein training the ML model comprises training the second neural network to remove the noise from the corrupted audio samples while maintaining the speech of the individual speakers.

### Claim 10 (depends on 8)

The method as claimed in claim 8, further comprising generating the speaker embedding vector for each individual speaker by:
obtaining at least one clean audio signal for an individual speaker; and
applying a speaker recognition model to the at least one clean audio signal to generate a speaker embedding vector representing a speech profile of the individual speaker.

### Claim 11 (depends on 1)

The method as claimed in claim 1, wherein:
the ML model is for text-to-speech conversion;
the input data item of the pairs of data items comprises at least one text sample and a corresponding audio sample; and
the conditioning vector is an embedding vector representing a target voice for the speech.

### Claim 12

A server for training a machine learning, ML, model having a first neural network and a second neural network, the server comprising:
at least one processor coupled to memory and arranged to:
obtain a training dataset comprising a plurality of pairs of data items to be processed by the ML model, each pair of data items comprising a conditioning vector that represents an individual user and an input data item; and
train the ML model by jointly:
training the first neural network of the ML model to transform weights of at least one layer of the second neural network into personalized weights for individual users using the conditioning vector of the pairs of data items; and
training the second neural network of the ML model to process the input data item of the pairs of data items using the personalized weights for the at least one layer, and to generate output data.
