# US 2021/0241037 A1 · Data processing apparatus and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241037 A1
- Document ID: US20210241037A1
- Status: Published application
- Kind: A1
- Application number: 16/919,329
- Filed: 2020-07-02
- Published: 2021-08-05
- Earliest claimed: 2020-01-30
- Assignees: Canon Medical Systems Corporation
- Applicants: Canon Medical Systems Corporation
- Inventors: Aneta LISOWSKA
- Web page: https://patentgrep.com/patent/US20210241037A1

## CPC

- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G06F 18/211: Selection of the most significant subset of features
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06F 18/2178: {based on feedback of a supervisor}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/091: Active learning
- G06N 20/00: Machine learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G16H 30/40: for processing medical images, e.g. editing
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A data processing apparatus for training models on data, comprises processing circuitry configured to: train a first model on a plurality of labelled data sets; apply the first trained model to a plurality of non-labelled data sets to obtain first pseudo-labels; train a second model using at least the labelled data sets, the non-labelled data sets and the first pseudo-labels; apply the second trained model to non-labelled data sets to obtain second pseudo-labels; and train a third model based on at least the labelled data sets, non-labelled data sets and the second pseudo-labels.

## Claims

### Claim 1

A data processing apparatus for training models on data, comprising processing circuitry configured to:
train a first model on a plurality of labelled data sets;
apply the first trained model to a plurality of non-labelled data sets to obtain first pseudo-labels;
train a second model using at least the labelled data sets, the non-labelled data sets and the first pseudo-labels;
apply the second trained model to non-labelled data sets to obtain second pseudo-labels; and
train a third model based on at least the labelled data sets, non-labelled data sets and the second pseudo-labels.

### Claim 2 (depends on 1)

Apparatus according to claim 1, wherein the processing circuitry is configured to provide an output identifying data sets for labelling by a user and/or identifying at least some of the pseudo-labels for verification or modification by a user.

### Claim 3 (depends on 1)

Apparatus according to claim 1, wherein the processing circuitry is configured to perform a series of training and labelling processes, thereby increasing the amount of the data that is labelled or pseudo-labelled and/or increasing an accuracy of the labelling and/or pseudo-labelling and/or increasing an accuracy of model output.

### Claim 4 (depends on 3)

Apparatus according to claim 3, wherein the series of labelling processes comprise automatically pseudo-labelling data and/or labelling based on user input.

### Claim 5 (depends on 3)

Apparatus according to claim 3, wherein the series of training and labelling processes comprises the training and applying of the first model, the training and applying of the second model, the training of the third model, an applying of the third model, and a training and applying of at least one further model such that N models are training and applied, where N is an integer.

### Claim 6 (depends on 5)

Apparatus according to claim 5, wherein at least one of: N is greater than 2, N is greater than 3, N is between 3 and 20.

### Claim 7 (depends on 1)

Apparatus according to claim 1, wherein the number of labelled data sets is at least one of: less than 50% of the number of said unlabelled data sets, less than 10% of the number of said unlabelled data sets, less than 1% of the number of said unlabelled data sets.

### Claim 8 (depends on 1)

Apparatus according to claim 1, wherein at least one of:
a) the number of unlabelled data sets is at least one of: greater than 50, greater than 100, greater than 1000;
b) the number of labelled data sets is at least one of: greater than 1, between 1 and 1000, or between 1 and 100.

### Claim 9 (depends on 3)

Apparatus according to claim 3, wherein the series of labelling and training processes is terminated in response to an output accuracy, a desired or predicted performance, an amount of labelled data, a number of iterations reaching a threshold value, or there being no or less than a threshold amount of improvement in comparison to a previous process in the series.

### Claim 10 (depends on 1)

An apparatus according to claim 1, wherein the first, second and third models make up or form part of a series of models that are trained and applied, and the processing circuitry is configured to apply a final trained model of the series to a data set to obtain an output.

### Claim 11 (depends on 10)

An apparatus according to claim 10, wherein the data set comprises a medical imaging data set and the output comprises or represent a classification and/or a segmentation and/or an identification of an anatomical feature or pathology.

### Claim 12 (depends on 1)

An apparatus according to claim 1, wherein at least some of the training is based on a combination of classification and uncertainty minimisation.

### Claim 13 (depends on 12)

An apparatus according to claim 12, wherein at least some of the training is based on determination of classification loss value(s) for the labelled data sets and determination of uncertainty minimisation loss value(s) for the unlabelled data sets and/or the labelled data sets alone or in combination.

### Claim 14 (depends on 12)

An apparatus according to claim 12, wherein the uncertainty minimisation comprises estimating uncertainty using a dropout layer of one or more of the models.

### Claim 15 (depends on 1)

An apparatus according to claim 1, wherein the processing circuitry is configured to determine a measure of uncertainty based on differences between predictions or other outputs of the models.

### Claim 16 (depends on 1)

An apparatus according to claim 1, wherein the data comprises medical imaging data or text data.

### Claim 17 (depends on 1)

An apparatus according to claim 1, wherein the data sets comprise at least one magnetic resonance (MR) data sets, computed tomography (CT) data sets, X-ray data sets, ultrasound data sets, positron emission tomography (PET) data sets, single photon emission computed tomography (SPECT) data sets, or patient record data sets.

### Claim 18 (depends on 1)

An apparatus according to claim 1, wherein labels of the labelled sub-set(s) of data comprise or represent a classification and/or a segmentation and/or an identification of an anatomical feature or pathology.

### Claim 19

A method of training models on data, comprising:
training a first model on a plurality of labelled data sets;
applying the first trained model to data plurality of non-labelled data sets to obtain first pseudo-labels;
training a second model using at least the labelled data sets, the non-labelled data sets and the first pseudo-labels;
applying the second trained model to non-labelled data sets to obtain second pseudo-labels; and
training a third model based on at least the labelled data sets, non-labelled data sets and the second pseudo-labels.

### Claim 20 (depends on 10)

A method of processing data comprising applying a final model trained using an apparatus according to claim 10 to a data set thereby to obtain an output.