# US 2021/0241038 A1 · Test and training data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241038 A1
- Document ID: US20210241038A1
- Status: Published application
- Kind: A1
- Application number: 17/049,426
- Filed: 2019-04-25
- Published: 2021-08-05
- Earliest claimed: 2018-04-25
- Assignees: Volume Limited
- Applicants: Volume Limited
- Inventors: Benoit ALVAREZ; Bryn HORSFIELD-SCHONHUT
- Web page: https://patentgrep.com/patent/US20210241038A1

## CPC

- G06Q 10/04: Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" (market predictions or forecasting for commercial activities G06Q30/0202)
- G06F 18/2115: by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/2178: {based on feedback of a supervisor}
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06Q 10/06: Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method, apparatus and computer-readable instructions are described including obtaining a plurality of sets of training and test data packages (wherein: each of the plurality of sets comprises test data and training data, wherein the test data comprises a subset of a first plurality of data items and the training data comprises a remainder of the first plurality of data items; each data item of the first plurality of data items is allocated as test data for only one of the plurality of sets of training and test data packages; and each data item comprises a classification identifier); receiving a new data item having a classification identifier; and adding the new data item to one of the plurality of sets of training and test data packages as test data and adding the new data item to the other of the plurality of sets of training and test data packages as training data, wherein the one of the plurality of sets of training and test data packages to which the new data item is added as test data is selected depending on the classification identifier of the new data item and the classification identifiers of the first plurality of data items.

## Claims

### Claim 1

A method comprising:
obtaining a plurality of sets of training and test data packages, wherein:
each of the plurality of sets comprises test data and training data, wherein the test data comprises a subset of a first plurality of data items and the training data comprises a remainder of the first plurality of data items;
each data item of the first plurality of data items is allocated as test data for only one of the plurality of sets of training and test data packages; and
each data item comprises a classification identifier;
receiving a new data item having a classification identifier; and
adding the new data item to one of the plurality of sets of training and test data packages as test data and adding the new data item to the other of the plurality of sets of training and test data packages as training data, wherein the one of the plurality of sets of training and test data packages to which the new data item is added as test data is selected depending on the classification identifier of the new data item and the classification identifiers of the first plurality of data items.

### Claim 2 (depends on 1)

A method as claimed in claim 1, wherein selecting said one of the plurality of sets of training and test data packages to which the new data item is added as test data comprises identifying the set of training and test data packages of the plurality having test data having the fewest number of data items having a classification identifier in common with the classification identifier of the new data item.

### Claim 3 (depends on 1)

A method as claimed in claim 1, wherein selecting said one of the plurality of sets of training and test data packages to which the new data item is added as test data comprises identifying the set of training and test data packages of the plurality having test data having the fewest number of data items.

### Claim 4 (depends on 1)

A method as claimed in claim 1, wherein selecting said one of the plurality of sets of training and test data packages to which the new data item is added as test data comprises:
identifying one or more set(s) of training and test data packages having test data having the fewest number of data items having a classification identifier in common with the classification identifier of the new data item; and
in the event that two or more of the sets of training and test data packages have test data having equal fewest data items having a classification identifier in common with the classification identifier of the new data item, identifying the set of training and test data packages of said two or more of the sets of training and test data packages having test data having the fewest number of data items.

### Claim 5 (depends on 1)

A method as claimed in claim 1, wherein selecting said one of the plurality of sets of training and test data packages comprises an arbitrary or random selection.

### Claim 6 (depends on 1)

A method as claimed in claim 1, further comprising, for any of the first plurality of data items that is deleted from said first plurality of data items, deleting said data item from each of the plurality of sets of training and test data packages.

### Claim 7 (depends on 6)

A method as claimed in claim 6, wherein in the event the first plurality of data items is modified by deleting one or more first old data items and adding one or more second new data items, the method comprising deleting said first old data item(s) from each of the plurality of sets of training and test data packages before adding the second new data item(s) to any of the plurality of sets of training and test data packages.

### Claim 8 (depends on 1)

A method as claimed in claim 1, wherein the classification identifier of a data item corresponds to an intended use of said data item.

### Claim 9 (depends on 1)

A method as claimed in claim 1, wherein obtaining the plurality of sets of training and test data packages comprises generating said plurality of sets of training and test data packages.

### Claim 10 (depends on 9)

A method as claimed in claim 9, wherein generating each training and test data package of the plurality comprises randomly selecting said subset of test data items from the plurality of data items for each of the plurality of sets of training and test data packages, subject to defined rules.

### Claim 11 (depends on 10)

A method as claimed in claim 10, wherein said defined rules require that data items are distributed amongst the test data of the plurality of sets of test and training data such that the number of data items having each classification identifier is evenly distributed amongst the plurality of sets of test and training data.

### Claim 12 (depends on 1)

A method as claimed in claim 1, wherein obtaining the plurality of sets of training and test data packages comprises receiving or retrieving said plurality of sets of training and test data packages.

### Claim 13 (depends on 1)

A method as claimed in claim 1, wherein:
the training data comprises training data for a model; and/or
the test data comprises test data for a model.

### Claim 14

A method comprising:
generating a plurality of sets of training and test data packages using a method as claimed in any one of the preceding claims;
training a model using training data of a selected one of said training and test data packages; and
generating a performance measurement for the trained model by applying the test data of the selected training and test data packages to the trained model.

### Claim 15 (depends on 1)

Computer readable instructions, which, when executed by a computing apparatus, cause the method of claim 1 to be performed.

### Claim 16

An apparatus comprising:
means for obtaining a plurality of sets of training and test data packages, wherein: each of the plurality of sets comprises test data and training data, wherein the test data comprises a subset of a first plurality of data items and the training data comprises a remainder of the first plurality of data items; each data item of the first plurality of data items is allocated as test data for only one of the plurality of sets of training and test data packages; and each data item comprises a classification identifier;
means for receiving a new data item having a classification identifier; and
means for adding the new data item to one of the plurality of sets of training and test data packages as test data and adding the new data item to the other of the plurality of sets of training and test data packages as training data, wherein the one of the plurality of sets of training and test data packages to which the new data item is added as test data is selected depending on the classification identifier of the new data item and the classification identifiers of the first plurality of data items.

### Claim 17 (depends on 16)

An apparatus as claimed in claim 16, wherein the means comprise:
at least one processor; and
at least one memory including computer program code configured to, with the at least one processor, cause the performance of the apparatus.

### Claim 18 (depends on 1)

Apparatus configured to carry out the method of claim 1.
