# US 2012/0084238 A1 · System and Method to Enable Training a Machine Learning Network in the Presence of Weak or Absent Training Exemplars

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0084238 A1
- Document ID: US20120084238A1
- Status: Published application
- Kind: A1
- Application number: 13/313,793
- Filed: 2011-12-07
- Published: 2012-04-05
- Earliest claimed: 2007-07-31
- Assignees: Cornell Research Foundation, Inc.
- Inventors: Bruce S. Kristal; Rolf J. Martin
- Web page: https://patentgrep.com/patent/US20120084238A1

## CPC

- G09B 7/06: of the multiple-choice answer-type, i.e. where a given question is provided with a series of answers and a choice has to be made from the answers
- G06Q 10/04: Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" (market predictions or forecasting for commercial activities G06Q30/0202)
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Described is a system and method for training a machine learning network. The method comprises initializing at least one of nodes in a machine learning network and connections between the nodes to a predetermined strength value, wherein the nodes represent factors determining an output of the network, providing a first set of questions to a plurality of users, the first set of questions relating to at least one of the factors, receiving at least one of choices and guesstimates from the users in response to the first set of questions and adjusting the predetermined strength value as a function of the choices/guesstimates. The real and simulated examples presented demonstrate that synthetic training sets derived from expert or non-expert human guesstimates can replace or augment training data sets comprised of actual training exemplars that are too limited in size, scope, or quality to otherwise generate accurate predictions.

## Claims

### Claim at position 1

1-58. (canceled)

### Claim 59

A method, comprising:
initializing at least one of (i) nodes in a first machine learning network and (ii) connections between nodes, to a predetermined strength value, wherein the nodes represent factors determining an output of the network;
providing a first set of questions to a plurality of users, the first set of questions relating to at least one of the factors;
receiving guesstimates from the users in response to the first set of questions; and
adjusting the predetermined strength value as a function of the guesstimates.

### Claim 60 (depends on 59)

The method of claim 59, further comprising applying a weight to the guesstimates of a particular user as a function of a user information including a level of expertise of the user.

### Claim 61 (depends on 59)

The method of claim 59, wherein at least one of the plurality of users has imperfect knowledge.

### Claim 62 (depends on 59)

The method of claim 59, wherein the guesstimates comprise a synthetic data set capable of being applied at any time to the network.

### Claim 63 (depends on 62)

The method of claim 62, further comprising combining the synthetic data set with additional training data derived from a non-human source.

### Claim 64 (depends on 59)

The method of claim 59, further comprising:
after the step of adjusting, providing the network output to a second machine learning network;
providing a second question to a plurality of users of the second network; and
receiving at least one of choices and guesstimates from the users of the second network in response to the second question.

### Claim 65 (depends on 59)

The method of claim 59, further comprising displaying at least one valid guesstimate option to a user who has yet to respond to the question.

### Claim 66 (depends on 59)

The method of claim 59, further comprising receiving explanations from the users regarding the guesstimates.

### Claim 67 (depends on 59)

The method of claim 59, wherein the predetermined strength value is one of a constant and a series of related constants.

### Claim 68 (depends on 59)

The method of claim 59, wherein a question provided to the plurality of users includes one of true/false, yes/no, multiple choice, a ranking inquiry and a rating inquiry.

### Claim 69 (depends on 59)

The method of claim 59, wherein the guesstimates include at least one of true/false, yes/no, a quality level, a probability, a percentage, a rank, a qualitative modifier, a quantitative modifier, an item and a series of items.

### Claim 70 (depends on 69)

The method of claim 69, wherein the users are provided an opportunity to add to a list of possible choices.

### Claim 71 (depends on 59)

The method of claim 59, further comprising generating a training set as a function of the user responses so that the adjusting of the predetermined strength values is performed automatically in accordance with a learning algorithm.

### Claim 72 (depends on 59)

The method of claim 59, further comprising:
receiving a query;
identifying connections between nodes in the network related to the query;
generating a response to the query as a function of strength values of the identified connections; and
outputting the response.

### Claim 73 (depends on 72)

The method of claim 72, wherein the connections include connections within and between input and output node layers.

### Claim 74 (depends on 73)

The method of claim 73, wherein the connections further include connections within and between a hidden node layer and each of the input and output node layers.

### Claim 75 (depends on 72)

The method of claim 72, wherein the users comprise a first user panel and the query is specified by a second user panel.

### Claim 76 (depends on 72)

The method of claim 72, wherein the outputted response is fed as input into a further machine learning network.

### Claim 77 (depends on 72)

The method of claim 72, further comprising:
feeding the response back into the network;
providing at least one of the first set of questions to, and receiving at least one of choices and guesstimates from, a third user panel; and
adjusting the strength value of the nodes as a function of the third user panel's choices/guesstimates and the actual input/output.

### Claim 78 (depends on 72)

The method of claim 72, further comprising:
selecting at least one of the users based on a degree to which the selected user's choice matches the response; and
providing a reward to the selected at least one user.

### Claim 79

A system, comprising:
a host computing device including a machine learning network, the host computing device initializing at least one of i) nodes in the network and ii) connections between the nodes, to a respective predetermined strength value, the host computing device outputting questions, each question corresponding to at least one of the connections; and
a plurality of client computing devices receiving the questions, the client computing devices transmitting guesstimates from users thereof in response to the questions,
wherein the host computing device adjusts the predetermined strength value as a function of the guesstimates.

### Claim 80 (depends on 79)

The system of claim 79, wherein the host computing device applies a weight to the guesstimates of a particular user as a function of a user information including a level of expertise of the user.

### Claim 81 (depends on 79)

The system of claim 79, wherein at least one of the plurality of users has imperfect knowledge.

### Claim 82 (depends on 79)

The system of claim 79, wherein the guesstimates comprise a synthetic data set that is capable of being applied at any time to the network.

### Claim 83 (depends on 79)

The system of claim 79, wherein, when the host computing device receives a query from one of the client computing devices, the host computing device identifies connections between nodes in the network related to the query, generates a response to the query as a function of respective strength values of the identified connections and transmits the response to the one of the client computing devices.

### Claim 84 (depends on 79)

The system of claim 79, wherein the machine learning network executes at least one of the following types of machine learning algorithms, rules, trees and decision strategies: Bayesnet, ComplementNaiveBayes, NaiveBayes, NaiveBayesMultinomial, NaiveBayesSimple, NaiveBayesUpdateable, HillClimber, RepeatedHillClimber, SimulatedAnnealing, TabuSearch, Logistic, SimpleLogistic, MultilayerPerceptron, VotedPerceptron, RBFNetwork, SMO, lazy.IB1, lazy.KStar, AdaBoostM1, ConjunctiveRule, DecisionTable, JRip, NNge, OneR, PART, ZeroR, J48, LMT, NBTree, RandomForest, RandomTree, REPTree, meta.AttributeSelectedClassifier, CfsSubsetEval, BestFirst, meta.Bagging, ClassificationViaRegression, M5P, CVParameterSelection, meta.Decorate, meta. FilteredClassifier, ClassOrder, supervised.attribute.AttributeSelection, Discretize, meta. Grading, meta.MultiScheme, MultiBoostAB, DecisionStump, NominalToBinary, OrdinalClassClassifier, meta.LogitBoost, meta.MultiClassClassifier, meta.RacedIncrementalLogitBoost, meta.RandomCommittee, meta.StackingC, meta.Stacking, meta.Vote, HyperPipes, VFI, Ridor and Hebbian learning.

### Claim 85 (depends on 79)

The system of claim 79, wherein additional nodes and connections are formed in the network in response to the guesstimates.

### Claim 86 (depends on 79)

The system of claim 79, wherein the guesstimates include estimates of plausible ranges for key values, and wherein upper and lower boundaries of the ranges define part of a training set.

### Claim 87 (depends on 79)

The system of claim 79, wherein the guesstimates are combined with output from a further machine learning network to generate a hybrid training set comprising both human and artificial machine learning choices.

### Claim 88 (depends on 87)

The system of claim 87, wherein a plurality of training sets are generated based on values close to those of the hybrid training set.

### Claim 89 (depends on 79)

The system of claim 79, wherein additional user guesstimates are made after examining output from the machine learning network.

### Claim 90 (depends on 79)

The system of claim 79, wherein the network is a consensus network and the users are able to a combine their own choices with those of other users to develop and evaluate the network.

### Claim 91 (depends on 79)

The system of claim 79, wherein the users are able to adjust a weight of their own guesstimates before the combining.

### Claim 92 (depends on 79)

The system of claim 79, wherein the guesstimates are made based on a perceived relevancy of specific input, and wherein the choices generate corresponding connections between nodes in a hidden layer of the network.

### Claim 93

A non-transitory computer-readable medium storing a set of instructions for execution by a processor to perform a method comprising:
initializing at least one of i) nodes in a machine learning network and ii) connections between the nodes, to a predetermined strength value, wherein the nodes represent factors determinative of an output of the network;
providing at least one question to a plurality of users, the at least one question relating to at least one of the factors;
receiving guesstimates from the users in response to the at least one question;
applying a weight to the guesstimates of a particular user as a function of a user information including a level of expertise of the user; and
adjusting the predetermined strength value as a function of the guesstimates.