# US 2021/0174130 A1 · System, Method, and Platform for Auto Machine Learning Via Optimal Hybrid AI Formulation from Crowd

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174130 A1
- Document ID: US20210174130A1
- Status: Published application
- Kind: A1
- Application number: 16/703,430
- Filed: 2019-12-04
- Published: 2021-06-10
- Earliest claimed: 2019-12-04
- Assignees: AT&T Intellectual Property I, L.P.
- Applicants: AT&T Intellectual Property I, L.P.
- Inventors: Mark Austin; ABHAY DABHOLKAR; VIJAYAN NAGARAJAN; Lauren Savage; Joshua Whitney
- Web page: https://patentgrep.com/patent/US20210174130A1

## CPC

- G06N 20/20: Ensemble learning
- G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/25: Fusion techniques
- G06F 18/251: {of input or preprocessed data}
- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G06V 10/96: Management of image or video recognition tasks
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Aspects of the subject disclosure may include, for example, receiving a plurality of proposed machine learning solutions to a machine learning problem including receiving, for each respective proposed machine learning solution of the plurality of proposed machine learning solutions, one or more of a machine learning model, a dataset and a data pipeline output; automatically determining hybrid solutions to the machine learning problem, including combining, by the processing system, at least one of a first component from a first proposed machine learning solution with at least one of a second component from a second proposed machine learning solution; and ranking the hybrid solutions including determining a log loss score for each hybrid solution and sorting the hybrid solutions according to the log loss score for each hybrid solution. Other embodiments are disclosed.

## Claims

### Claim 1

A device, comprising:
a processing system including a processor; and
a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations, the operations comprising:
receiving information defining a machine learning problem to be solved;
developing an initial machine learning solution to the machine learning problem;
scoring the initial machine learning solution;
receiving information about a plurality of other machine learning solutions to the machine learning problem, each respective machine learning solution of the plurality of machine learning solutions comprising two or more components;
scoring each respective machine learning solution of the plurality of machine learning solutions to determine respective scores;
ranking the respective machine learning solutions and the initial machine learning solution;
developing one or more hybrid machine learning solutions including at least one component of the initial machine learning solution and at least one component of the plurality of machine learning solutions;
scoring the one or more hybrid machine learning solutions;
ranking the one or more hybrid machine learning solutions with the respective machine learning solutions and the initial machine learning solution to produce a leaderboard ranking;
providing a human readable indication of the leaderboard ranking; and
receiving a human input, the human input indicating a selected machine learning solution among the one or more hybrid machine learning solutions, the respective machine learning solutions, the initial machine learning solution.

### Claim 2 (depends on 1)

The device of claim 1, wherein the operations further comprise:
selecting a machine learning solution of the plurality of machine learning solutions;
selecting a component of the selected machine learning solution; and
developing a hybrid machine learning solution by combining the component of the selected machine learning solution with components of the initial machine learning solution.

### Claim 3 (depends on 2)

The device of claim 2, wherein the selecting the component of the selected machine learning solution comprises selecting one of a data set, feature set, a machine learning algorithm, a parameter set, and a validation technique of the selected machine learning solution.

### Claim 4 (depends on 2)

The device of claim 2, wherein the operations further comprise:
scoring the hybrid machine learning solution; and
comparing a score for the hybrid machine learning solution with scores for the respective machine learning solutions and a score for the initial machine learning solution to produce the leaderboard ranking.

### Claim 5 (depends on 2)

The device of claim 2, wherein the operations further comprise:
iteratively selecting each machine learning solution of the plurality of machine learning solutions;
iteratively selecting each component of the selected machine learning solution;
developing the hybrid machine learning solution by combining the selected component of the selected machine learning solution with components of the initial machine learning solution or other components of one or more other machine learning solutions of the plurality of machine learning solutions; and
scoring the hybrid machine learning solution.

### Claim 6 (depends on 1)

The device of claim 1, wherein the operations further comprise:
conducting a fraud detection competition among the one or more hybrid machine learning solutions, wherein the fraud detection conducting the competition comprises:
providing, by an automatic process forming a judge of the fraud detection competition, authentication event data to a competitor, wherein the competitor is a hybrid machine learning solution of the one or more hybrid machine learning solutions;
providing. by the automatic process, transaction event data to the competitor, wherein the transaction event data comprises information about a potentially fraudulent transaction;
pausing, by the automatic process, a predetermined amount of time, wherein the pausing comprises suspending provision of additional authentication event data and additional transaction event data to the competitor during the predetermined amount of time to prevent the competitor from receiving future data;
receiving, by the automatic process, a fraud risk score from the competitor;
scoring, by the automatic process, the competitor based at least in part on the fraud risk score; and
if the fraud risk score is not received from the competitor within the predetermined amount of time, penalizing, by the automatic process, the competitor in the fraud detection competition.

### Claim 7 (depends on 1)

The device of claim 1, wherein the operations further comprise:
receiving the information defining a machine learning problem to be solved from a user;
providing at least some of the information defining a machine learning problem to be solved to a plurality of other users;
receiving from the other users information about a plurality of other machine learning solutions to the machine learning problem, wherein the plurality of other machine learning solutions are developed by the other users;
providing to the user and the other users the human readable indication of the leaderboard ranking; and
receiving from the user the human input to select a machine learning solution satisfying requirements of the user.

### Claim 8 (depends on 1)

The device of claim 1, wherein the operations further comprise:
receiving, from the other users, information about revised machine learning solutions, wherein the revised machine learning solutions are based on the human readable indication of the leaderboard ranking;
developing one or more new hybrid machine learning solutions based on the information about revised machine learning solutions;
scoring the one or more new hybrid machine learning solutions;
ranking the one or more new hybrid machine learning solutions to produce an updated leaderboard ranking; and
providing a human readable indication of the updated leaderboard ranking.

### Claim 9

A machine-readable medium, comprising executable instructions that, when executed by a processing system including a processor, facilitate performance of operations, the operations comprising:
providing information about a machine learning problem to be solved, the providing comprising presenting the information about the machine learning problem to be solved to a web accessible interface;
receiving, from a plurality of users, information about proposed solutions to the machine learning problem, wherein the receiving comprises receiving the information about the proposed solutions from the web accessible interface;
ranking the proposed solutions, wherein the ranking comprises sorting the proposed solutions according to a logarithmic loss determination for each respective proposed solution of the proposed solutions;
forming a plurality of hybrid solutions, wherein the forming comprises selecting one or more first components of a first respective proposed solution and one or more second components of a second respective proposed solution and combining the one or more first components with the one or more second components to form a hybrid solution;
ranking the plurality of hybrid solutions with the proposed solutions, wherein the ranking comprises sorting the plurality of hybrid solutions with the proposed solutions according to a logarithmic loss determination for each respective hybrid solution and for each respective proposed solution of the proposed solutions;
providing information about the ranking of the plurality of hybrid solutions with the proposed solutions; and
receiving a user selection of a selected machine learning solution of the plurality of hybrid solutions and the proposed solutions, wherein the user selection corresponds to a selected machine learning solution that satisfies requirements of the user as selected by the user.

### Claim 10 (depends on 9)

The machine-readable medium of claim 9, wherein the operations further comprise:
receiving information about the machine learning problem to be solved, wherein the receiving comprises receiving information about a business problem, an initial dataset associated with the problem, and a scoring method.

### Claim 11 (depends on 9)

The machine-readable medium of claim 9, wherein the receiving the information about proposed solutions to the machine learning problem comprises:
receiving one or more proposed solutions from other users, wherein the receiving comprises receiving machine learning solutions, wherein the machine learning solutions are designed by the other users to solve the machine learning problem.

### Claim 12 (depends on 11)

The machine-readable medium of claim 11, wherein the receiving the machine learning solutions comprises:
receiving, for each machine learning solution of the machine learning solutions, a plurality of machine learning model components, wherein the receiving the plurality of machine learning components comprises receiving one or more of a machine learning model, a dataset and a data pipeline output.

### Claim 13 (depends on 12)

The machine-readable medium of claim 12, wherein the forming a plurality of hybrid solutions comprises:
combining one or more of a first machine learning model, a first dataset and a first data pipeline output of a first proposed solution with one or more of a second machine learning model, a second dataset and a second data pipeline output of a second proposed solution to form a hybrid solution.

### Claim 14 (depends on 12)

The machine-readable medium of claim 12, wherein the operations further comprise:
providing training data to the machine learning model of the each machine learning solution to train the machine learning model; and
providing test data to the machine learning model to characterize performance of the machine learning model according to the logarithmic loss.

### Claim 15 (depends on 9)

The machine-readable medium of claim 9, wherein the operations further comprise:
automatically determining one or more components of each received machine learning solution, wherein the automatically determining comprises identifying for each respective received machine learning solution one or more of a machine learning model, a dataset and a data pipeline output.

### Claim 16

A method, comprising:
receiving, by a processing system including a processor, a plurality of proposed machine learning solutions to a machine learning problem, wherein the receiving comprises receiving, by the processing system, for each respective proposed machine learning solution of the plurality of proposed machine learning solutions, one or more of a machine learning model, a dataset and a data pipeline output;
determining, by the processing system, hybrid solutions to the machine learning problem, wherein the determining comprises, combining, by the processing system, at least one of a first component from a first proposed machine learning solution with at least one of a second component from a second proposed machine learning solution, wherein the first component comprises one or more of the machine learning model, the dataset and the data pipeline output of the first proposed machine learning solution, and wherein the second component comprises one or more of the machine learning model, the dataset and the data pipeline output of the second proposed machine learning solution; and
ranking, by the processing system, the hybrid solutions, wherein the ranking comprises determining, by the processing system, a quality score for each hybrid solution and sorting, by the processing system, the hybrid solutions according to the quality score for each hybrid solution.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
determining, by the processing system, a quality score for the plurality of proposed machine learning solutions;
ranking, by the processing system, the plurality of proposed machine learning solutions with the hybrid solutions;
providing, by the processing system, information about the ranking to a user associated with the machine learning problem to be solve, wherein the providing comprises, generating, by the processing system, a leaderboard showing the ranking and making the leaderboard available at a web interface; and
receiving from the user, by processing system, at the web interface, a user selection of a selected machine learning solution of the plurality of the proposed solutions and the hybrid solutions, wherein the user selection corresponds to a selected machine learning solution that satisfies requirements of the user as selected by the user.

### Claim 18 (depends on 17)

The method of claim 17, further comprising:
receiving, by the processing system, from the user, information about the machine learning problem, wherein the receiving comprises receiving, at the web interface by the processing system, information defining the machine learning problem, an initial dataset and a scoring method; and
providing, by the processing system, to other users at the web interface, the information about the machine learning problem.

### Claim 19 (depends on 17)

The method of claim 17, further comprising:
providing, by the processing system, to a plurality of users, information about the machine learning problem;
receiving, by the processing system, at the web interface, the plurality of proposed machine learning solutions to the machine learning problem;
prompting, by the processing system, the plurality of users to provide a component breakdown of each respective proposed machine learning solution; and
receiving, by the processing system, the component breakdown of each respective proposed machine learning solution.

### Claim 20 (depends on 17)

The method of claim 17, further comprising:
receiving, by the processing system, at the web interface, the plurality of proposed machine learning solutions to the machine learning problem; and
determining, by the processing system, a component breakdown of each respective proposed machine learning solution.
