# US 2019/0173918 A1 · Systems and methods for aida based a/b testing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0173918 A1
- Document ID: US20190173918A1
- Status: Published application
- Kind: A1
- Application number: 15/829,742
- Filed: 2017-12-01
- Published: 2019-06-06
- Earliest claimed: 2017-12-01
- Assignees: KnowBe4, Inc.
- Applicants: KnowBe4, Inc.
- Inventors: Eric Sites
- Web page: https://patentgrep.com/patent/US20190173918A1

## CPC

- H04L 63/1483: {service impersonation, e.g. phishing, pharming or web spoofing (detection of rogue wireless access points H04W12/12)}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- H04L 63/1425: {Traffic logging, e.g. anomaly detection}
- H04L 63/1433: {Vulnerability analysis}
- G06F 2221/2119: Authenticating web pages, e.g. with suspicious links
- G06N 3/047: Probabilistic or stochastic networks
- G06N 20/00: Machine learning
- H04L 63/1491: {using deception as countermeasure, e.g. honeypots, honeynets, decoys or entrapment}
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and methods are described by which a serving module of a campaign controller identifies a first version of a model which the campaign controller uses to communicate a first simulated phishing communication to a plurality of users. The campaign controller receives a first response from a first user to the simulated phishing communication and a second response from a second user to the simulated phishing communication and determines that the first and second responses are corresponding, for example are the same or similar. The serving module assigns a first user to a first group of users and a second user to a second group of users and identifies a second version of the model to use for the first user and a third version of the model to use for the second user.

## Claims

### Claim 1

A method for testing one or more versions of a model used for simulated phishing campaigns, the method comprising:
(a) identifying, by a serving module of a campaign controller, a first version of a model to be used by the campaign controller for executing simulated phishing campaigns;
(b) communicating, by the campaign controller, a first simulated phishing communication to a plurality of users using the first version of the model;
(c) receiving, by the campaign controller, a first response from a first user of the plurality of users to the first simulated phishing communication and a second response from a second user of the plurality of users to the first simulated phishing communication;
(d) determining, by the campaign controller, that the first response and the second response are corresponding responses;
(e) assigning, by the serving module responsive to the determination, the first user to a first group of users and the second user to a second group of users different from the group of users; and
(f) identifying, by the serving module, a second version of the model to use for the first user and a third version of the model to use for the second user.

### Claim 2 (depends on 1)

The method of claim 1, further comprising communicating, by the campaign controller, one or more simulated phishing communications to the first group of users using the second version of the model and communicating, by the campaign controller, one or more simulated phishing communications to the second group of users using the third version of the model.

### Claim 3 (depends on 2)

The method of claim 2, further comprises receiving, by the campaign controller, a first plurality of responses from the first group of users and a second plurality of responses from the second group of users.

### Claim 4 (depends on 3)

The method of claim 3, further comprises determining, by the serving module, whether the second version of the model performed better than the third version of the model in causing one or more users to interact with a link of simulated phishing communications.

### Claim 5 (depends on 3)

The method of claim 3, further comprising assigning, by the serving module responsive to the first plurality of responses, one or more users of the first group of users to a third group of users.

### Claim 6 (depends on 3)

The method of claim 3, further comprising assigning, by the serving module responsive to the second plurality of responses, one or more users of the second group of users to a third group of users.

### Claim 7 (depends on 4)

The method of claim 4, further comprising establishing, by the serving module, the third version of the model as a model to use for one or more subsequent simulated phishing campaigns.

### Claim 8 (depends on 1)

The method of claim 1, wherein (d) further comprises determining, by the campaign controller, that the first response is the same as the second response.

### Claim 9 (depends on 1)

The method of claim 1, wherein the second version of the model is a same model as the first version of the model.

### Claim 10

A system for testing one or more versions of a model used for simulated phishing campaigns, the system comprising:
one or more hardware processors, coupled to memory;
a serving module of a campaign controller executable on the one or more hardware processors, coupled to memory and configured to identify a first version of a model to be used by the campaign controller for executing simulated phishing campaigns;
wherein the campaign controller is configured to communicate a first simulated phishing communication to a plurality of users using the first version of the model; receive a first response from a first user of the plurality of users to the first simulated phishing communication and a second response from a second user of the plurality of users to the first simulated phishing communication;
determine that the first response and the second response are corresponding responses;
wherein the serving module is configured to assign, responsive to the determination, the first user to a first group of users and the second user to a second group of users different from the group of users; and
identify a second version of the model to use for the first user and a third version of the model to use for the second user.

### Claim 11 (depends on 10)

The system of claim 10, wherein the campaign controller is further configured to communicate one or more simulated phishing communications to the first group of users using the second version of the model and communicate one or more simulated phishing communications to the second group of users using the third version of the model.

### Claim 12 (depends on 11)

The system of claim 11, wherein the campaign controller is further configured to receive a first plurality of responses from the first group of users and a second plurality of responses from the second group of users.

### Claim 13 (depends on 12)

The system of claim 12, wherein the serving module is further configured to determine whether the second version of the model performed better than the third version of the model in causing one or more users to interact with a link of simulated phishing communications.

### Claim 14 (depends on 12)

The system of claim 12, wherein the serving module is further configured to assign, responsive to the first plurality of responses, one or more users of the first group of users to a third group of users.

### Claim 15 (depends on 12)

The system of claim 12, wherein the serving module is further configured to assign, responsive to the second plurality of responses, one or more users of the second first group of users to a third group of users.

### Claim 16 (depends on 12)

The system of claim 12, wherein the serving module is further configured to establish the second version of the model as a model to use for one or more subsequent simulated phishing campaigns.

### Claim 17 (depends on 10)

The system of claim 10, wherein the campaign controller is further configured to determine that the first response is the same as the second response.

### Claim 18 (depends on 10)

The system of claim 10, wherein the second version of the model is a same model as the first version of the model.

### Claim at position 19

19-30.