# US 2020/0396235 A1 · Automated Estimation of Network Security Policy Risk > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0396235 A1 - Document ID: US20200396235A1 - Status: Published application - Kind: A1 - Application number: 16/898,760 - Filed: 2020-06-11 - Published: 2020-12-17 - Earliest claimed: 2019-06-11 - Assignees: Zscaler, Inc. - Applicants: Zscaler, Inc. - Inventors: John O 'Neil; Michael J. Melson - Web page: https://patentgrep.com/patent/US20200396235A1 ## CPC - H04L 63/1425: {Traffic logging, e.g. anomaly detection} - H04L 41/142: using statistical or mathematical methods - H04L 41/145: {involving simulating, designing, planning or modelling of a network} - H04L 41/149: for prediction of maintenance - H04L 61/5007: Internet protocol [IP] addresses - H04L 43/0876: Network utilisation, e.g. volume of load or congestion level - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract A computer system automatically tests a network communication model by predicting whether particular traffic (whether actual or simulated) should be allowed on the network, and then estimating the accuracy of the network communication model based on the prediction. Such an estimate may be generated even before the model has been applied to traffic on the network. For example, the model may be generated based on a first set of network traffic. The accuracy of the model may then be estimated based on a second set of network traffic. This allows the accuracy of the model to be estimated without first waiting to apply the model to actual network traffic, thereby reducing the risk associated with applying the model before its accuracy is known. ## Claims ### Claim 1 A method performed by at least one computer processor executing computer program instructions stored in at least one non-transitory computer-readable medium, the method comprising: (A) for each of a plurality of observed communications over a network between applications executing on a plurality of computer systems, collecting and storing observed communications data representing the plurality of observed communications; (B) generating a network communication model based on the observed communications data; (C) generating, based on the observed communications data, hypothetical communications data representing a plurality of hypothetical communications that is distinct from the plurality of observed communications; and (D) identifying allowed data representing a plurality of allowed network communications, including at least some of the plurality of observed communications and at least some of the plurality of hypothetical communications, that are allowed by the network communication model; (E) identifying positive data representing a plurality of network communications that should be allowed by the network communication model; and (F) calculating an accuracy of the network communication model based on the allowed data and the positive data. ### Claim 2 (depends on 1) The method of claim 1, wherein the plurality of observed communications does not include any of the plurality of hypothetical communications. ### Claim 3 (depends on 1) The method of claim 1, wherein collecting and storing the observed communications data comprises collecting and storing, for each of the plurality of observed communications: data representing a source application of the observed communication; data representing a destination application of the observed communication; data representing a local Internet Protocol (IP) address of the observed communication; and data representing a remote IP address of the observed communication. ### Claim 4 (depends on 1) The method of claim 1, wherein (F) comprises dividing: (1) a size of an intersection of the allowed data and the positive data by (2) the size of the intersection of the allowed data and the positive data plus a size of a subset of the positive data which are not allowed by the network communication model. ### Claim 5 (depends on 1) The method of claim 1, wherein (F) comprises: (F) (1) calculating a precision value P based on the allowed data and the positive data; (F) (2) calculating a recall value R based on the allowed data and the positive data; and (F) (3) calculating the accuracy F based on the precision value and the recall value. ### Claim 6 (depends on 5) The method of claim 5, wherein (F) (3) comprises calculating F as (2×P×R)/(P+R). ### Claim 7 (depends on 6) The method of claim 6, wherein (F) (1) comprises dividing: (1) a size of an intersection of the allowed data and the positive data by (2) the size of the intersection of the allowed data and the positive data plus a size of a subset of the hypothetical data which are allowed by the network communication model. ### Claim 8 (depends on 7) The method of claim 7, wherein (F) (2) comprises dividing: (1) a size of an intersection of the allowed data and the positive data by (2) the size of the intersection of the allowed data and the positive data plus a size of a subset of the positive data which are not allowed by the network communication model. ### Claim 9 (depends on 1) The method of claim 1, wherein calculating the accuracy comprises calculating the accuracy before applying the network communication model to any communications on the network. ### Claim 10 (depends on 1) The method of claim 1, wherein identifying the positive data comprises receiving input indicating that the plurality of network communications should be allowed by the network communication model and storing data representing the input indicating that the plurality of network communications should be allowed by the network communication model. ### Claim 11 A system comprising at least one non-transitory computer-readable medium storing computer program instructions executable by at least one computer processor to perform a method, the method comprising: (A) for each of a plurality of observed communications over a network between applications executing on a plurality of computer systems, collecting and storing observed communications data representing the plurality of observed communications; (B) generating a network communication model based on the observed communications data; (C) generating, based on the observed communications data, hypothetical communications data representing a plurality of hypothetical communications that is distinct from the plurality of observed communications; and (D) identifying allowed data representing a plurality of allowed network communications, including at least some of the plurality of observed communications and at least some of the plurality of hypothetical communications, that are allowed by the network communication model; (E) identifying positive data representing a plurality of network communications that should be allowed by the network communication model; and (F) calculating an accuracy of the network communication model based on the allowed data and the positive data. ### Claim 12 (depends on 11) The system of claim 11, wherein the plurality of observed communications does not include any of the plurality of hypothetical communications. ### Claim 13 (depends on 11) The system of claim 11, wherein collecting and storing the observed communications data comprises collecting and storing, for each of the plurality of observed communications: data representing a source application of the observed communication; data representing a destination application of the observed communication; data representing a local Internet Protocol (IP) address of the observed communication; and data representing a remote IP address of the observed communication. ### Claim 14 (depends on 11) The system of claim 11, wherein (F) comprises dividing: (1) a size of an intersection of the allowed data and the positive data by (2) the size of the intersection of the allowed data and the positive data plus a size of a subset of the positive data which are not allowed by the network communication model. ### Claim 15 (depends on 11) The system of claim 11, wherein (F) comprises: (F) (1) calculating a precision value P based on the allowed data and the positive data; (F) (2) calculating a recall value R based on the allowed data and the positive data; and (F) (3) calculating the accuracy F based on the precision value and the recall value. ### Claim 16 (depends on 15) The system of claim 15, wherein (F) (3) comprises calculating F as (2×P×R)/(P+R). ### Claim 17 (depends on 16) The system of claim 16, wherein (F) (1) comprises dividing: (1) a size of an intersection of the allowed data and the positive data by (2) the size of the intersection of the allowed data and the positive data plus a size of a subset of the hypothetical data which are allowed by the network communication model. ### Claim 18 (depends on 17) The system of claim 17, wherein (F) (2) comprises dividing: (1) a size of an intersection of the allowed data and the positive data by (2) the size of the intersection of the allowed data and the positive data plus a size of a subset of the positive data which are not allowed by the network communication model. ### Claim 19 (depends on 11) The system of claim 11, wherein calculating the accuracy comprises calculating the accuracy before applying the network communication model to any communications on the network. ### Claim 20 (depends on 11) The system of claim 11, wherein identifying the positive data comprises receiving input indicating that the plurality of network communications should be allowed by the network communication model and storing data representing the input indicating that the plurality of network communications should be allowed by the network communication model.