System and method for observing and controlling a programmable network using time varying data collection

A system for simultaneously and centrally analyzing a plurality of networks and improving security and performance of at least one network within the plurality of networks, the system comprising:

• each network in the plurality of networks including a plurality of network elements, wherein a first network of the plurality of networks is from a first company and a second network of the plurality of networks is from a second company, the plurality of network elements including one or more physical switches, one or more physical routers, one or more wireless access points, one or more physical firewalls, one or more virtual switches, one or more virtual routers, and one or more physical network packet brokers;
• a programmable network element that is a programmable switch, router, or wireless access point;
• and
• one or more collectors configured to collect filtered network traffic data from the plurality of network elements in the plurality of networks based on a time-varying schedule;
• a remote network manager comprising a network interface and configured to connect to the one or more collectors over the Internet via the network interface, and further configured to simultaneously and centrally analyze (1) the network traffic data from the plurality of network elements in the plurality of networks and (2) network management data from a plurality of enterprise systems in the plurality of networks, wherein the network management data includes L1 through L7 network topology data, network configuration data, and simple network management protocol data, wherein data from a network management system is network management data, wherein L1 through L7 network topology data is data from L1 through L7 network topology, and wherein configuration data from the network elements is network configuration data;
• wherein the remote network manager combines the network traffic data from the plurality of networks and the network management data from the plurality of enterprise systems in the plurality of networks into combined cross-network data from multiple companies for simultaneous and central analysis of the combined cross-network data from the multiple companies;
• wherein the system extracts, using deep packet inspection, Quality of Experience (QoE) metadata for users and devices that differs on an application-by-application basis, wherein the Quality of Experience (QoE) metadata is time aligned with metadata extracted across an application stack, and
• wherein the system compares the Quality of Experience (QoE) metadata for users and devices on an application-by-application basis, and analyzes time series data for trends over dimensions including time, device types, operating systems, and access points, and wherein the system programs the programmable network element to remediate security and performance issues, wherein the remediation of security and performance issues includes detecting and mitigating cyber-attacks,
• wherein the system implements control policies that summarize and index higher layer information about users, applications, devices, and behaviors using machine learning, wherein the higher layer information is quarriable using natural language processing, and wherein the control policies automatically adjust to changes in network topology.

A method for simultaneously and centrally analyzing a plurality of networks and improving security and performance of at least one network within the plurality of networks, the system comprising:

• providing a plurality of network elements in each network of the plurality of networks, wherein a first network of the plurality of networks is from a first company and a second network of the plurality of networks is from a second company, the plurality of network elements including one or more physical switches, one or more physical routers, one or more wireless access points, one or more physical firewalls, one or more virtual switches, one or more virtual routers, and one or more physical network packet brokers;
• providing a programmable network element that is a programmable switch, router, or wireless access point;
• collecting, via one or more collectors, filtered network traffic data from the plurality of network elements based on a time-varying schedule,
• programming the programmable network element based on the filtered network traffic data;
• providing a remote network manager comprising a network interface and configured to connect to the one or more collectors over the Internet via the network interface, and further configured to simultaneously and centrally analyze (1) the network traffic data from the plurality of network elements in the plurality of networks and (2) network management data from a plurality of enterprise systems in the plurality of networks, wherein the network management data includes L1 through L7 network topology data, network configuration data, and simple network management protocol data, wherein data from a network management system is network management data, wherein L1 through L7 network topology data is data from L1 through L7 network topology, and wherein configuration data from the network elements is network configuration data;
• wherein the remote network manager combines the network traffic data from the plurality of networks and the network management data from the plurality of enterprise systems in the plurality of networks into combined cross-network data from multiple companies for simultaneous and central analysis of the combined cross-network data from the multiple companies;
• extracting, using deep packet inspection, Quality of Experience (QoE) metadata for users and devices that differs on an application-by-application basis, wherein the Quality of Experience (QoE) metadata is time aligned with metadata extracted across an application stack;
• analyzing time series data for trends over dimensions including time, device types, operating systems, and access points;
• programming the programmable network element to remediate security and performance issues identified by the Quality of Experience (QoE) metadata comparison;
• and
• detecting and mitigating cyber-attacks,
• wherein the system implements control policies that summarize and index higher layer information about users, applications, devices, and behaviors using machine learning, wherein the higher layer information is quarriable using natural language processing, and wherein the control policies automatically adjust to changes in network topology.