Autonomous distributed forwarding plane traceability based anomaly detection in application traffic for hyper-scale SD-WAN

A non-transitory machine readable medium storing a program for execution by a set of processing units, the program for detecting and remediating anomalies in a software-defined wide area network (SD-WAN) connecting a plurality of branch sites, the SD-WAN comprising a controller, at least one enterprise datacenter, and at least one node at each branch site, each node at each branch site comprising a set of packet processing stages for processing packet flows that traverse the SD-WAN, the program comprising sets of instructions for:

• at the controller:
• receiving, from a particular node of a particular branch site in the SD-WAN, a flow notification indicating an anomaly in processing of packets that is detected by the particular node;
• dynamically generating a set of trace monitoring rules based on the detected anomaly and providing the set of trace monitoring rules to the particular node and at least one other node of another branch site in the SD-WAN, wherein the set of trace monitoring rules specify one or more packet flows to be traced by the packet processing stages of the particular node and the at least one other node;
• receiving, from the particular node of the particular branch site and the at least one other node of the other branch site, a set of trace monitoring results collected in response to the provided set of trace monitoring rules;
• and
• identifying (i) one or more anomalies and (ii) one or more dynamic actions to correct the identified anomalies by analyzing the trace monitoring results.