System and method for monitoring application security in a network environment

Logic encoded in non-transitory media that includes instructions for execution and when executed by a processor, is operable to perform operations comprising:

• detecting an application role in a plurality of application roles of a distributed application executing in a plurality of virtual machines in a network environment;
• generating a role profile for the application role from an interaction pattern of the application role with the other application roles in the distributed application;
• mapping the role profile to a virtual machine (VM) executing the application role;
• generating a small state machine from the role profile;
• running the small state machine on a port associated with the VM;
• inspecting an application level traffic at the port;
• and
• determining an anomaly in traffic of the VM based on a change of state of the small state machine, wherein the anomaly indicates a security breach of the VM.

An apparatus, comprising:

• a memory element for storing data;
• and
• a processor operable to execute instructions associated with the data, wherein the processor and the memory element cooperate, such that the apparatus is configured for:
• detecting an application role in a plurality of application roles of a distributed application executing in a plurality of virtual machines in a network environment;
• generating a role profile for the application role from an interaction pattern of the application role with the other application roles in the distributed application;
• mapping the role profile to a virtual machine (VM) executing the application role;
• generating a small state machine from the role profile;
• running the small state machine on a port associated with the VM;
• inspecting an application level traffic at the port;
• and
• determining an anomaly in traffic of the VM based on a change of state of the small state machine, wherein the anomaly indicates a security breach of the VM.