# US 2018/0024874 A1 · Anomaly detection using sequences of system calls

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0024874 A1
- Document ID: US20180024874A1
- Status: Published application
- Kind: A1
- Application number: 15/213,874
- Filed: 2016-07-19
- Published: 2018-01-25
- Earliest claimed: 2016-07-19
- Assignees: 2236008 Ontario Inc.
- Applicants: 2236008 Ontario Inc.
- Inventors: Mohamed Al Sharnouby
- Web page: https://patentgrep.com/patent/US20180024874A1

## CPC

- G06F 9/546: {Message passing systems or structures, e.g. queues}
- G06F 9/4812: {by interrupt, e.g. masked}
- G06F 9/4843: {by program, e.g. task dispatcher, supervisor, operating system}
- G06F 21/52: during program execution, e.g. stack integrity {; Preventing unwanted data erasure; Buffer overflow}
- G06F 21/554: {involving event detection and direct action}
- G06F 21/56: Computer malware detection or handling, e.g. anti-virus arrangements
- G06F 21/566: {Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities}
- G06F 2221/034: Test or assess a computer or a system
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and methods of detecting a call sequence anomaly in a message-based operating system are provided. A message may be received that indicates a programmatic procedure of an operating system was invoked. The message may include a programmatic procedure identifier, a sender process identifier, and a receiver process identifier. An invocation hash may be generated based on the message. The invocation hash may be translated to a smaller invocation identifier. The invocation identifier may be included in a translated call sequence that comprises invocation identifiers for a series of invocations. Depending on whether the translated call sequence is included in previously generated predetermined call sequences, the translated call sequence may be determined as an anomaly or not an anomaly.

## Claims

### Claim 1

A method of detecting a call sequence anomaly in a message-based operating system, the method comprising:
receiving a message indicating an invocation of a programmatic procedure of an operating system, the message including a programmatic procedure identifier, a sender process identifier, and a receiver process identifier, wherein the invocation of the programmatic procedure is one invocation in a series of invocations of operating system programmatic procedures made from a process identified by the sender process identifier;
generating an invocation hash based on at least the programmatic procedure identifier and the receiver process identifier;
translating the invocation hash to an invocation identifier;
including the invocation identifier in a translated call sequence that comprises invocation identifiers for the series of invocations;
determining whether the translated call sequence is included in a plurality of predetermined call sequences, each of the predetermined call sequences comprising corresponding invocation identifiers, wherein each of the corresponding invocation identifiers are mapped to invocation hashes; and
identifying the translated call sequence as an anomaly if the translated call sequence is not included in the predetermined call sequences otherwise determining the translated call sequence is not an anomaly.

### Claim 2 (depends on 1)

The method of claim 1, wherein invocation hash is also based on the sender process identifier.

### Claim 3 (depends on 1)

The method of claim 1, wherein the message includes an indication of parameters passed to the programmatic procedure, and wherein the invocation hash is also based on the indication of parameters passed to the programmatic procedure.

### Claim 4 (depends on 1)

The method of claim 1, wherein the message includes a channel identifier, and wherein the invocation hash is also based on the channel identifier.

### Claim 5 (depends on 1)

The method of claim 1, wherein the message includes a receiver node identifier, and wherein the invocation hash is also based on the receiver node identifier.

### Claim 6 (depends on 1)

The method of claim 1, wherein translating the invocation hash comprises searching for the invocation hash in a translation table comprising invocation hashes associated with the invocation identifiers.

### Claim 7 (depends on 6)

The method of claim 6, wherein the invocation identifier is a row number in the translation table that includes the invocation hash.

### Claim 8

A non-transitory computer readable storage medium comprising computer executable instructions, the computer executable instructions executable by a processor, the computer executable instructions comprising:
instructions executable to receive a message indicating an invocation of a programmatic procedure of an operating system, the message including a programmatic procedure identifier, a sender process identifier, and a receiver process identifier, wherein the invocation of the programmatic procedure is one invocation in a series of invocations of operating system programmatic procedures made from a process identified by the sender process identifier;
instructions executable to generate an invocation hash based on at least a portion of the message;
instructions executable to translate the invocation hash to an invocation identifier;
instructions executable to include the invocation identifier in a translated call sequence that comprises invocation identifiers for the series of invocations;
instructions executable to determine whether the translated call sequence is included in a plurality of predetermined call sequences, each of the predetermined call sequences comprising corresponding invocation identifiers, wherein each of the corresponding invocation identifiers are mapped to invocation hashes; and
instructions executable to identify the translated call sequence as an anomaly if the translated call sequence is not included in the predetermined call sequences and, otherwise, to determine the translated call sequence is not an anomaly.

### Claim 9 (depends on 8)

The non-transitory computer readable storage medium of claim 8, wherein a determination of whether the translated call sequence is included in the predetermined call sequences is based on a search for the translated call sequence in a table, wherein the table comprises rows of sets of invocation identifiers, each of the invocation identifiers is associated in a translation data structure with a corresponding invocation hash.

### Claim 10 (depends on 8)

The non-transitory computer readable storage medium of claim 8, wherein receipt of the message comprises a read of information about the invocation of the programmatic procedure from a trace file.

### Claim 11 (depends on 8)

The non-transitory computer readable storage medium of claim 8, wherein receipt of the message comprises receipt of the message in the form of a registered callback from the operating system.

### Claim 12 (depends on 8)

The non-transitory computer readable storage medium of claim 8, wherein the generation of the invocation hash comprises an application of a hash function that maps data of arbitrary size to data of a fixed size, the hash function applied to data comprising the programmatic procedure identifier, the sender process identifier, and the receiver process identifier.

### Claim 13

A system to detect a call sequence anomaly in a message-based operating system, the system comprising:
a processor configured to:
receive a message indicating an invocation of a programmatic procedure of an operating system, the message including a programmatic procedure identifier, a sender process identifier, and a receiver process identifier, wherein the invocation of the programmatic procedure is one invocation in a series of invocations of operating system programmatic procedures made from a process identified by the sender process identifier;
generate an invocation hash based on at least a portion of the message;
translate the invocation hash to an invocation identifier;
include the invocation identifier in a translated call sequence that comprises invocation identifiers for the series of invocations;
determine whether the translated call sequence is included in a plurality of predetermined call sequences, each of the predetermined call sequences comprising corresponding invocation identifiers, wherein each of the corresponding invocation identifiers are mapped to invocation hashes; and
identify the translated call sequence as an anomaly if the translated call sequence is not included in the predetermined call sequences and, otherwise, to determine the translated call sequence is not an anomaly.

### Claim 14 (depends on 13)

The system of claim 13, wherein the processor is configured to generate the invocation hash based on at least the receiver process identifier.

### Claim 15 (depends on 13)

The system of claim 13, wherein the processor is configured to generate the invocation hash based on at least the sender process identifier and the receiver process identifier.

### Claim 16 (depends on 13)

The system of claim 13, wherein the message comprises the programmatic procedure identifier, the sender process identifier, the receiver process identifier, a channel identifier, a receiver node identifier, parameters of the programmatic procedure, and/or an indication of the parameters passed to the programmatic procedure

### Claim 17 (depends on 16)

The system of claim 16, wherein the processor is configured to generate the invocation hash based on at least one of the programmatic procedure identifier, the sender process identifier, the receiver process identifier, the channel identifier, the receiver node identifier, the parameters of the programmatic procedure, and/or the indication of the parameters passed to the programmatic procedure.

### Claim 18 (depends on 13)

The system of claim 13, wherein the processor is configured to change a security level in response to identification of the translated call sequence as an anomaly.

### Claim 19 (depends on 18)

The system of claim 18, wherein the processor is configured to change the security level in response to a determination that the identification of the translated call sequence as an anomaly exceeds a threshold number of anomalies.
