# US 2019/0318081 A1 · Injecting trap code in an execution path of a process executing a program to generate a trap address range to detect potential malicious code

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0318081 A1
- Document ID: US20190318081A1
- Status: Published application
- Kind: A1
- Application number: 15/954,334
- Filed: 2018-04-16
- Published: 2019-10-17
- Earliest claimed: 2018-04-16
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Lokesh M. Gupta; Matthew G. Borlick; Trung N. Nguyen; Micah Robison
- Web page: https://patentgrep.com/patent/US20190318081A1

## CPC

- G06F 21/52: during program execution, e.g. stack integrity {; Preventing unwanted data erasure; Buffer overflow}
- G06F 8/70: Software maintenance or management
- G06F 9/30058: {Conditional branch instructions}
- G06F 9/445: Program loading or initiating (bootstrapping G06F9/4401; security arrangements for program loading or initiating G06F21/57)
- G06F 21/577: {Assessing vulnerabilities and evaluating computer system security}
- G06F 2221/033: Test or assess software
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Provided are a computer program product, system, and method for injecting trap code in an execution path of a process executing a program to generate a trap address range to detect potential malicious code. A specified type of command is processed in application code and, in response, trap code is executed to allocate a trap address range. The specified type of command is executed in the application code. A determination is made as to whether an accessing application accesses the trap address range. At least one of transmitting a notification that the accessing application comprises potentially malicious code, monitoring the execution of the accessing application, and restricting execution of the accessing application is performed in response to determining that the accessing application accessed the trap address range.

## Claims

### Claim 1

A computer program product for detecting potentially malicious code accessing data from a storage, the computer program product comprising a computer readable storage medium having computer readable program code embodied therein that when executed performs operations, the operations comprising:
processing a specified type of command in application code;
executing trap code in response to processing the specified type of command to allocate a trap address range;
executing the specified type of command in the application code;
determining whether an accessing application accesses the trap address range; and
performing at least one of transmitting a notification that the accessing application comprises potentially malicious code, monitoring the execution of the accessing application, and restricting execution of the accessing application in response to determining that the accessing application accessed the trap address range.

### Claim 2 (depends on 1)

The computer program product of claim 1, wherein the trap address range is not accessed by the application code being executed.

### Claim 3 (depends on 1)

The computer program product of claim 1, wherein the specified type of command comprises a command to switch from a user mode to kernel mode to access kernel addresses, and wherein the trap address range is defined in a kernel address space.

### Claim 4 (depends on 3)

The computer program product of claim 3, wherein the operations further comprise:
receiving a segmentation fault in response to the accessing application accessing an address in the kernel address space not mapped in an address table to valid data in a cache, wherein the determining whether the accessing application has accessed the trap address range is performed in response to receiving the segmentation fault.

### Claim 5 (depends on 4)

The computer program product of claim 4, wherein the specified type of command comprises a system call, wherein the system call is executed after executing the trap code.

### Claim 6 (depends on 1)

The computer program product of claim 1, wherein the trap address range maps to invalid physical addresses in an address table, wherein the determining whether the accessing application has accessed the trap address range comprises detecting a fault generated when the accessing application accesses the trap address range mapping to invalid locations.

### Claim 7 (depends on 1)

The computer program product of claim 1, wherein the executing the trap code maps the trap address range in an address table, wherein the determining whether the accessing application has accessed the trap address range comprises detecting that the accessing application attempts to use trap data from the trap address range to access computational resources.

### Claim 8 (depends on 1)

The computer program product of claim 1, wherein multiple trap address ranges are allocated in response to multiple instances of executing the trap code.

### Claim 9 (depends on 1)

The computer program product of claim 1, wherein the processor executing the application code speculatively executes both branches of conditional branches of the application code, wherein a result of only one of the conditional branches is maintained depending on a condition used to determine which of the conditional branches to traverse.

### Claim 10 (depends on 9)

The computer program product of claim 9, wherein the trap code is executed in a conditional branch of the conditional branches speculatively by a processor when processing the conditional branches.

### Claim 11

A system for detecting potentially malicious code accessing data from a storage, comprising:
processor; and
a computer readable storage medium having computer readable program that when executed by the processor performs operations, the operations comprising:
processing a specified type of command in application code;
executing trap code in response to processing the specified type of command to allocate a trap address range;
executing the specified type of command in the application code;
determining whether an accessing application accesses the trap address range; and
performing at least one of transmitting a notification that the accessing application comprises potentially malicious code, monitoring the execution of the accessing application, and restricting execution of the accessing application in response to determining that the accessing application accessed the trap address range.

### Claim 12 (depends on 11)

The system of claim 11, wherein the specified type of command comprises a command to switch from a user mode to kernel mode to access kernel addresses, and wherein the trap address range is defined in a kernel address space.

### Claim 13 (depends on 12)

The system of claim 12, wherein the operations further comprise:
receiving a segmentation fault in response to the accessing application accessing an address in the kernel address space not mapped in an address table to valid data in a cache, wherein the determining whether the accessing application has accessed the trap address range is performed in response to receiving the segmentation fault.

### Claim 14 (depends on 13)

The system of claim 13, wherein the specified type of command comprises a system call, wherein the system call is executed after executing the trap code.

### Claim 15 (depends on 11)

The system of claim 11, wherein the trap address range maps to invalid physical addresses in an address table, wherein the determining whether the accessing application has accessed the trap address range comprises detecting a fault generated when the accessing application accesses the trap address range mapping to invalid locations.

### Claim 16 (depends on 11)

The system of claim 11, wherein the executing the trap code maps the trap address range in an address table, wherein the determining whether the accessing application has accessed the trap address range comprises detecting that the accessing application attempts to use trap data from the trap address range to access computational resources.

### Claim 17 (depends on 11)

The system of claim 11, wherein the processor executing the application code speculatively executes both branches of conditional branches of the application code, wherein a result of only one of the conditional branches is maintained depending on a condition used to determine which of the conditional branches to traverse.

### Claim 18 (depends on 17)

The system of claim 17, wherein the trap code is executed in a conditional branch of the conditional branches speculatively by a processor when processing the conditional branches.

### Claim 19

A method for detecting potentially malicious code accessing data from a storage, comprising:
processing a specified type of command in application code;
executing trap code in response to processing the specified type of command to allocate a trap address range;
executing the specified type of command in the application code;
determining whether an accessing application accesses the trap address range; and
performing at least one of transmitting a notification that the accessing application comprises potentially malicious code, monitoring the execution of the accessing application, and restricting execution of the accessing application in response to determining that the accessing application accessed the trap address range.

### Claim 20 (depends on 19)

The method of claim 19, wherein the specified type of command comprises a command to switch from a user mode to kernel mode to access kernel addresses, and wherein the trap address range is defined in a kernel address space.

### Claim 21 (depends on 20)

The method of claim 20, further comprising:
receiving a segmentation fault in response to the accessing application accessing an address in the kernel address space not mapped in an address table to valid data in a cache, wherein the determining whether the accessing application has accessed the trap address range is performed in response to receiving the segmentation fault.

### Claim 22 (depends on 21)

The method of claim 21, wherein the specified type of command comprises a system call, wherein the system call is executed after executing the trap code.

### Claim 23 (depends on 19)

The method of claim 19, wherein the trap address range maps to invalid physical addresses in an address table, wherein the determining whether the accessing application has accessed the trap address range comprises detecting a fault generated when the accessing application accesses the trap address range mapping to invalid locations.

### Claim 24 (depends on 19)

The method of claim 19, wherein the executing the trap code maps the trap address range in an address table, wherein the determining whether the accessing application has accessed the trap address range comprises detecting that the accessing application attempts to use trap data from the trap address range to access computational resources.

### Claim 25 (depends on 19)

The method of claim 19, wherein the processor executing the application code speculatively executes both branches of conditional branches of the application code, wherein a result of only one of the conditional branches is maintained depending on a condition used to determine which of the conditional branches to traverse.

### Claim 26 (depends on 25)

The method of claim 25, wherein the trap code is executed in a conditional branch of the conditional branches speculatively by a processor when processing the conditional branches.
