# US 2021/0084059 A1 · Assessing technical risk in information technology service management using visual pattern recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0084059 A1
- Document ID: US20210084059A1
- Status: Published application
- Kind: A1
- Application number: 16/571,088
- Filed: 2019-09-14
- Published: 2021-03-18
- Earliest claimed: 2019-09-14
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Hongtan Sun; Larisa Shwartz; Rohit Madhukar Khandekar; Qing Wang; Bing Zhou
- Web page: https://patentgrep.com/patent/US20210084059A1

## CPC

- H04L 41/16: using machine learning or artificial intelligence
- H04L 63/1425: {Traffic logging, e.g. anomaly detection}
- G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/091: Active learning
- H04L 41/142: using statistical or mathematical methods
- H04L 43/045: for graphical visualisation of monitoring data
- H04L 63/1416: {Event detection, e.g. attack signature detection}
- H04L 63/1433: {Vulnerability analysis}
- G06Q 10/0635: Risk analysis of enterprise or organisation activities
- H04L 43/0817: by checking functioning
- H04L 43/0888: Throughput
- H04L 2463/121: Timestamp
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

A computer system, non-transitory computer storage medium, and a computer-implemented method of assessing technical risk using visual pattern recognition in an Information Technology (IT) Service Management System. A data visualization engine and a time series generation engine receive the operational data, respectively. A first representation of the data is generated by the data visualization engine, and a second representation of the data is generated by the time series generation engine. Anomaly patterns are identified by a pattern recognition engine configured to perform feature extraction and data transformation. An ensembler is configured to accept the outputs from two AI anomaly engines and make a final decision of whether anomaly patterns are captured. Risk scores based on the identified anomaly patterns are output by a pattern recognition engine to an automated management system. The anomalies includes information regarding vulnerabilities of devices or components of the IT Service Management System.

## Claims

### Claim 1

A computing system comprising:
a processor;
a plurality of engines coupled to the processor and configured to perform acts, comprising:
receive an operational data of a system being monitored by a data visualization engine and a time series generation engine of the plurality of engines, respectively;
process the operational data by the data visualization engine to create a first representation of the operational data;
process the operational data by the time series generation engine to create a second representation of the operational data;
receive, by a pattern recognition engine of the plurality of engines, the first representation of the operational data and the second representation of the operational data, and perform feature extraction and data transformation on the respective first representation of the operational data and the second representation of the operational data;
identify respective anomaly patterns in the feature extracted and data transformed first representation and second representation of the operational data;
render, by an ensembler, a decision as to whether the identified anomaly patterns in the feature extracted and data transformed respective first and second representation of the operational data are associated with vulnerabilities of devices or components in the system being monitored; and
output the decision rendered by the ensembler to an output node.

### Claim 2 (depends on 1)

The computing system according to claim 1, wherein:
the first representation of the operational data comprises analytical visual charts including visualized operational data generated by the data visualization engine, and
the second representation of the operational data comprises time series data patterns generated by the time series generation engine.

### Claim 3 (depends on 2)

The computing system according to claim 2, further comprising a machine learning model that is configured during a training phase to:
analyze historical patterns of analytical visual charts generated by the data visualization engine;
analyze historical patterns of time series data generated by the time series generation engine; and
receive a label of historical patterns of analytical visual charts and historical patterns of time series data as being anomalies according to predetermined criteria.

### Claim 4 (depends on 1)

The computing system according to claim 1, wherein the ensembler is configured to render the decision as to whether the identified anomaly patterns are associated with vulnerabilities of devices or components in the system being monitored.

### Claim 5 (depends on 4)

The computing system according to claim 4, wherein the ensemble is configured to perform a voting operation combining a largest number of classifiers associated with the anomaly patterns.

### Claim 6 (depends on 5)

The computing system according to claim 5, wherein the ensembler is configured to perform the voting operation to include a weighting system that is assigned to the classifers associated with the anomaly patterns.

### Claim 7 (depends on 1)

The computing system according to claim 1, further comprising a risk assessment engine configured to calculate one or more risk assessment scores associated with the vulnerabilities of devices or components in the system being monitored, wherein:
the risk assessment engine outputs the calculated one or more risk assessment scores to the output node, and
the output node comprises an Information Technology (IT) service management system.

### Claim 8 (depends on 7)

The computing system according to claim 7, wherein the one or more risk assessment scores are output to a User Interface (UI) in communication with the output node.

### Claim 9 (depends on 6)

The computing system according to claim 6, wherein the risk assessment engine assesses risk scores in real time utilizing the operational data processed by the data visualization engine and the time series generation engine.

### Claim 10 (depends on 3)

The computing system according to claim 3, wherein:
the pattern recognition engine includes at least two artificial intelligence (AI) engines comprising:
a first anomaly AI engine comprising a convolutional neural network (CNN) system configured to identify anomaly patterns of the visualized operational data generated by the data visualization engine; and
a second AI anomaly engine comprising a Long Short-Term Memory (LS™) model configured to identify anomaly patterns from the patterns of time series data generated by the time series generation engine; and
the ensembler is configured to receive outputs of the first AI anomaly engine and the second AI anomaly engine.

### Claim 11 (depends on 10)

The computing system according to claim 10, wherein, during an active phase, patterns of time series data unidentified by the first AI anomaly engine or patterns of visual analytical charts unidentified by the second AI anomaly engine are labeled during a training phase of the machine learning model and output to the pattern recognition engine.

### Claim 12 (depends on 10)

The computing system according to claim 10, wherein the pattern recognition engine is configured to perform the data transformation by a Fast Fourier Transform that transforms the visual analytic charts from a time space to a frequency space.

### Claim 13

A non-transitory computer readable storage medium tangibly embodying a computer readable program code having computer readable instructions that, when executed, causes a computer device to carry out a method of assessing technical risk using visual pattern recognition and pattern recognition of time series data, the method comprising:
processing an operational data of a system being monitored by a data visualization engine and a time series generation engine, respectively, to create two representations from the operational data;
generating, by the data visualization engine, a first representation of the operational data comprising analytical visual charts including visualized operational data;
generating, by the time series generation engine, a second representation of the operational data comprising patterns of time series data;
identifying, by a pattern recognition engine, anomaly patterns in the first representation of the operational data and the second representation of the operational data, respectively;
rendering a decision as to whether the identified respective anomaly patterns in the first representation of the operational data and the second representation of the operational data are associated with vulnerabilities of devices or components in the system being monitored; and
outputting the decision to an output node.

### Claim 14 (depends on 13)

The non-transitory computer readable storage medium of claim 13, further comprising:
calculating, by a risk assessment engine, one or more risk assessment scores associated with the vulnerabilities of devices or components in the system being monitored, based on anomaly patterns in the first representation of the operational data and the second representation of the operational data, respectively; and
outputting the one or more risk assessment scores to an Information Technology (IT) service management system.

### Claim 15 (depends on 14)

The non-transitory computer readable storage medium of claim 14, wherein:
the pattern recognition engine includes a first artificial intelligence (AI) anomaly engine comprising a convolutional neural network (CNN) system, and a second AI anomaly engine comprising a Long Short-Term Memory (LSTM) model;
the identifying of anomaly patterns of the visualized operational data generated by the data visualization engine is performed by the first AI engine; and
the identifying of anomaly patterns from the time series data generated by the time series generation engine is performed by the second AI engine.

### Claim 16 (depends on 15)

The non-transitory computer readable storage medium of claim 15, further comprising:
receiving, by an ensembler, outputs of the first AI anomaly engine and the second AI anomaly engine, wherein the ensembler is further configured to render a decision as to whether the outputs of the first AI anomaly engine and the second AI anomaly engine comprise anomaly patterns associated with the vulnerabilities of devices or components in the system being monitored, based on a voting operation combining a largest number of classifiers associated with the anomaly patterns.

### Claim 17 (depends on 13)

The non-transitory computer readable storage medium of claim 13, wherein:
the pattern recognition engine is configured to perform feature extraction and data transformation of the first representation of the operational data by the data visualization engine, and the second representation of the operational data by the time series generation engine, and
the pattern recognition engine performs the data transformation by at least one of a Fast Fourier Transform, binning, or normalization.

### Claim 18

A computer-implemented method of risk assessment utilizing visual pattern recognition, the computer-implemented method comprising:
receiving an operational data by a data visualization engine and a time series generation engine;
generating, by the data visualization engine, visual analytic charts of the operational data;
generating, by the time series generation engine, a time series representation of the operational data;
performing, by a pattern recognition engine, feature extraction and data transformation on the visual analytic charts and the time series representation of the operational data;
identifying, by a first Artificial Intelligence (AI) anomaly engine including a convolutional neural network system, anomaly patterns from the visualized analytic charts;
identifying, by a second AI anomaly engine including a Long Short-Term Memory (LSTM) model, anomaly patterns from the time series representation of the operational data;
receiving, by an ensembler, outputs of the first AI anomaly engine and the second AI anomaly engine and rendering a decision as to whether the outputs of the first AI anomaly engine and the second AI anomaly engine comprise anomaly patterns associated with vulnerabilities of device or components in a system being monitored;
rendering, by a risk assessment engine that receives the decision from the ensembler, risk scores associated with the identified vulnerabilities of devices or components in an Information Technology service management system; and
outputting the risk scores to a user interface (UI).

### Claim 19 (depends on 18)

The computer-implemented method according to claim 18, wherein:
the pattern recognition engine identifies the anomaly patterns using a first representation and a second representation of the same operational data, respectively,
the performing of feature extraction and data transformation on the visual analytic data and the time series data includes at least one of a dimensionality reduction, a Fast Fourier Transformation of the visual analytic data and the time series representation of the operational data, or calculating a natural logarithm of the operational data.

### Claim 20 (depends on 19)

The computer-implemented method according to claim 19, further comprising configuring a machine learning model during a training phase to:
analyze historical patterns of analytical visual charts generated by the data visualization engine;
analyze historical patterns of time series data generated by the time series generation engine; and
receive a label of historical patterns of analytical visual charts and historical patterns of time series data as being anomalies according to predetermined criteria.
