# US 2019/0317850 A1 · Intelligent responding to error screen associated errors

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0317850 A1
- Document ID: US20190317850A1
- Status: Published application
- Kind: A1
- Application number: 15/955,058
- Filed: 2018-04-17
- Published: 2019-10-17
- Earliest claimed: 2018-04-17
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Kwan Yin Andrew CHAU; Tony LE; Patrick WONG; Dien D. NGUYEN
- Web page: https://patentgrep.com/patent/US20190317850A1

## CPC

- G06F 9/4494: {data driven}
- G06F 8/31: {Programming languages or programming paradigms}
- G06F 8/61: Installation
- G06F 8/65: Updates (security arrangements therefor G06F21/57)
- G06F 9/451: Execution arrangements for user interfaces
- G06F 11/0769: {Readable error formats, e.g. cross-platform generic formats, human understandable formats}
- G06F 11/0793: {Remedial or corrective actions (recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40)}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/241: relating to the classification model, e.g. parametric or non-parametric approaches
- G06F 40/30: Semantic analysis
- G06N 20/00: Machine learning
- G06V 10/75: Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V 2201/02: Recognising information on displays, dials, clocks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods, computer program products, and systems are presented. The method computer program products, and systems can include, for instance: obtaining iteratively captured frames of image data representing a user interface screen, wherein one or more of the frames of image data represents an error screen indicating an error condition of one or more resource of a plurality of resources of a services system; performing recognition processing using image data of the captured frames of image data to determine an error classifier associated to the error screen: determining one or more action based on the error classifier; and performing the one or more action in response to the determining.

## Claims

### Claim 1

A method comprising:
obtaining iteratively captured frames of image data representing a user interface screen, wherein one or more of the frames of image data represents an error screen indicating an error condition of one or more resource of a plurality of resources of a services system;
performing recognition processing using image data of the captured frames of image data to determine an error classifier associated to the error screen indicating an error condition of one or more resource of a plurality of resources of a services system, wherein the performing recognition processing includes using historical data of a data repository, the historical data including data of past errors of the services system;
determining one or more action based on the error classifier, wherein the determining includes using a decision data structure that cognitively maps, for first through Nth error classifiers, an error classifier to at least one action associated to the error classifier; and
performing the one or more action in response to the determining, wherein the one or more action is an action for correction of the error condition indicated by the error screen indicating an error condition of one or more resource of a plurality of resources of a services system.

### Claim 2 (depends on 1)

The method of claim 1, wherein the one or more action includes automatically installing a software patch to a software program associated to a services system resource.

### Claim 3 (depends on 1)

The method of claim 1, wherein the one or more action includes automatically installing a software patch to a software program associated to the error condition of one or more resource of a plurality of resources of the services system, wherein the automatically installing includes automatically sending the software patch to a proper address location using data returned by an IT management system of the services system that tracks a current location of the software program.

### Claim 4 (depends on 1)

The method of claim 1, wherein the one or more action includes automatically installing a software patch to a software program associated to a services system resource based on one or manually entered input by an administrator user in response to one or more prompt displayed on a displayed administer user interface associated to the administrator user.

### Claim 5 (depends on 1)

The method of claim 1, wherein the one or more action is an action to activate a logging profile for logging activity of a services system.

### Claim 6 (depends on 1)

The method of claim 1, wherein the one or more action is an action to activate a logging profile for logging activity of a services system, wherein the logging profile activated increases logging activity of the one or more resource having the error condition.

### Claim 7 (depends on 1)

The method of claim 1, wherein the method includes monitoring performance of the one or more action to yield a success rating parameter of the one or more action, applying the success rating parameter to a machine learning process, and updating the decision data structure using an output of the machine learning process.

### Claim 8 (depends on 1)

The method of claim 1, wherein the method includes monitoring performance of the one or more action to yield a success rating parameter of the one or more action, applying the success rating parameter to a machine learning process, and updating the decision data structure using an output of the machine learning process, wherein the success rating parameter is obtained using each of data returned by an IT management system that monitors performance of the services system, and survey data returned by an administrator user.

### Claim 9 (depends on 1)

The method of claim 1, wherein the method includes in response to a failure of performing recognition to determine the error classifier, sending a communication to a support administrator user prompting action by the support administrator user to register a new error classifier.

### Claim 10 (depends on 1)

The method of claim 1, wherein the method includes in response to the performing recognition processing opening a communication channel between a customer administrator user and a support administrator user, and wherein the performing the one or more action includes sending text based action selection data to the customer administrator user and the support administrator user.

### Claim 11 (depends on 1)

The method of claim 1, wherein the performing recognition processing includes determining that a current error screen matches a prior registered error screen.

### Claim 12 (depends on 1)

The method of claim 1, wherein the performing recognition processing includes determining that a current error screen matches a prior registered error screen, and wherein the determining includes using an image hash of the current error screen and an image hash of a prior registered error screen.

### Claim 13 (depends on 1)

The method of claim 1, wherein the performing recognition processing includes determining that a current error screen matches a prior registered error screen, wherein the determining includes using Natural Language Processing (NLP) parameters obtained by subjecting the current error screen to NLP processing, and using Natural Language Processing (NLP) parameters obtained by subjecting the prior registered error screen to NLP processing.

### Claim 14 (depends on 1)

The method of claim 1, wherein the performing recognition processing includes determining that context data of a current error screen matches context data of a prior registered error screen.

### Claim 15 (depends on 1)

The method of claim 1, wherein the performing recognition processing includes determining that context data of a current error screen matches context data of a prior registered error screen, wherein the context data indicates activity events of an administrator user within a time window that is associated to the current error screen.

### Claim 16 (depends on 1)

The method of claim 1, wherein the performing recognition processing includes determining that context data of an error associated to a current error screen matches prior registered context data of a prior registered error screen, wherein the context data of the error associated to the current error screen indicates a certain sequence of activity events by an administrator user, and wherein the prior registered context data of the prior registered error screen commonly indicates the same certain sequence of certain sequence of activity events by an administrator user.

### Claim 17 (depends on 1)

The method of claim 1, wherein the one or more action includes automatically installing a software patch to a software program associated to the error condition of one or more resource of a plurality of resources of the services system, wherein the automatically installing includes automatically sending the software patch to a proper address location using data returned by an IT management system of the services system that tracks a current location of the software program, wherein the method includes in response to the performing recognition processing opening a communication channel between a customer administrator user and a support administrator user, and wherein the performing the one or more action includes sending text based action selection data to the customer administrator user and the support administrator user.

### Claim 18 (depends on 1)

The method of claim 1, wherein the method includes in response to a failure of performing recognition to determine the error classifier, sending a communication to a support administrator user prompting action by the support administrator user to register a new error classifier, wherein the performing recognition processing includes determining that a current error screen matches a prior registered error screen, wherein the determining includes using Natural Language Processing (NLP) parameters obtained by subjecting the current error screen to NLP processing, and using Natural Language Processing (NLP) parameters obtained by subjecting the prior registered error screen to NLP processing, and wherein the obtaining iteratively captured frames of image data representing a user interface screen includes obtaining the iteratively captured frames in response to an error report activation by an administrator user using an administrator user interface defining the user interface screen, wherein the administrator user interface is displayed on an administrator client computer device that continuously buffers screen shots of the user interface screen, wherein the obtaining iteratively captured frames of image data representing a user interface screen includes obtaining screen shots of the buffered screen shots having capture times prior to a time of the error report activation by the administrator user.

### Claim 19

A computer program product comprising:
a computer readable storage medium readable by one or more processing circuit and storing instructions for execution by one or more processor for performing a method comprising:
obtaining iteratively captured frames of image data representing a user interface screen, wherein one or more of the frames of image data represents an error screen indicating an error condition of one or more resource of a plurality of resources of a services system;
performing recognition processing using image data of the captured frames of image data to determine an error classifier associated to the error screen indicating an error condition of one or more resource of a plurality of resources of a services system, wherein the performing recognition processing includes using historical data of a data repository, the historical data including data of past errors of the services system;
determining one or more action based on the error classifier, wherein the determining includes using a decision data structure that cognitively maps, for first through Nth error classifiers, an error classifier to at least one action associated to the error classifier; and
performing the one or more action in response to the determining, wherein the one or more action is an action for correction of the error condition indicated by the error screen indicating an error condition of one or more resource of a plurality of resources of a services system.

### Claim 20

A system comprising:
a memory;
at least one processor in communication with memory; and
program instructions executable by one or more processor via the memory to perform a method comprising:
obtaining iteratively captured frames of image data representing a user interface screen, wherein one or more of the frames of image data represents an error screen indicating an error condition of one or more resource of a plurality of resources of a services system;
performing recognition processing using image data of the captured frames of image data to determine an error classifier associated to the error screen indicating an error condition of one or more resource of a plurality of resources of a services system, wherein the performing recognition processing includes using historical data of a data repository, the historical data including data of past errors of the services system;
determining one or more action based on the error classifier, wherein the determining includes using a decision data structure that cognitively maps, for first through Nth error classifiers, an error classifier to at least one action associated to the error classifier; and
performing the one or more action in response to the determining, wherein the one or more action is an action for correction of the error condition indicated by the error screen indicating an error condition of one or more resource of a plurality of resources of a services system.
