# US 2020/0402658 A1 · User-aware explanation selection for machine learning systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0402658 A1
- Document ID: US20200402658A1
- Status: Published application
- Kind: A1
- Application number: 16/447,393
- Filed: 2019-06-20
- Published: 2020-12-24
- Earliest claimed: 2019-06-20
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Richard J. Tomsett; Graham White; David S. Braines
- Web page: https://patentgrep.com/patent/US20200402658A1

## CPC

- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N 20/00: Machine learning
- G16H 50/30: for calculating health indices; for individual health risk assessment
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems, computer-implemented methods and computer program products are provided facilitating explanation selection for machine learning system output. In one embodiment, the computer-implemented method includes: accessing, by a device operatively coupled to a processor, a plurality of explanation generation components to generate different types of explanations of a machine learning output; monitoring, by the device, indicators of an entity receiving the machine learning system output to obtain user emotional state descriptors; and providing, by the device, an explanation from a selected explanation generation component to a user interface for evaluation of the machine learning system output with the explanation.

## Claims

### Claim 1

A system, comprising:
a memory that stores computer executable components;
a processor, operably coupled to the memory, and that executes the computer executable components stored in the memory, wherein the computer executable components comprise:
an explanation selection component that accesses a plurality of explanation generation components to generate different types of explanations of a machine learning output; and
a monitoring component that monitors indicators of an entity receiving the machine learning system output to obtain user emotional state descriptors, and wherein the explanation selection component also provides an explanation from a selected explanation generation component to a user interface for evaluation of the machine learning system output with the explanation.

### Claim 2 (depends on 1)

The system of claim 1, wherein the explanation selection component also accesses metadata for the explanation generation component, wherein the metadata defines one or more types of explanations, and wherein the metadata comprises information indicative of a level of cognitive engagement required by the entity for the one or more types of explanations.

### Claim 3 (depends on 1)

The system of claim 1, wherein the indicators comprise biological or behavioral indicators, and further comprising an actuation component that actuates the selected explanation generator that provides an explanation of the machine learning system output.

### Claim 4 (depends on 3)

The system of claim 3, wherein the biological and behavioral indicators comprise at least one of: a galvanic skin response; pupil dilation; a heart rate; blood oxygen level; an electroencephalogram; an electromyography; skin temperature; or a behavioral pattern.

### Claim 5 (depends on 2)

The system of claim 2, wherein the metadata comprises information indicative of a level of cognitive engagement required by the entity for the type of explanation.

### Claim 6 (depends on 1)

The system of claim 1, wherein the monitoring component adds content to the query, and wherein the content added is based on the entity.

### Claim 7 (depends on 1)

The system of claim 1, wherein the one or more explanations are annotated with metadata that describes the explanation generation component in terms of stored answer preferences and a state of the entity.

### Claim 8 (depends on 1)

The system of claim 1, wherein the computer executable components further comprise a user reaction component that monitors user reaction to a machine learning system outcome explanation and provides feedback to learn and update user preferences.

### Claim 9 (depends on 1)

The system of claim 1, wherein the explanation generators provide different types of explanations comprising text-based explanations, visual explanations, in-depth explanations, summary explanations, analogy explanations, and counterfactual explanations.

### Claim 10 (depends on 1)

The system of claim 1, wherein the monitoring component considers one or more preferences of the entity and a state of the entity to determine a current entity description, and detects the state of the entity based on stress level, concentration level, calmness, skin response, pupil dilation, or heart rate, and wherein the one or more preferences are set by the entity or learned by the explanation selection component based on a quality of a decision made by the entity when provided different explanations.

### Claim 11 (depends on 4)

The system of claim 4, wherein the monitoring component uses a plurality of sensors to monitor galvanic skin response, pupil dilation, heart rate, blood oxygen level, electroencephalogram, electromyography, or behavioral patterns for the entity.

### Claim 12

A computer implemented method, comprising;
accessing, by a device operatively coupled to a processor, a plurality of explanation generation components to generate different types of explanations of a machine learning output;
monitoring, by the device, indicators of an entity receiving the machine learning system output to obtain user emotional state descriptors; and
providing, by the device, an explanation from a selected explanation generation component to a user interface for evaluation of the machine learning system output with the explanation.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, wherein the accessing comprises accessing metadata for the explanation generation component, wherein the metadata defines one or more types of explanations, and wherein the metadata comprises information indicative of a level of cognitive engagement required by the entity for the one or more types of explanations.

### Claim 14 (depends on 12)

The computer-implemented method of claim 12, wherein the indicators comprise biological or behavioral indicators, and the computer-implemented method further comprising:
actuating the selected explanation generator to provide an explanation of the machine learning system output.

### Claim 15 (depends on 12)

The computer implemented method of claim 12, further comprising:
comparing, by the device, employing constraint solving user emotional state descriptors to the generator metadata to match to a desired explanation type.

### Claim 16 (depends on 15)

The computer implemented method of claim 15, further comprising:
accessing, by the device, user preferences for types of explanation and including the user preferences during the comparing to select an explanation generation component; and
monitoring, by the device, user reaction to a machine learning system outcome explanation and providing feedback to the entity to learn and update the user preferences.

### Claim 17 (depends on 14)

The computer-implemented method of claim 14, wherein the biological and behavioral indicators comprise at least one of: a galvanic skin response; pupil dilation; a heart rate; blood oxygen level; an electroencephalogram; an electromyography; skin temperature; or a behavioral pattern.

### Claim 18

A computer program product facilitating explanation selection for machine learning system output, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:
access a plurality of explanation generation components to generate different types of explanations of a machine learning output;
monitor indicators of an entity receiving the machine learning system output to obtain user emotional state descriptors; and
provide an explanation from a selected explanation generation component to a user interface for evaluation of the machine learning system output with the explanation.

### Claim 19 (depends on 18)

The computer program product of claim 18, wherein an accessing of comprises accessing metadata for the explanation generation component, wherein the metadata defines one or more types of explanations, and wherein the metadata comprises information indicative of a level of cognitive engagement required by the entity for the one or more types of explanations.

### Claim 20 (depends on 18)

The computer program product of claim 18, wherein the indicators comprise biological or behavioral indicators, and wherein the program instructions are executable by the processor to cause the processor to:
actuate the selected explanation generator to provide an explanation of the machine learning system output; and
compare employing constraint solving user emotional state descriptors to the generator metadata to match to a desired explanation type.
