# US 2022/0285024 A1 · Facilitating interpretability of classification model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0285024 A1
- Document ID: US20220285024A1
- Status: Published application
- Kind: A1
- Application number: 17/637,831
- Filed: 2020-08-31
- Published: 2022-09-08
- Earliest claimed: 2019-08-29
- Assignees: Koninklijke Philips N.V.
- Applicants: Koninklijke Philips N.V.
- Inventors: Nicola PEZZOTTI; Jacek Lukasz KUSTRA
- Web page: https://patentgrep.com/patent/US20220285024A1

## CPC

- G06N 7/00: Computing arrangements based on specific mathematical models
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 10/60: for patient-specific data, e.g. for electronic patient records
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

A system and computer-implemented method are provided for generating a visualization of the classification uncertainty of a classification model which is applied to clinical data, wherein said visualization is provided in a lower-dimensional space which is obtained by applying a non-linear and manifold preserving dimensionality reduction technique to feature vectors of the clinical data. The visualization techniques consider the classification model as a ‘black box’ by not being dependent on internal parameters of the classification model.

## Claims

### Claim 1

A system configured to apply a classification model to clinical data, the system comprising:
a data interface for accessing:
clinical data comprising data instances which are each representable as a feature vector in a multi-dimensional feature space;
a classification model configured to be applied to the feature vector to provide a classification of the respective data instance;
a processor subsystem configured to:
apply a non-linear and manifold-preserving dimensionality reduction technique to all or a subset of the feature vectors to obtain a plurality of clinical data points in a lower-dimensional space wherein the lower-dimensional space is a two-dimensional space;
create synthetic data points in the lower-dimensional space and determine feature vectors for the synthetic data points by applying an interpolation technique to the feature vectors of the clinical data points, thereby obtaining an interpolated feature vector for each of the synthetic clinical data points;
for each synthetic clinical data point:
apply the classification model to the respective interpolated feature vector to obtain a classification for the synthetic clinical data point, and
determine a classification uncertainty of the classification; and
generate a visualization of the lower-dimensional space for display to a user, wherein the visualization comprises a visualization of the classification uncertainty in visual relation to the synthetic clinical data points
generate, in the visualization of the lower-dimensional space, a visualization of the classification by the classification model; and
generate the visualization as a 2D image in which the classification uncertainty is assigned to a visual property of respective pixels of the 2D image.

### Claim 2

(canceled)

### Claim 3

(canceled)

### Claim 4 (depends on 3)

The system according to claim 3, wherein the visual property is a saturation or an intensity of a respective pixel.

### Claim 5 (depends on 4)

The system according to claim 4, further comprising a user interface subsystem comprising:
a display output for displaying said visualizations; and
a user input interface for receiving user input data from a user input device operable by a user;
wherein the processor subsystem is configured to, via the user interface subsystem, enable a user to select a synthetic clinical data point, and in response to said selection, provide a visualization of the respective interpolated feature vector.

### Claim 6 (depends on 5)

The system according to claim 5, wherein the processor subsystem is configured to, via the user interface subsystem, enable the user to select two synthetic clinical data points, and in response to said selection, provide a visualization of a difference between the respective interpolated feature vectors.

### Claim 7 (depends on 1)

The system according to claim 1, wherein the classification model is trained on training data, and wherein the clinical data for which the visualization is provided is the training data of the classification model.

### Claim 8 (depends on 7)

The system according to claim 7, wherein all or a subset of the data instance of the training data comprise or are associated with a respective ground truth classification, and wherein the processor subsystem is configured to generate a visualization of the ground truth classification in visual relation with the clinical data points in the visualization of the lower-dimensional space.

### Claim 9 (depends on 1)

The system according to claim 1, wherein the data interface is configured to access further clinical data, and wherein the processor subsystem is configured to:
generate further clinical data points representing the further clinical data in the lower-dimensional space; and
visualize the further clinical data points in the visualization of the lower-dimensional space.

### Claim 10 (depends on 1)

The system according to claim 1, wherein the processor subsystem is configured to determine the classification and the classification uncertainty, and to visualize the classification uncertainty, for a regular grid of synthetic clinical data points in the lower-dimensional space.

### Claim 11 (depends on 1)

The system according to claim 1, wherein the non-linear and manifold-preserving dimensionality reduction technique is a t-distributed Stochastic Neighbor Embedding (t-SNE) algorithm.

### Claim 12 (depends on 1)

The system according to claim 1, wherein applying the interpolation technique comprises using a KD-tree algorithm to search for clinical data points to be used in the interpolation.

### Claim 13 (depends on 1)

A workstation or imaging apparatus comprising the system according to claim 1.

### Claim 14

A computer-implemented method for applying a classification model to clinical data, comprising:
accessing:
clinical data comprising data instances which are each representable as a feature vector in a multi-dimensional feature space;
a classification model configured to be applied to the feature vector to provide a classification of the respective data instance;
applying a non-linear and manifold-preserving dimensionality reduction technique to all or a subset of the feature vectors to obtain a plurality of clinical data points in a lower-dimensional space, wherein the lower-dimensional space is a two-dimensional space;
creating synthetic data points in the lower-dimensional space and determining feature vectors for the synthetic data points by applying an interpolation technique to the feature vectors of the clinical data points, thereby obtaining an interpolated feature vector for each of the synthetic clinical data points;
for each synthetic clinical data point:
applying the classification model to the respective interpolated feature vector to obtain a classification for the synthetic clinical data point, and
determining a classification uncertainty of the classification; and
generating a visualization of the lower-dimensional space for display to a user, wherein the visualization comprises a visualization of the classification uncertainty in visual relation to the synthetic clinical data points, wherein the visualization is generated as a 2D image in which the classification uncertainty is assigned to a visual property of respective pixels of the 2D image; and
generating, in the visualization of the lower-dimensional space, a visualization of the classification by the classification model.

### Claim 15 (depends on 14)

A computer-readable medium comprising transitory or non-transitory data representing a computer program, the computer program comprising instructions for causing a processor system to perform the method of claim 14.
