# US 2022/0027610 A1 · Classifying pharmacovigilance documents using image analysis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0027610 A1
- Document ID: US20220027610A1
- Status: Published application
- Kind: A1
- Application number: 17/345,632
- Filed: 2021-06-11
- Published: 2022-01-27
- Earliest claimed: 2020-07-24
- Assignees: Bristol-Myers Squibb Company
- Applicants: Bristol-Myers Squibb Company
- Inventors: Sameen Mayur DESAI
- Web page: https://patentgrep.com/patent/US20220027610A1

## CPC

- G06V 30/40: Document-oriented image-based pattern recognition
- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06F 16/55: Clustering; Classification
- G06F 16/583: using metadata automatically derived from the content
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06V 2201/10: Recognition assisted with metadata
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Provided herein are system, apparatus, device, method, and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for classifying a document using image analysis. In an embodiment, a server may train and implement a deep learning model to classify documents using image representations of the documents.

## Claims

### Claim 1

A method for automatically classifying documents, the method comprising:
retrieving, by one or more computing devices, a set of documents and metadata for each document in the set of documents;
converting, by the one or more computing devices, each document of the set of documents into an image file, resulting in a set of image files;
training, by the one or more computing devices, a learning engine to classify each document of the set of documents using the set of image files and the metadata corresponding to each respective document of the set of documents;
receiving, by the one or more computing devices, a request to classify a different document;
converting, by the one or more computing devices, the different document into a new image file;
blending, by the one or more computing devices, pages of the different image file into a single image stack;
assigning, by the one or more computing devices, using the trained learning engine, a first, second, and third classification to the different document based on the single image stack; and
determining, by the one or more computing devices, a priority value of the different document based on one or more of the first, second, or third classification.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first classification corresponds to the validity of the different document, the second classification corresponds to an indicator of a seriousness of a content on the different document, the third classification corresponds to one or more elements corresponding to the seriousness of the content.

### Claim 3 (depends on 1)

The method of claim 1, wherein training the learning engine comprises:
recursively:
identifying, by the one or more computing devices, an attribute associated with each image file of the set of image files by executing image analysis on each corresponding image file of the set of image files;
assigning, by the or more computing devices, using the learning engine, the first, second, and third classification to each document of the set of documents based on the identified attribute for each image file of the set of image files; and
validating, by the one or more computing devices, the classification of each of a subset of documents of the set of documents by comparing the metadata for each document to the classification of each document of the set of documents.

### Claim 4 (depends on 3)

The method of claim 3, wherein executing image analysis comprises:
extracting, by the one or more computing devices, a set of features from each image file of the set of image files;
generating, by the one or more computing devices, a feature map for each image file of the set of image files, the feature map for each image file including the set of features extracted from the image file;
generating, by the one or more computing devices, a reduced feature map for each image file of the set of image files by reducing a dimensionality characteristic of the feature map for each image file of the set of image files;
transforming, by the one or more computing devices, the reduced feature map for each image file of the set of image files into a vector; and
performing, by the one or more computing devices, a non-linear transformation on the vector for each image file of the set of image files to identify the attribute associated with each image file of the set of image files corresponding to each image file of the set of image files.

### Claim 5 (depends on 1)

The method of claim 1, wherein the learning engine implements a supervised learning algorithm.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
identifying, by the one or more computing devices, a document type for a document of the set of documents that is different than a predetermined document type; and
converting, by the one or more computing devices, the document to the predetermined document type.

### Claim 7

A system comprising:
a memory;
a processor coupled to the memory, the processor configured to:
retrieve a set of documents and metadata for each document in the set of documents;
convert each document of the set of documents into an image file, resulting in a set of image files;
train a learning engine to classify each document of the set of documents using the set of image files and the metadata corresponding to each respective document of the set of documents;
receive a request to classify a different document;
convert the different document into a new image file;
blend pages of the different image file into a single image stack;
assign using the trained learning engine, a first, second, and third classification to the different document based on the single image stack; and
determine a priority value of the different document based on one or more of the first, second, or third classification.

### Claim 8 (depends on 7)

The system of claim 7, wherein the first classification corresponds to the validity of the different document, the second classification corresponds to an indicator of a seriousness of a content on the different document, the third classification corresponds to one or more elements corresponding to the seriousness of the content.

### Claim 9 (depends on 7)

The system of claim 7, wherein training the learning engine comprises:
recursively:
identifying an attribute associated with each image file of the set of image files by executing image analysis on each corresponding image file of the set of image files;
assigning using the learning engine, the first, second, and third classification to each document of the set of documents based on the identified attribute for each image file of the set of image files; and
validating the classification of each of a subset of documents of the set of documents by comparing the metadata for each document to the classification of each document of the set of documents.

### Claim 10 (depends on 9)

The system of claim 9, wherein executing image analysis comprises:
extracting a set of features from each image file of the set of image files;
generating a feature map for each image file of the set of image files, the feature map for each image file including the set of features extracted from the image file;
generating a reduced feature map for each image file of the set of image files by reducing a dimensionality characteristic of the feature map for each image file of the set of image files;
transforming the reduced feature map for each image file of the set of image files into a vector; and
performing a non-linear transformation on the vector for each image file of the set of image files to identify the attribute associated with each image file of the set of image files corresponding to each image file of the set of image files.

### Claim 11 (depends on 7)

The system of claim 7, wherein the learning engine implements a supervised learning algorithm.

### Claim 12 (depends on 7)

The system of claim 7, wherein the processor is further configured to:
identifying a document type for a document of the set of documents that is different than a predetermined document type; and
converting the document to the predetermined document type.

### Claim 13

A non-transitory computer-readable medium having instructions stored thereon, execution of which, by one or more processors of a device, cause the one or more processors to perform operations comprising:
retrieving a set of documents and metadata for each document in the set of documents;
converting each document of the set of documents into an image file, resulting in a set of image files;
training a learning engine to classify each document of the set of documents using the set of image files and the metadata corresponding to each respective document of the set of documents;
receiving a request to classify a different document;
converting the different document into a new image file;
blending pages of the different image file into a single image stack; and
assigning using the trained learning engine, a first, second, and third classification to the different document based on the single image stack.

### Claim 14 (depends on 13)

The non-transitory computer-readable medium of claim 13, the first classification corresponds to the validity of the different document, the second classification corresponds to an indicator of a seriousness of a content on the different document, the third classification corresponds to one or more elements corresponding to the seriousness of the content.

### Claim 15 (depends on 13)

The non-transitory computer-readable medium of claim 13, wherein training the learning engine comprises:
recursively:
identifying an attribute associated with each image file of the set of image files by executing image analysis on each corresponding image file of the set of image files;
assigning using the learning engine, the first, second, and third classification to each document of the set of documents based on the identified attribute for each image file of the set of image files; and
validating the classification of each of a subset of documents of the set of documents by comparing the metadata for each document to the classification of each document of the set of documents.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein executing image analysis comprises:
extracting a set of features from each image file of the set of image files;
generating a feature map for each image file of the set of image files, the feature map for each image file including the set of features extracted from the image file;
generating a reduced feature map for each image file of the set of image files by reducing a dimensionality characteristic of the feature map for each image file of the set of image files;
transforming the reduced feature map for each image file of the set of image files into a vector; and
performing a non-linear transformation on the vector for each image file of the set of image files to identify the attribute associated with each image file of the set of image files corresponding to each image file of the set of image files.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the processor is further configured to:
identifying a document type for a document of the set of documents that is different than a predetermined document type; and
converting the document to the predetermined document type.

### Claim 18

A device comprising:
a memory; and
a processor coupled to the memory, the processor configured to:
retrieve a set of documents and metadata for each document in the set of documents;
convert each document of the set of documents into an image file, resulting in a set of image files;
train a learning engine to classify each document of the set of documents using the set of image files and the metadata corresponding to each respective document of the set of documents;
receive a request to classify a different document;
blend pages of the different document into a single image stack;
assign using the trained learning engine, a classification to the different document based on the single image stack; and
cause display of the classification.

### Claim 19 (depends on 18)

The device of claim 18, wherein training the learning engine comprises:
recursively:
identifying an attribute associated with each image file of the set of image files by executing image analysis on each corresponding image file of the set of image files;
assigning using the learning engine, the first, second, and third classification to each document of the set of documents based on the identified attribute for each image file of the set of image files; and
validating the classification of each of a subset of documents of the set of documents by comparing the metadata for each document to the classification of each document of the set of documents.

### Claim 20 (depends on 18)

The device of claim 18, wherein the processor is configured to:
extract a set of features from each image file of the set of image files;
generate a feature map for each image file of the set of image files, the feature map for each image file including the set of features extracted from the image file;
generate a reduced feature map for each image file of the set of image files by reducing a dimensionality characteristic of the feature map for each image file of the set of image files;
transform the reduced feature map for each image file of the set of image files into a vector; and
perform a non-linear transformation on the vector for each image file of the set of image files to identify the attribute associated with each image file of the set of image files corresponding to each image file of the set of image files.

### Claim 21 (depends on 18)

The device of claim 18, further comprising a camera coupled to the processor, wherein the processor is further configured to: receive an image of the different document captured by the camera.

### Claim 22 (depends on 21)

The device of claim 21, wherein the blending of the pages of the different document into a single image stack includes blending the pages of the image of the different document.

### Claim 23 (depends on 18)

The device of claim 18, wherein the classification indicates a seriousness of a content of the different document.

### Claim 24 (depends on 18)

The device of claim 18, wherein the different document includes Personal Identifiable Information (PII).

### Claim 25 (depends on 18)

The device of claim 18, wherein the processor is further configured to determine a recommendation for a user corresponding to the different document based on the classification.

### Claim 26 (depends on 18)

The device of claim 18, wherein the processor is configured to receive the different document from an Internet of Things (IoT) device communicatively coupled to the device.

### Claim 27 (depends on 18)

The device of claim 18, wherein the learning engine resides on the device.
