# US 12,112,564 B2 · Contextual analysis for digital image processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,564 B2
- Document ID: US12112564B2
- Status: Granted
- Kind: B2
- Application number: 18/484,187
- Filed: 2023-10-10
- Published: 2024-02-01
- Granted: 2024-10-08
- Earliest claimed: 2021-05-25
- Assignees: Bank of America Corporation
- Applicants: Bank of America Corporation
- Inventors: Michael R. Young
- Web page: https://patentgrep.com/patent/US12112564B2

## CPC

- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06F 40/279: Recognition of textual entities
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06V 10/82: using neural networks
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A device configured to receive a request for processing a digital document that identifies a target data structure. The device is further configured to obtain the digital document and to process the digital document to identify objects within the digital document. The device is further configured to input the digital document into a machine learning model that is configured to output contextual metadata for the digital document based on the digital document. The contextual metadata describes attributes of the identified objects within the digital document. The device is further configured to populate a first subset of entries in the target data structure with identifiers for the objects identified within the digital document and to populate a second subset of entries in the target data structure with the contextual metadata. The device is further configured to send the populated target data structure to the network device.

## Claims

### Claim 1

A digital document analysis device, comprising:
a memory operable to store a plurality of data structures, wherein:
each data structure comprises a plurality of entries;
each entry is associated with a particular data type; and
a processor operably coupled to the network interface and the memory, and configured to:
receive a request for processing a digital document;
obtain the digital document;
process the digital document to identify objects within the digital document;
input the digital document into a machine learning model, wherein the machine learning model is configured to:
receive the digital document; and
output contextual metadata for the digital document based on the digital document, wherein the contextual metadata for the digital document describes attributes of the identified objects within the digital document;
identify a first subset of entries in a target data structure that are associated with a data type for the objects;
populate the first subset of entries in the target data structure with identifiers for the objects identified within the digital document;
identify a second subset of entries in the target data structure that are associated with a data type for contextual metadata;
populate the second subset of entries in the target data structure with the contextual metadata; and
send the populated target data structure to a network device.

### Claim 2 (depends on 1)

The device of claim 1, wherein:
the digital document is an image of a scene; and
the contextual metadata for the digital document identifies a distance between objects within the image.

### Claim 3 (depends on 1)

The device of claim 1, wherein:
the digital document is an image of a scene; and
the contextual metadata for the digital document identifies characteristics of people within the image.

### Claim 4 (depends on 1)

The device of claim 1, wherein:
the digital document is an image of a scene; and
the contextual metadata for the digital document identifies a time of the day within the image.

### Claim 5 (depends on 1)

The device of claim 1, wherein:
the digital document comprises text on a physical document; and
the contextual metadata for the digital document identifies characteristics of the physical document.

### Claim 6 (depends on 1)

The device of claim 1, wherein:
the digital document comprises a graphical representation of data; and
the contextual metadata for the digital document comprises a text describing the graphical representation of the data.

### Claim 7 (depends on 1)

The device of claim 1, wherein:
the digital document comprises text and hand-written annotations; and
the contextual metadata for the digital document identifies text associated with the hand-written annotations.

### Claim 8 (depends on 1)

The device of claim 1, wherein:
the digital document is generated by a first device; and
the processor is further configured to:
obtain sensor data that is associated with the digital document, wherein the sensor data is captured by a second device that is present when the digital document is generated;
input the sensor data into the machine learning model;
receive contextual metadata for the sensor data from the machine learning model;
identify a third subset of entries in the target data structure that are associated with a data type for contextual metadata for the sensor data; and
populate the third subset of entries in the target data structure with the contextual metadata for the sensor data.

### Claim 9 (depends on 8)

The device of claim 8, wherein:
the sensor data comprises an audio signal; and
the contextual metadata for the sensor data identifies words spoken in the audio signal.

### Claim 10 (depends on 8)

The device of claim 8, wherein:
the sensor data comprises a plurality of device identifiers; and
the contextual metadata for the sensors identifies user devices associated with the device identifiers.

### Claim 11

A digital document analysis method, comprising:
receiving a request for processing a digital document, wherein:
the request is associated with a target data structure; and
the target data structure comprises a plurality of entries that are each associated with a data type;
obtaining the digital document;
processing the digital document to identify objects within the digital document;
inputting the digital document into a machine learning model, wherein the machine learning model is configured to:
receive the digital document; and
output contextual metadata for the digital document based on the digital document, wherein the contextual metadata for the digital document describes attributes of the identified objects within the digital document;
identifying a first subset of entries in the target data structure that are associated with a data type for objects;
populating the first subset of entries in the target data structure with identifiers for the objects identified within the digital document;
identifying a second subset of entries in the target data structure that are associated with a data type for contextual metadata;
populating the second subset of entries in the target data structure with the contextual metadata; and
sending the populated target data structure to a network device.

### Claim 12 (depends on 11)

The method of claim 11, wherein:
the digital document is an image of scene; and
the contextual metadata for the digital document identifies a distance between objects within the image.

### Claim 13 (depends on 11)

The method of claim 11, wherein:
the digital document is an image of a scene; and
the contextual metadata for the digital document identifies characteristics of people within the image.

### Claim 14 (depends on 11)

The method of claim 11, wherein:
the digital document comprises text on a physical document; and
the contextual metadata for the digital document identifies characteristics of the physical document.

### Claim 15 (depends on 11)

The method of claim 11, wherein:
the digital document comprises text and hand-written annotations; and
the contextual metadata for the digital document identifies text associated with the hand-written annotations.

### Claim 16 (depends on 11)

The method of claim 11, wherein the digital document is generated by a first device; and
further comprising:
obtaining sensor data that is associated with the digital document, wherein the sensor data is captured by a second device that is present when the digital document is generated;
inputting the sensor data into the machine learning model;
receiving contextual metadata for the sensor data from the machine learning model;
identifying a third subset of entries in the target data structure that are associated with a data type for contextual metadata for the sensor data; and
populating the third subset of entries in the target data structure with the contextual metadata for the sensor data.

### Claim 17

A non-transitory computer-readable medium storing instructions that when executed by a processor cause the processor to:
receive a request for processing a digital document, wherein:
the request is associated with a target data structure;
the target data structure comprises a plurality of entries that are each associated with a data type;
obtain the digital document;
process the digital document to identify objects within the digital document;
input the digital document into a machine learning model, wherein the machine learning model is configured to:
receive the digital document; and
output contextual metadata for the digital document based on the digital document, wherein the contextual metadata for the digital document describes attributes of the identified objects within the digital document;
identify a first subset of entries in the target data structure that are associated with a data type for objects;
populate the first subset of entries in the target data structure with identifiers for the objects identified within the digital document;
identify a second subset of entries in the target data structure that are associated with a data type for contextual metadata;
populate the second subset of entries in the target data structure with the contextual metadata; and
send the populated target data structure to a network device.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the digital document is generated by a first device; and
the instructions further cause the processor to:
obtain sensor data that is associated with the digital document, wherein the sensor data is captured by a second device that is present when the digital document is generated;
input the sensor data into the machine learning model;
receive contextual metadata for the sensor data from the machine learning model;
identify a third subset of entries in the target data structure that are associated with a data type for contextual metadata for the sensor data; and
populate the third subset of entries in the target data structure with the contextual metadata for the sensor data.

### Claim 19 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein:
the digital document comprises text on a physical document; and
the contextual metadata for the digital document identifies characteristics of the physical document.

### Claim 20 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein:
the digital document comprises text and hand-written annotations; and
the contextual metadata for the digital document identifies text associated with the hand-written annotations.