# US 2023/0066906 A1 · Techniques for digital document analysis using document image fingerprinting

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0066906 A1
- Document ID: US20230066906A1
- Status: Published application
- Kind: A1
- Application number: 17/459,642
- Filed: 2021-08-27
- Published: 2023-03-02
- Earliest claimed: 2021-08-27
- Assignees: Optum, Inc.
- Applicants: Optum, Inc.
- Inventors: Nilav Baran Ghosh; Indranil Basu; Piyal De
- Web page: https://patentgrep.com/patent/US20230066906A1

## CPC

- G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06V 30/418: Document matching, e.g. of document images
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Various embodiments of the present invention provide methods, apparatus, systems, computing devices, computing entities, and/or the like for performing predictive structural analysis. Certain embodiments of the present invention utilize systems, methods, and computer program products that perform digital document analysis using at least one of spatial fingerprint document image pair distance measures for document image pairs, object fingerprint document image pair distance measures for document image pairs, and semantic fingerprint document image pair distance measures for document image pairs.

## Claims

### Claim 1

A computer-implemented method for determining a predicted document image pair distance measure for a document image pair, the computer-implemented method comprising:
identifying, using one or more processors, one or more document image bounding boxes for each document image data object in the document image pair;
identifying, using the one or more processors, a plurality of document image dimensions associated with the first document image data object and the second document image data object;
for each document image dimension, using the one or more processors:
determining a dimension-wise bounding box pair distance measure graph data object for each document image data object in the document pair, wherein the dimension-wise bounding box pair distance measure graph data object for a document image data object describes, for each bounding box pair selected from the one or more document image bounding boxes of the document image data object, a dimension-wise bounding box pair distance measure for the bounding box pair with respect to the document image dimension, and
determining a cross-graph document image pair distance measure between each dimension-wise bounding box pair distance measure graph data object for the document image dimension;
determining, using the one or more processors, a spatial fingerprint document image pair distance measure for the document image pair based at least in part on each cross-graph document image pair distance measure for a document image dimension;
determining, using the one or more processors, the predicted document image pair distance measure based at least in part on the spatial fingerprint document image pair distance measure; and
performing, using the one or more processors, one or more prediction-based actions based at least in part on the predicted document image pair distance measure.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein determining the cross-graph document image pair distance measure between a first dimension-wise bounding box pair distance measure graph data object and a second dimension-wise bounding box pair distance measure graph data object comprises:
determining a graph edit document image pair distance measure between the first dimension-wise bounding box pair distance measure graph data object and the second dimension-wise bounding box pair distance measure graph data object.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein determining the dimension-wise bounding box pair distance measure between a first document image bounding box and a second document image bounding box with respect to a first document image dimension comprises:
determining a first dimension-wise mean coordinate for the first document image bounding box with respect to the first document image dimension;
determining a second dimension-wise mean coordinate for the second document image bounding box with respect to the first document image dimension; and
determining the dimension-wise bounding box pair distance measure based at least in part on a distance measure between the first dimension-wise mean coordinate and the second dimension-wise mean coordinate.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein determining the first dimension-wise mean coordinate comprises:
determining a right-bottom coordinate for the first document image bounding box;
determining a left-top coordinate for the first document image bounding box; and
determining the first dimension-wise mean coordinate based at least in part on the right-bottom coordinate and the left-top coordinate.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein determining the predicted document image pair distance measure further comprises:
for each document image data object in the document image pair:
determining one or more image crops corresponding to the one or more document image bounding boxes of the document image data object;
generating one or more image crop embeddings based at least in part on the one or more image crops; and
generating an object fingerprint document image representation for the document image data object based at least in part on the one or more image crop embeddings;
determining an object fingerprint document image pair distance measure for the document image pair based at least in part on each object fingerprint document image representation for a document image data object; and
determining the predicted document image pair distance measure based at least in part on the object fingerprint document image pair distance measure.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein determining the object fingerprint document image pair distance measure based at least in part on a first object fingerprint document image representation and a second object fingerprint document image representation comprises:
performing a transposition operation on the second object fingerprint document image representation to generate a transposed object fingerprint document image representation; and
determining the object fingerprint document image pair distance measure based at least in part on an L2 document image pair distance measure between first object fingerprint document image representation and the transposed object fingerprint document image representation.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein determining the predicted document image pair distance measure further comprises:
for each document image data object in the document image pair:
identifying a textual representation of the document image data object; and
generating a semantic fingerprint document image representation of the document image data object based at least in part on the textual representation;
determining a semantic fingerprint document image pair distance measure for the document image pair based at least in part on each semantic fingerprint document image representation for a document image data object; and
determining the predicted document image pair distance measure based at least in part on the semantic fingerprint document image pair distance measure.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein determining the semantic fingerprint document image representation for a first document image data object comprises:
generating, based at least in part on the textual representation for the first document image data object and using a document encoding machine learning model, the semantic fingerprint document image representation.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein performing the one or more prediction-based actions comprises:
identifying a plurality of document image pairs that comprises the document image pair, wherein the plurality of document image pairs is associated with a plurality of document image data objects; and
determining a plurality of document image clusters of the plurality of document image data objects based at least in part on each predicted document image pair distance measure for a document image pair in the plurality of document image pairs.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more document image bounding boxes for each document image data object in the document image pair are generated by an optical character recognition engine based at least in part on the document image data object.

### Claim 11

An apparatus for determining a predicted document image pair distance measure for a document image pair, the apparatus comprising at least one processor and at least one memory including program code, the at least one memory and the program code configured to, with the processor, cause the apparatus to at least:
identify one or more document image bounding boxes for each document image data object in the document image pair;
identify a plurality of document image dimensions associated with the first document image data object and the second document image data object;
for each document image dimension:
determine a dimension-wise bounding box pair distance measure graph data object for each document image data object in the document pair, wherein the dimension-wise bounding box pair distance measure graph data object for a document image data object describes, for each bounding box pair selected from the one or more document image bounding boxes of the document image data object, a dimension-wise bounding box pair distance measure for the bounding box pair with respect to the document image dimension, and
determine a cross-graph document image pair distance measure between each dimension-wise bounding box pair distance measure graph data object for the document image dimension;
determine a spatial fingerprint document image pair distance measure for the document image pair based at least in part on each cross-graph document image pair distance measure for a document image dimension;
determine the predicted document image pair distance measure based at least in part on the spatial fingerprint document image pair distance measure; and
perform one or more prediction-based actions based at least in part on the predicted document image pair distance measure.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein determining the cross-graph document image pair distance measure between a first dimension-wise bounding box pair distance measure graph data object and a second dimension-wise bounding box pair distance measure graph data object comprises:
determining a graph edit document image pair distance measure between the first dimension-wise bounding box pair distance measure graph data object and the second dimension-wise bounding box pair distance measure graph data object.

### Claim 13 (depends on 11)

The apparatus of claim 11, wherein determining the dimension-wise bounding box pair distance measure between a first document image bounding box and a second document image bounding box with respect to a first document image dimension comprises:
determining a first dimension-wise mean coordinate for the first document image bounding box with respect to the first document image dimension;
determining a second dimension-wise mean coordinate for the second document image bounding box with respect to the first document image dimension; and
determining the dimension-wise bounding box pair distance measure based at least in part on a distance measure between the first dimension-wise mean coordinate and the second dimension-wise mean coordinate.

### Claim 14 (depends on 11)

The apparatus of claim 11, wherein determining the first dimension-wise mean coordinate comprises:
determining a right-bottom coordinate for the first document image bounding box;
determining a left-top coordinate for the first document image bounding box; and
determining the first dimension-wise mean coordinate based at least in part on the right-bottom coordinate and the left-top coordinate.

### Claim 15 (depends on 11)

The apparatus of claim 11, wherein determining the predicted document image pair distance measure further comprises:
for each document image data object in the document image pair:
determining one or more image crops corresponding to the one or more document image bounding boxes of the document image data object;
generating one or more image crop embeddings based at least in part on the one or more image crops; and
generating an object fingerprint document image representation for the document image data object based at least in part on the one or more image crop embeddings;
determining an object fingerprint document image pair distance measure for the document image pair based at least in part on each object fingerprint document image representation for a document image data object; and
determining the predicted document image pair distance measure based at least in part on the object fingerprint document image pair distance measure.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein determining the object fingerprint document image pair distance measure based at least in part on a first object fingerprint document image representation and a second object fingerprint document image representation comprises:
performing a transposition operation on the second object fingerprint document image representation to generate a transposed object fingerprint document image representation; and
determining the object fingerprint document image pair distance measure based at least in part on an L2 document image pair distance measure between first object fingerprint document image representation and the transposed object fingerprint document image representation.

### Claim 17 (depends on 11)

The apparatus of claim 11, wherein determining the predicted document image pair distance measure further comprises:
for each document image data object in the document image pair:
identifying a textual representation of the document image data object; and
generating a semantic fingerprint document image representation of the document image data object based at least in part on the textual representation;
determining a semantic fingerprint document image pair distance measure for the document image pair based at least in part on each semantic fingerprint document image representation for a document image data object; and
determining the predicted document image pair distance measure based at least in part on the semantic fingerprint document image pair distance measure.

### Claim 18 (depends on 17)

The apparatus of claim 17, wherein determining the semantic fingerprint document image representation for a first document image data object comprises:
generating, based at least in part on the textual representation for the first document image data object and using a document encoding machine learning model, the semantic fingerprint document image representation.

### Claim 19 (depends on 11)

The apparatus of claim 11, wherein performing the one or more prediction-based actions comprises:
identifying a plurality of document image pairs that comprises the document image pair, wherein the plurality of document image pairs is associated with a plurality of document image data objects; and
determining a plurality of document image clusters of the plurality of document image data objects based at least in part on each predicted document image pair distance measure for a document image pair in the plurality of document image pairs.

### Claim 20

A computer program product for determining a predicted document image pair distance measure for a document image pair, the computer program product comprising at least one non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions configured to:
identify one or more document image bounding boxes for each document image data object in the document image pair;
identify a plurality of document image dimensions associated with the first document image data object and the second document image data object;
for each document image dimension:
determine a dimension-wise bounding box pair distance measure graph data object for each document image data object in the document pair, wherein the dimension-wise bounding box pair distance measure graph data object for a document image data object describes, for each bounding box pair selected from the one or more document image bounding boxes of the document image data object, a dimension-wise bounding box pair distance measure for the bounding box pair with respect to the document image dimension, and
determine a cross-graph document image pair distance measure between each dimension-wise bounding box pair distance measure graph data object for the document image dimension;
determine a spatial fingerprint document image pair distance measure for the document image pair based at least in part on each cross-graph document image pair distance measure for a document image dimension;
determine the predicted document image pair distance measure based at least in part on the spatial fingerprint document image pair distance measure; and
perform one or more prediction-based actions based at least in part on the predicted document image pair distance measure.
