# US 11,907,977 B2 · Collaborative text detection and text recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,907,977 B2
- Document ID: US11907977B2
- Status: Granted
- Kind: B2
- Application number: 17/972,499
- Filed: 2022-10-24
- Published: 2023-04-27
- Granted: 2024-02-20
- Earliest claimed: 2021-10-27
- Assignees: Zaru, Inc.
- Applicants: Zaru, Inc.
- Inventors: Susheel Khamkar; Lino Fabiani; Edwin Villanueva
- Web page: https://patentgrep.com/patent/US11907977B2

## CPC

- G06Q 30/04: Billing or invoicing
- G06F 16/93: Document management systems
- G06F 30/27: using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
- G06F 40/117: Tagging; Marking up (details of markup languages G06F40/143); Designating a block; Setting of attributes (style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154)
- G06F 40/186: Templates
- G06Q 10/063112: {Skill-based matching of a person or a group to a task}
- G06V 10/225: {based on a marking or identifier characterising the area}
- G06V 10/95: {structured as a network, e.g. client-server architectures}
- G06V 30/1463: {Orientation detection or correction, e.g. rotation of multiples of 90 degrees}
- G06V 30/19013: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching (specially adapted for image segmentation G06T7/10; specially adapted for the analysis of motion G06T7/20; specially adapted for image alignment G06T7/30; specially adapted for the calculation of depth from stereo images G06T7/50; specially adapted for position determination G06T7/70)}
- G06V 30/19133: {Interactive pattern learning with a human teacher}
- G06V 30/19147: {Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting}
- G06V 30/1916: {Validation; Performance evaluation}
- G06V 30/19167: {Active pattern learning}
- G06V 30/1983: {Syntactic or structural pattern recognition, e.g. symbolic string recognition}
- G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06V 30/416: Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
- G06V 30/418: Document matching, e.g. of document images
- G06V 30/10: Character recognition
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Described are approaches for assigning tasks between machine resources (e.g., AI task performers, AI task validators), human resources (e.g., task performers, task validators), and/or other smart systems to facilitate collaborative text detection, text recognition, and text retrieval in order to optimize system performance along a variety of different selection criteria specifying various performant dimensions, including, but not limited to improving system efficiency, reducing task performer and/or task validator idle time, improving triage outcomes, reducing data processing loads, maintaining client confidentiality, etc., that may be associated with one or more customers.

## Claims

### Claim 1

A computing system, comprising:
at least one computing processor; and
memory including instructions that, when executed by the at least one computing processor, enable the computing system to:
obtain training data comprising representations of a plurality of invoices associated with a vendor template, the vendor template including a plurality of document fields for a type of invoice;
train a vendor detection model on the plurality of invoices to generate a trained vendor detection model, the trained vendor detection model operable to generate a vendor identification score quantifying a degree of similarity for candidate vendor templates, the trained vendor detection model constructed according to a machine learning model;
obtain image data that includes a representation of a plurality of documents;
orientate individual documents of the plurality of documents with respect to a reference point to generate machine-orientated documents;
maintain the machine-orientated documents at a task performer queue associated with a task performer network;
receive first task performer information from the task performer network, the first task performer information including, at least a portion of the machine-orientated documents being associated with a merge document tag;
merge documents associated with the merge document tag to generate a merged document, the merged document comprising an invoice;
evaluate the trained vendor detection model on the invoice to match the invoice to a stored invoice associated with vendor information;
retrieve the vendor information, wherein the vendor information is associated with a configuration file specifying a set of document fields associated with a plurality of regions of text;
recognize text of the plurality of regions of text using a machine-based optical character recognition engine to generate recognized text associated with the plurality of regions of text;
segment the recognized text into a plurality of subgroups of recognized text based on the configuration file and the plurality of regions of text;
maintain the plurality of subgroups of recognized text at the task performer queue;
cause an overlay of at least one subgroup of the plurality of subgroups of recognized text to be presented as user-selectable elements to a human task performer associated with the task performer network, the overlay being presented with a vendor template associated with the invoice;
enable modification of the at least one subgroup of recognized text by the human task performer;
receive second human reviewer information corresponding to a modified version of the at least one subgroup of recognized text;
validate the modified Version of the at least one subgroup of recognized text to determine machine-validated invoice entries;
maintain the machine-validated invoice entries at a human task validator queue associated with a task validation network;
receive human task validator information approving the machine-validated invoice entries; and
generate at least one of itemized data or vendor template training data from validated machine-validated invoice entries.

### Claim 2 (depends on 1)

The computing system of claim 1, wherein the instructions when executed by the at least one computing processor further enable the computing system to:
scan the plurality of documents for a flag, the flag identifying one of personally identifiable information or an unauthorized document type; and
maintain documents associated with the flag at the human task validator queue for further processing.

### Claim 3 (depends on 1)

The computing system of claim 1, wherein the instructions when executed by the at least one computing processor further enable the computing system to:
execute a document merge function on documents flagged as being related to generate merged documents.

### Claim 4 (depends on 1)

The computing system of claim 1, wherein the instructions when executed by the at least one computing processor further enable the computing system to:
detect a plurality of document fields; and
analyze the plurality of document fields to locate a plurality of regions of text represented in the plurality of document fields.

### Claim 5 (depends on 4)

The computing system of claim 4, wherein the instructions when executed by the at least one computing processor further enable the computing system to:
apply a bounding box around each document field.

### Claim 6 (depends on 1)

The computing system of claim 1, wherein the instructions when executed by the at least one computing processor further enable the computing system to:
assign at least one subgroup of recognized text of the plurality of subgroups of recognized text to a human task performer of a plurality of human task performers based on performance information associated with the plurality of human task performers.

### Claim 7 (depends on 1)

The computing system of claim 1, wherein modification includes an addition of text to the recognized text, a removal of recognized text, or a change to the recognized text.

### Claim 8 (depends on 1)

The computing system of claim 1, wherein a document field includes one of quantity field, a description field, a rate field, product code field, unit of measure field, or an amount field.

### Claim 9 (depends on 1)

The computing system of claim 1, wherein the instructions when executed by the at least one computing processor further enable the computing system to:
receive modification of the text by at the human task performer to generate modified text; and
generate a consensus string of text comprising at least a portion of the text recognized using the machine-based optical character recognition engine and the modified text.

### Claim 10 (depends on 1)

The computing system of claim 1, wherein the instructions when executed by the at least one computing processor further enable the computing system to:
iteratively train the vendor detection model using the training data.

### Claim 11 (depends on 1)

The computing system of claim 1, wherein the instructions when executed by the at least one computing processor further enable the computing system to:
enable a text edit input region for the text of the plurality of regions of text.

### Claim 12 (depends on 1)

The computing system of claim 1, wherein the instructions when executed by the at least one computing processor further enable the computing system to:
provide instructions on how to review the at least one subgroup of recognized text to the human task performer.

### Claim 13 (depends on 1)

The computing system of claim 1, wherein the instructions when executed by the at least one computing processor further enable the computing system to:
compare recognized text with the modified version of the at least one subgroup of recognized text received from the human task performer; and
determine an accuracy of the human task performer.

### Claim 14 (depends on 1)

The computing system of claim 1, wherein the image data includes a representation of a financial statement.

### Claim 15

A computer-implemented method, comprising:
obtaining training data comprising representations of a plurality of invoices associated with a vendor template, the vendor template including a plurality of document fields for a type of invoice;
training a vendor detection model on the plurality of invoices to generate a trained vendor detection model, the trained vendor detection model operable to generate a vendor identification score quantifying a degree of similarity for candidate vendor templates, the trained vendor detection model constructed according to a machine learning model;
obtaining image data that includes a representation of a plurality of documents;
orientating individual documents of the plurality of documents with respect to a reference point to generate machine-orientated documents;
maintaining the machine-orientated documents at a task performer queue associated with a task performer network;
receiving first task performer information from the task performer network, the first task performer information including at least a portion of the machine-orientated documents being associated with a merge document tag;
merging documents associated with the merge document tag to generate a merged document, the merged document comprising an invoice;
evaluating the trained vendor detection model on the invoice to match the invoice to a stored invoice associated with vendor information;
retrieving the vendor information, wherein the vendor information is associated with a configuration file specifying a set of document fields associated with a plurality of regions of text;
recognizing text of the plurality of regions of text using a machine-based optical character recognition engine to generate recognized text associated with the plurality of regions of text;
segmenting the recognized text into a plurality of subgroups of recognized text based on the configuration file and the plurality of regions of text;
maintaining the plurality of subgroups of recognized text at the task performer queue;
causing an overlay a at least one subgroup of the plurality of subgroups of recognized text to be presented as user-selectable elements to a human task performer associated with the task performer network, the overlay being presented with a vendor template associated with the invoice;
enabling modification of the at least one subgroup of recognized text by the human task performer;
receiving second human reviewer information corresponding to a modified version of the at least one subgroup of recognized text;
validating the modified version of the at least one subgroup of recognized text to determine machine-validated invoice entries;
maintaining the machine-validated invoice entries at a human task validator queue associated with a task validation network;
receiving human task validator information approving the machine-validated invoice entries; and
generating at least one of itemized data or vendor template training data from validated machine-validated invoice entries.

### Claim 16 (depends on 15)

The computer-implemented method of claim 15, further comprising:
scanning the plurality of documents for a flag, the flag identifying one of personally identifiable information or an unauthorized document type; and
maintaining documents associated with the flag at the human task validator queue for further processing.

### Claim 17 (depends on 15)

The computer-implemented method of claim 15, further comprising:
detecting a plurality of document fields; and
analyzing the plurality of document fields to locate a plurality of regions of text represented in the plurality of document fields.

### Claim 18 (depends on 15)

The computer-implemented method of claim 15, further comprising:
receiving modification of the text by at the human task performer to generate modified text; and
generating a consensus string of text comprising at least a portion of the text recognized using the machine-based optical character recognition engine and the modified text.

### Claim 19 (depends on 15)

The computer-implemented method of claim 15, further comprising:
iteratively train the vendor detection model using the training data.

### Claim 20

A non-transitory computer readable storage medium storing instructions that, when executed by at least one processor of a computing system, causes the computing system to:
obtain training data comprising representations of a plurality of invoices associated with a vendor template, the vendor template including a plurality of document fields for a type of invoice;
train a vendor detection model on the plurality of invoices to generate a trained vendor detection model, the trained vendor detection model operable to generate a vendor identification score quantifying a degree of similarity for candidate vendor templates, the trained vendor detection model constructed according to a machine learning model;
obtain image data that includes a representation of a plurality of documents;
orientate individual documents of the plurality of documents with respect to a reference point to generate machine-orientated documents;
maintain the machine-orientated documents at a task performer queue associated with a task performer network;
receive first task performer information from the task performer network, the first task performer information including at least a portion of the machine-orientated documents being associated with a merge document, tag;
merge documents associated with the merge document tag to generate a merged document, the merged document comprising an invoice;
evaluate the trained vendor detection model on the invoice to match the invoice to a stored invoice associated with vendor information;
retrieve the vendor information, wherein the vendor information is associated with a configuration file specifying a set of document fields associated with a plurality of regions of text;
recognize text of the plurality of regions of text using a machine-based optical character recognition engine to generate recognized text associated with the plurality of regions of text;
segment, the recognized text into a plurality of subgroups of recognized text based on the configuration file and the plurality of regions of text;
maintain the plurality of subgroups of recognized text at the task performer queue;
cause an overlay of at least one subgroup of the plurality of subgroups of recognized text to be presented as user-selectable elements to a human task performer associated with the task performer network, the overlay being presented with a vendor template associated with the invoice;
enable modification of the at least one subgroup of recognized text by the human task performer;
receive second human reviewer information corresponding to a modified version of the at least one subgroup of recognized text;
validate the modified version of the at least one subgroup of recognized text to determine machine-validated invoice entries;
maintain the machine-validated invoice entries at a human task validator queue associated with a task validation network;
receive human task validator information approving the machine-validated invoice entries; and
generate at least one of itemized data or vendor template training data from validated machine-validated invoice entries.