# US 12,183,106 B1 · Methods and apparatus for extracting data from a document by encoding it with textual and visual features and using machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,183,106 B1
- Document ID: US12183106B1
- Status: Granted
- Kind: B1
- Application number: 18/759,395
- Filed: 2024-06-28
- Published: 2025-01-02
- Granted: 2024-12-31
- Earliest claimed: 2023-06-30
- Assignees: Greenhouse Software, Inc.
- Applicants: Greenhouse Software, Inc.
- Inventors: Triantafyllos Xylouris
- Web page: https://patentgrep.com/patent/US12183106B1

## CPC

- G06V 30/153: {using recognition of characters or words}
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06F 40/103: Formatting, i.e. changing of presentation of documents (automatic justification G06F40/189; automatic line break hyphenation G06F40/191)
- G06N 20/00: Machine learning
- G06Q 10/1053: Employment or hiring
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 30/133: {Evaluation of quality of the acquired characters}
- G06V 30/18: Extraction of features or characteristics of the image
- G06V 30/18057: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 30/191: {Design or setup of recognition systems or techniques; Extraction of features in feature space; Clustering techniques; Blind source separation}
- G06V 30/19147: {Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting}
- G06V 30/41: Analysis of document content (recognition of printed characters based on code marks G06V30/224)
- G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06V 30/416: Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
- G06V 30/42: based on the type of document
- G06F 40/30: Semantic analysis
- G06V 30/245: {Font recognition}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

An apparatus including a processor caused to receive document images, each including representations of characters. The processor is caused to parse each document image to extract, based on structure type, subsets of characters, to generate a text encoding for that document image. For each document, the processor is caused to extract visual features to generate a visual encoding for that document image, each visual feature associated with a subset of characters. The processor is caused to generate parsed documents, each parsed document uniquely associated with a document image and based on the text and visual encoding for that document image. For each parsed document, the processor is caused to identify sections uniquely associated with section type. The processor is caused to train machine learning models, each machine learning model associated with one section type and trained using a portion of each parsed document associated with that section type.

## Claims

### Claim 1

A non-transitory, processor-readable medium storing instructions that executed by a processor, cause the processor to:
receive a plurality of document images each including a plurality of representations of characters:
parse each document image from the plurality of document images to extract a plurality of subsets of characters from the plurality of representations of characters to generate a text encoding for that document image, each subset of characters being associated with a structure type from a plurality of structure types;
for each document image from the plurality of document images, extract a plurality of visual features to generate a visual encoding for that document image, each visual feature from the plurality of visual features associated with at least one subset of characters from the plurality of subsets of characters;
generate a plurality of parsed documents, each parsed document from the plurality of parsed documents uniquely associated with a document image from the plurality of document images and being based on the text encoding and the visual encoding for that document image;
for each parsed document from the plurality of parsed documents, identify a plurality of sections, each section from the plurality of sections uniquely associated with a section type from a plurality of section types; and
train a plurality of machine learning models to produce a plurality of trained machine learning models, each machine learning model from the plurality of machine learning models associated with one section type from the plurality of section types and trained using a portion of each parsed document that is from the plurality of parsed documents and that is associated with that section type.

### Claim 2 (depends on 1)

The non-transitory, processor-readable medium of claim 1, wherein the instructions to cause the processor to parse each document from the plurality of documents further comprises instructions to cause the processor to:
define a bounding box associated with at least one of a word, phrase, date, time, or sentence, a subset of characters from the plurality of subset of characters being associated with the bounding box.

### Claim 3 (depends on 1)

The non-transitory, processor-readable medium of claim 1, wherein the instructions to cause the processor to parse each document from the plurality of documents further comprises instructions to cause the processor to:
define a bounding box associated with at least one of a word, phrase, date, time, or sentence,
a subset of characters from the plurality of subset of characters being associated with the bounding box, and
a visual feature from the plurality of visual features being associated with the bounding box.

### Claim 4 (depends on 1)

The non-transitory, processor-readable medium of claim 1, wherein each trained machine learning model from the plurality of trained machine learning models is configured to extract at least a feature from the section type that is uniquely associated with that trained machine learning model.

### Claim 5 (depends on 1)

The non-transitory, processor-readable medium of claim 1, wherein each trained machine learning model from the plurality of trained machine learning models is configured to:
extract at least a feature from the section type that is uniquely associated with that trained machine learning model; and
populate a structured data file with the feature extracted by that trained machine learning model.

### Claim 6 (depends on 1)

The non-transitory, processor-readable medium of claim 1, the instructions further comprising instructions configured to cause the processor to:
receive a first document image that is not from the plurality of document images;
identify a first section in the first document image that is uniquely associated with a section type from the plurality of section types; and
apply a first trained machine learning model from the plurality of trained machine learning models to the first section in the first document image to produce at least a portion of a structured data file that identifies a feature of the first section in the first document.

### Claim 7 (depends on 1)

The non-transitory, processor-readable medium of claim 1, wherein the instructions to cause the processor to train the plurality of machine learning models further includes instructions to train at least one machine learning model from the plurality of machine learning models to:
extract a feature from the section type that is uniquely associated with that trained machine learning model; and
generate a confidence score associated with the feature extracted from the section type.

### Claim 8 (depends on 1)

The non-transitory, processor-readable medium of claim 1, the instructions further comprising instructions configured to cause the processor to:
receive a first document image that is not from the plurality of document images:
identify a first section in the first document image that is uniquely associated with a first section type from the plurality of section types;
identify a second section in the first document that is uniquely associated with a second section type from the plurality of section types;
apply a first trained machine learning model from the plurality of trained machine learning models to the first section in the first document image to produce at least a first portion of a structured data file that identifies a feature of the first section in the first document; and
apply a second trained machine learning model from the second plurality of trained machine learning models to the second section in the first document image to produce at least a second portion of the structured data file that identifies a feature of the second section in the first document.

### Claim 9

A non-transitory, processor-readable medium storing instructions that executed by a processor, cause the processor to:
receive a document image that includes a plurality of representations of characters;
parse the document image to extract a plurality of subsets of characters from the plurality of representations of characters to generate a text encoding for the document image, each subset of characters being associated with a structure type from a plurality of predefined structure types;
extract a plurality of visual features based on the text encoding to generate a visual encoding for the document image, each visual feature from the plurality of visual features associated with at least one subset of characters from the plurality of subsets of characters;
generate a parsed document based on the text encoding and the visual encoding;
identify a plurality of sections based on the parsed document, each section from the plurality of sections uniquely associated with a section type from a plurality of predefined section types;
execute, for each section from the plurality of sections, a machine learning model from a plurality of machine learning models that is uniquely associated with the section type of that section to extract at least a feature from a plurality of features from the parsed document, each feature from the plurality of features associated with a predefined feature type from a plurality of predefined feature types; and
generate a structured data file based on the least a feature from the plurality of features extracted from the parsed document.

### Claim 10 (depends on 9)

The non-transitory, processor-readable medium of claim 9, the instructions further comprising instructions to cause the processor to:
generate a score for each feature from the plurality of features from the structured data; and
remove a feature from the plurality of features based on the score for that feature being below a predetermined threshold.

### Claim 11 (depends on 9)

The non-transitory, processor-readable medium of claim 9, the instructions further comprising instructions to cause the processor to train the plurality of machine learning models.

### Claim 12 (depends on 9)

The non-transitory, processor-readable medium of claim 9, the instructions further comprising instructions to cause the processor to:
receive a plurality of training document images;
for each training document image from the plurality of training document images, identify at least one section associated with at least one section type from the plurality of predefined section types; and
train a first machine learning model from the plurality of machine learning models that is associated with a first predefined section type from the plurality of predefined section types with a using data from the training document images that are associated with the first predefined section type.

### Claim 13 (depends on 9)

The non-transitory, processor-readable medium of claim 9, the instructions further comprising instructions to cause the processor to:
receive a plurality of training document images;
for each training document image from the plurality of training document images, identify at least one section associated with at least one section type from the plurality of predefined section types;
for each training document image from the plurality of training document images having a first predefined section type from the plurality of predefined section types, extract, from the first section type a text encoding and a visual feature associated with at least a portion of the text encoding; and
train a first machine learning model from the plurality of machine learning models that is associated with the first predefined section type from the plurality of predefined section types with the text encoding and the visual feature.

### Claim 14 (depends on 9)

The non-transitory, processor-readable medium of claim 9, wherein the document image is a resume.

### Claim 15 (depends on 9)

The non-transitory, processor-readable medium of claim 9, wherein the structured data file is a JSON file.

### Claim 16 (depends on 9)

The non-transitory, processor-readable medium of claim 9, wherein the instructions to cause the processor to execute, for each section from the plurality of sections, a machine learning model from a plurality of machine learning models further includes instructions to cause the processor to extract a first feature from the plurality of features from a first section type using a first machine learning model and extract a second feature from the plurality of features from a second section type using a second machine learning model.

### Claim 17 (depends on 9)

The non-transitory, processor-readable medium of claim 9, wherein the instructions to cause the processor to:
execute, for each section from the plurality of sections, a machine learning model from a plurality of machine learning models further includes instructions to cause the processor to extract a first feature from the plurality of features from a first section type using a first machine learning model and extract a second feature from the plurality of features from a second section type using a second machine learning model; and
generate the structured data file further includes instructions to cause the processor to include the first feature and the second feature in the structured data file.

### Claim 18 (depends on 9)

The non-transitory, processor-readable medium of claim 9, wherein the instructions to cause the processor to:
execute, for each section from the plurality of sections, a machine learning model from a plurality of machine learning models further includes instructions to cause the processor to extract a first feature from the plurality of features from a first section type using a first machine learning model and extract a second feature from the plurality of features from a second section type using a second machine learning model; and
generate the structured data file further includes instructions to cause the processor to include the first feature in the structure data file associated with a first section feature mapping and the second feature in the structured data file associated with a second section feature mapping.

### Claim 19 (depends on 9)

The non-transitory, processor-readable medium of claim 9, wherein the instructions to cause the processor to:
execute, for each section from the plurality of sections, a machine learning model from a plurality of machine learning models further includes instructions to cause the processor to extract a first feature from the plurality of features and a second feature from the plurality of features from a first section type using a first machine learning model; and
generate the structured data file further includes instructions to cause the processor to include the first feature and the second feature in the structured data file.

### Claim 20

An apparatus, comprising:
a processor; and
a memory operatively connected to the processor, the memory storing instructions to cause the processor to:
receive a plurality of document images, each document image from the plurality of document images including a plurality of representations of characters;
for each document image from the plurality of document images:
parse that document image to extract a plurality of subsets of characters from the plurality of representations of characters to generate a text encoding for that document image,
each subset of characters from the plurality of subsets of characters being associated with a structure type from a plurality of structure types and to generate a text encoding for that document image based on the plurality of subsets of characters;
for each document image from the plurality of document images, extract a plurality of visual features to generate a visual encoding for that document image, each visual feature from the plurality of visual features associated with at least one subset of characters from the plurality of subsets of characters;
generate a plurality of parsed documents, each parsed document from the plurality of parsed documents uniquely associated with a document image from the plurality of document images and based on the text encoding and the visual encoding for that document image;
for each parsed document from the plurality of parsed documents, identify a plurality of sections, each section from the plurality of sections uniquely associated with a section type from a plurality of section types; and
train a plurality of machine learning models to produce a plurality of trained machine learning models, each machine learning model from the plurality of machine learning models associated with one section type from the plurality of section types and trained using a portion of each parsed document from the plurality of parsed documents, the portion of each parsed document associated with that section type.

### Claim 21 (depends on 20)

The apparatus of claim 20, the memory further storing instructions to cause the processor to execute the plurality of trained machine learning models on a document image that is not from the plurality of document images.
