Methods and apparatus for extracting data from a document by encoding it with textual and visual features and using machine learning

A non-transitory, processor-readable medium storing instructions that executed by a processor, cause the processor to:

• receive a document image that includes a plurality of representations of characters;
• parse the document image to extract a plurality of subsets of characters from the plurality of representations of characters to generate a text encoding for the document image, each subset of characters being associated with a structure type from a plurality of predefined structure types;
• extract a plurality of visual features based on the text encoding to generate a visual encoding for the document image, each visual feature from the plurality of visual features associated with at least one subset of characters from the plurality of subsets of characters;
• generate a parsed document based on the text encoding and the visual encoding;
• identify a plurality of sections based on the parsed document, each section from the plurality of sections uniquely associated with a section type from a plurality of predefined section types;
• execute, for each section from the plurality of sections, a machine learning model from a plurality of machine learning models that is uniquely associated with the section type of that section to extract at least a feature from a plurality of features from the parsed document, each feature from the plurality of features associated with a predefined feature type from a plurality of predefined feature types;
• and
• generate a structured data file based on the least a feature from the plurality of features extracted from the parsed document.

An apparatus, comprising:

• a processor;
• and
• a memory operatively connected to the processor, the memory storing instructions to cause the processor to:
• receive a plurality of document images, each document image from the plurality of document images including a plurality of representations of characters;
• for each document image from the plurality of document images:
• parse that document image to extract a plurality of subsets of characters from the plurality of representations of characters to generate a text encoding for that document image,
• each subset of characters from the plurality of subsets of characters being associated with a structure type from a plurality of structure types and to generate a text encoding for that document image based on the plurality of subsets of characters;
• for each document image from the plurality of document images, extract a plurality of visual features to generate a visual encoding for that document image, each visual feature from the plurality of visual features associated with at least one subset of characters from the plurality of subsets of characters;
• generate a plurality of parsed documents, each parsed document from the plurality of parsed documents uniquely associated with a document image from the plurality of document images and based on the text encoding and the visual encoding for that document image;
• for each parsed document from the plurality of parsed documents, identify a plurality of sections, each section from the plurality of sections uniquely associated with a section type from a plurality of section types;
• and
• train a plurality of machine learning models to produce a plurality of trained machine learning models, each machine learning model from the plurality of machine learning models associated with one section type from the plurality of section types and trained using a portion of each parsed document from the plurality of parsed documents, the portion of each parsed document associated with that section type.