Machine Learning Based Document Visual Element Extraction

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• obtaining a document comprising:
• a series of textual fields;
• and
• a visual element;
• for each respective textual field of the series of textual fields, determining a respective textual offset for the respective textual field, the respective textual offset indicating a location of the respective textual field relative to each other textual field of the series of textual fields in the document;
• detecting, using a machine learning vision model, the visual element;
• determining a visual element offset indicating a location of the visual element relative to each textual field of the series of textual fields in the document;
• assigning the visual element a visual element anchor token;
• inserting the visual element anchor token into the series of textual fields in an order based on the visual element offset and the respective textual offsets;
• and
• after inserting the visual element anchor token into the series of textual fields, extracting, using a text-based extraction model, from the series of textual fields, a plurality of structured entities, the plurality of structured entities representing the series of textual fields and the visual element.