Table extraction from images using language models

A system comprising:

• one or more storage media storing instructions;
• and
• one or more processors configured to execute the instructions to cause the system to perform processing comprising:
• detecting, within an image, a first area that includes a first table;
• extracting, from the first area of the image, tabular data for the first table, the tabular data comprising a first plurality of content items in the first table and first structural information, wherein the first structural information defines a spatial location for each content item of the first plurality of content items within the image;
• generating a structured prompt that includes the first plurality of content items and the first structural information, wherein generating the structured prompt comprises serializing each content item of the first plurality of content items with its corresponding spatial location into a single data string that includes prompt part identifiers to delineate the first plurality of content items and their spatial locations;
• providing the structured prompt as input to a first large language model;
• and
• responsive to providing the structured prompt as input to the first large language model, generating, by the first large language model, a first parsable representation of the first table by interpreting the serialized spatial locations of the first plurality of content items within the single data string to reconstruct a logical structure of the first table, wherein the first parsable representation is in a first format and includes the first plurality of content items arranged according to the reconstructed logical structure.

One or more non-transitory computer-readable storage media storing instructions that, upon execution by one or more processors of a system, cause the system to perform operations comprising:

• detecting, within an image, a first area that includes a first table;
• extracting, from the first area of the image, tabular data for the first table, the tabular data comprising a first plurality of content items in the first table and first structural information, wherein the first structural information defines a spatial location for each content item of the first plurality of content items within the image;
• generating a structured prompt that includes the first plurality of content items and the first structural information, wherein generating the structured prompt comprises serializing each content item of the first plurality of content items with its corresponding spatial location into a single data string that includes prompt part identifiers to delineate the first plurality of content items and their spatial locations;
• providing the structured prompt as input to a first large language model;
• and
• responsive to providing the structured prompt as input to the first large language model, generating, by the first large language model, a first parsable representation of the first table by interpreting the serialized spatial locations of the first plurality of content items within the single data string to reconstruct a logical structure of the first table, wherein the first parsable representation is in a first format and includes the first plurality of content items arranged according to the reconstructed logical structure.