Merging multiple model outputs for extraction

A method of training an encoder-decoder model to extract information from one or more documents, the method performed by an electronic device coupled to the encoder-decoder model and comprising:

• receiving, over a communications network, a plurality of extraction model outputs from a corresponding plurality of extraction models, each extraction model including a respective plurality of key-value pairs corresponding to extracted text from one or more training documents;
• receiving, over the communications network, character recognition data corresponding to the one or more training documents;
• receiving, over the communications network, ground truth key-value data corresponding to the one or more training documents;
• training the encoder-decoder model based at least in part on the plurality of extraction model outputs, the character recognition data, and the ground truth key-value data, wherein the trained encoder-decoder model is configured to generate key-value pairs for subsequent outputs of the plurality of extraction models;
• receiving a plurality of subsequent outputs from the plurality of extraction models, the subsequent outputs indicating predicted key-value pairs associated with one or more subsequent documents;
• and
• predicting, using the trained encoder-decoder model and the subsequent outputs, a sequence of output key-value pairs corresponding to the one or more subsequent documents.

A system for training an encoder-decoder model to extract information from one or more documents, the system associated with the encoder-decoder model and comprising:

• one or more processors;
• and
• a memory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
• receiving, over a communications network, a plurality of extraction model outputs from a corresponding plurality of extraction models, each extraction model output including a respective plurality of key-value pairs corresponding to extracted text from one or more training documents;
• receiving, over the communications network, character recognition data corresponding to the one or more training documents;
• receiving, over the communications network, ground truth key-value data corresponding to the one or more training documents;
• training the encoder-decoder model based at least in part on the plurality of extraction model outputs, the character recognition data, and the ground truth key-value data, wherein the trained encoder-decoder model is configured to generate key-value pairs for subsequent outputs of the plurality of extraction models;
• receiving a plurality of subsequent outputs from the plurality of extraction models, the subsequent outputs indicating predicted key-value pairs associated with one or more subsequent documents;
• and
• predicting, using the trained encoder-decoder model, a sequence of output key-value pairs corresponding to the one or more subsequent documents.

A system for training an encoder-decoder model to extract information from one or more documents, the system associated with the encoder-decoder model and comprising:

• one or more processors;
• and
• a memory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
• receiving, over a communications network, a plurality of extraction model outputs from a corresponding plurality of extraction models, each extraction model including a respective plurality of key-value pairs corresponding to extracted text from one or more training documents;
• receiving, over the communications network, character recognition data corresponding to the one or more training documents;
• receiving, over the communications network, ground truth key-value data corresponding to the one or more training documents;
• training the encoder-decoder model based at least in part on the plurality of extraction model outputs, the character recognition data, and the ground truth key-value data, wherein the trained encoder-decoder model is configured to generate key-value pairs for subsequent outputs of the plurality of extraction models;
• and
• generating an input to the encoder-decoder model based at least in part on concatenating the extraction model outputs by generating a plurality of field predictions separated by a first predetermined delimiter, wherein each field prediction includes a field name and an ordered plurality of predicted field values separated by a second predetermined delimiter, each predicted field value generated by a different extraction model and including a predicted value associated with the field name.