Image analysis based document processing for inference of key-value pairs in non-fixed digital documents

A non-transitory computer-readable storage medium storing computer program instructions executable to perform operations, the operations comprising:

• receiving, from a client device, a form image;
• obtaining a template indicating one or more fields to extract from the form image, wherein at least one field is associated with a set of candidate keys for the field;
• obtaining, from the form image, a set of phrases and obtaining locations of phrases;
• and
• for the at least one field:
• determining key scores for phrases from the set of phrases, a key score for a phrase indicating a likelihood that the phrase is a key for the field on the form,
• identifying a set of candidate values for the field from the set of phrases,
• determining a set of candidate value-neighbor pairs by identifying a set of neighbors for candidate values from the set of phrases,
• determining neighbor scores for the set of candidate value-neighbor pairs, each neighbor score for each candidate-value pair determined from a combination of the key score for the neighbor of the candidate value-neighbor pair and a spatial relationship of the neighbor to the candidate value for the candidate value-neighbor pair,
• selecting a candidate value-neighbor pair associated with a neighbor score above a threshold, and
• setting the selected candidate value as a value of the field and the selected neighbor as the key of the field.

A system, comprising:

• a processor for executing computer program instructions;
• and
• a non-transitory computer-readable storage medium storing computer program instructions executable to perform steps comprising:
• receiving, from a client device, a form image;
• obtaining a template indicating one or more fields to extract from the form image, wherein at least one field is associated with a set of candidate keys for the field;
• obtaining, from the form image, a set of phrases and obtaining locations of phrases;
• and
• for the at least one field:
• determining key scores for phrases from the set of phrases, a key score for a phrase indicating a likelihood that the phrase is a key for the field on the form,
• identifying a set of candidate values for the field from the set of phrases,
• determining a set of candidate value-neighbor pairs by identifying a set of neighbors for candidate values from the set of phrases,
• determining neighbor scores for the set of candidate value-neighbor pairs, each neighbor score for each candidate value-neighbor pair determined from a combination of the key score for the neighbor of the candidate value-neighbor pair and a spatial relationship of the neighbor to the candidate value for the candidate value-neighbor pair,
• selecting candidate value-neighbor pair associated with a neighbor score above a threshold, and
• setting the selected candidate value as a value of the field and the selected neighbor as the key of the field.