Information extraction from documents containing handwritten text

A computer system for information extraction, comprising:

• one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more computer-readable tangible storage media for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, wherein the computer system is capable of performing a method comprising:
• receiving, by an integrated system including a key-value pair extraction model interacting with a handwriting detection model, a mixed-text document including a combination of typed text and handwritten text, wherein the mixed-text document includes at least one key-value pair;
• receiving, by the handwriting detection model of the integrated system, from the key-value pair extraction model of the integrated system, a first location information of at least one key corresponding to the at least one key-value pair in the mixed-text document, wherein the key-value pair extraction model is encoded with a text embedding layer and a text-position embedding layer and trained to identify the at least one key-value pair from a plurality of input text extracted from the mixed-text document;
• based on performing object detection by the handwriting detection model using the first location information of the at least one key as context, detecting, by the handwriting detection model, at least one handwritten text in a vicinity of the at least one key in the mixed-text document, wherein the at least one handwritten text represents at least one value corresponding to the at least one key;
• and
• updating at least one embedding layer of the key-value pair extraction model with the at least one handwritten text detected by the handwriting detection model to iteratively train the key-value pair extraction model to identify the at least one handwritten text as the at least one value corresponding to the at least one key.

A computer program product for information extraction, comprising:

• one or more computer-readable tangible storage media and program instructions stored on at least one of the one or more computer-readable tangible storage media, the program instructions executable by a processor to cause the processor to perform a method comprising:
• receiving, by an integrated system including a key-value pair extraction model interacting with a handwriting detection model, a mixed-text document including a combination of typed text and handwritten text, wherein the mixed-text document includes at least one key-value pair;
• receiving, by the handwriting detection model of the integrated system, from the key-value pair extraction model of the integrated system, a first location information of at least one key corresponding to the at least one key-value pair in the mixed-text document, wherein the key-value pair extraction model is encoded with a text embedding layer and a text-position embedding layer and trained to identify the at least one key-value pair from a plurality of input text extracted from the mixed-text document;
• based on performing object detection by the handwriting detection model using the first location information of the at least one key as context, detecting, by the handwriting detection model, at least one handwritten text in a vicinity of the at least one key in the mixed-text document, wherein the at least one handwritten text represents at least one value corresponding to the at least one key;
• and
• updating at least one embedding layer of the key-value pair extraction model with the at least one handwritten text detected by the handwriting detection model to iteratively train the key-value pair extraction model to identify the at least one handwritten text as the at least one value corresponding to the at least one key.