Vision-based document language identification by joint supervision

A cloud infrastructure node comprising:

• one or more processors;
• and
• a non-transitory computer-readable medium including instructions that, when executed by the one or more processors, cause the one or more processors to:
• receive an image-based document including multiple text lines;
• perform a text localization process to identify each of the multiple text lines in the image-based document, wherein each of the multiple text lines is bounded by a bounding box specifying a position of the text line in the image-based document;
• process each text line of the multiple text lines using a convolutional neural network (CNN) of a model to derive visual features from the text line;
• for each text line of the multiple text lines, process the derived visual features for the text line using a recurrent neural network (RNN) of the model to extract contextual features of the text line;
• predict a predicted language of each text line by processing the contextual features of the text line;
• derive, based at least in part on the predicted language for each text line of the multiple text lines, a predicted language for the image-based document.

A non-transitory computer-readable medium having stored thereon a sequence of instructions which, when executed by one or more processors, causes the one or more processors to execute a process comprising:

• receiving an image-based document including multiple text lines;
• executing, for each text line included in the image-based document, a model to perform a language and text prediction process, wherein executing the model includes:
• processing the text line using a convolutional neural network (CNN) to derive visual features from the text line,
• processing the visual features using a recurrent neural network (RNN) to extract contextual features of the text line,
• predicting a predicted language of the text line by processing the contextual features of the text line,
• deriving a cross-entropy (CE) loss based on a comparison between the predicted language and a known language of the text line,
• predicting a predicted set of text for the text line, and
• deriving a connectionist temporal classification (CTC) loss based on a difference between the predicted set of text for the text line and known text for the text line.