# US 2023/0067033 A1 · Vision-based document language identification by joint supervision

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0067033 A1
- Document ID: US20230067033A1
- Status: Published application
- Kind: A1
- Application number: 17/897,055
- Filed: 2022-08-26
- Published: 2023-03-02
- Earliest claimed: 2021-08-27
- Assignees: Oracle International Corporation
- Applicants: Oracle International Corporation
- Inventors: Liyu Gong; Yuying Wang; Zhonghai Deng; Iman Zadeh; Jun Qian
- Web page: https://patentgrep.com/patent/US20230067033A1

## CPC

- G06V 30/246: using linguistic properties, e.g. specific for English or German language
- G06F 40/263: Language identification
- G06V 10/82: using neural networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The present embodiments relate to a language identification system for predicting a language and text content of text lines in an image-based document. The language identification system uses a trainable neural network model that integrates multiple neural network models in a single unified end-to-end trainable architecture. A CNN and an RNN of the model can process text lines and derive visual and contextual features of the text lines. The derived features can be used to predict a language and text content for the text line. The CNN and the RNN can be jointly trained by determining losses based on the predicted language and content and corresponding language labels and text labels for each text line.

## Claims

### Claim 1

A computer-implemented comprising:
training a model based, at least in part, on a language predicted loss and a text predicted loss for each of one or more image-based documents that include multiple text lines of text in a predicted language, the language predicted loss and text predicted loss being derived from executing the model;
receiving an image-based document that includes multiple text lines;
executing, for each text line included in the image-based document, the trained model to perform a language and text prediction process, wherein executing the trained model includes:
processing the text line using a convolutional neural network (CNN) to derive visual features from the text line;
processing the visual features using a recurrent neural network (RNN) to extract contextual features of the text line;
predicting a predicted language of the text line by processing the contextual features of the text line;
deriving a language prediction loss based on a comparison between the predicted language and a known language of the text line;
predicting a predicted set of text for the text line; and
deriving a text prediction loss based on a difference between the predicted set of text for the text line and known text for the text line.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
performing a text localization process to identify each of the multiple text lines in the image-based document, wherein each of the multiple text lines are bounded by a bounding box specifying a position of each of the multiple text lines in the image-based document.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
aggregating predicted languages for each of the multiple text lines to derive a predicted language for the image-based document.

### Claim 4 (depends on 1)

The method of claim 1, wherein the RNN includes a series of Bi-Directional Long Short Term Memory (Bi-LSTM) layers.

### Claim 5 (depends on 1)

The method of claim 1, wherein predicting the predicted language is performed using a linear layer, a one-dimensional global average pooling (GAP) layer, and a soft-max activation layer.

### Claim 6 (depends on 1)

The method of claim 1, wherein the language prediction loss comprises a cross-entropy (CE) loss.

### Claim 7 (depends on 1)

The method of claim 1, wherein the text prediction loss is a connectionist temporal classification (CTC) loss.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
jointly training the CNN and the RNN using both the language prediction loss and text prediction loss.

### Claim 9

A cloud infrastructure node comprising:
one or more processors; and
a non-transitory computer-readable medium including instructions that, when executed by the processor, cause the one or more processors to:
receive an image-based document including multiple text lines;
perform a text localization process to identify each of the multiple text lines in the image-based document, wherein each of the multiple text lines are bounded by a bounding box specifying a position of each of the multiple text lines in the image-based document;
process each text line of the multiple text lines using a convolutional neural network (CNN) of the model to derive visual features from the text line;
process the derived visual features using a recurrent neural network (RNN) of the model to extract contextual features of each text line;
predict a predicted language of each text line by processing the contextual features of the text line;
aggregate predicted languages for each of the multiple text lines; and
derive, based on the aggregated predicted languages, a predicted language for the image-based document.

### Claim 10 (depends on 9)

The cloud infrastructure node of claim 9, wherein the RNN includes a series of Bi-Directional Long Short Term Memory (Bi-LSTM) layers.

### Claim 11 (depends on 9)

The cloud infrastructure node of claim 9, wherein predicting the predicted language is performed using a linear layer, a one-dimensional global average pooling (GAP) layer, and a soft-max activation layer.

### Claim 12 (depends on 9)

The cloud infrastructure node of claim 9, wherein the predicted languages for each of the multiple text lines comprise a discrete probability distribution for assigning each text line to predicted languages for each of the multiple text lines.

### Claim 13 (depends on 9)

The cloud infrastructure node of claim 9, wherein the non-transitory computer-readable medium further includes instructions that, when executed by the one or more processors, cause the one or more processors to:
derive, for each of the multiple text lines, a CE loss based on a comparison between the predicted language and a known language of each text line.

### Claim 14 (depends on 13)

The cloud infrastructure node of claim 13, wherein the non-transitory computer-readable medium further includes instructions that, when executed by the one or more processors, cause the one or more processors to:
predict a predicted set of text for the text line; and
derive a CTC loss based on a difference between the predicted set of text for the text line and known text for the text line.

### Claim 15 (depends on 14)

The cloud infrastructure node of claim 14, wherein the non-transitory computer-readable medium further includes instructions that, when executed by the one or more processors, cause the one or more processors to:
jointly train the CNN and the RNN of the model using both the CE loss and CTC loss.

### Claim 16

A non-transitory computer-readable medium having stored thereon a sequence of instructions which, when executed by one or more processors, causes the one or more processors to execute a process comprising:
receiving an image-based document including multiple text lines;
executing, for each text line included in the image-based document, a model to perform a language and text prediction process, wherein executing the model includes:
processing the text line using a convolutional neural network (CNN) to derive visual features from the text line;
processing the visual features using a recurrent neural network (RNN) to extract contextual features of the text line;
predicting a predicted language of the text line by processing the contextual features of the text line;
deriving a cross-entropy (CE) loss based on a comparison between the predicted language and a known language of the text line;
predicting a predicted set of text for the text line; and
deriving a connectionist temporal classification (CTC) loss based on a difference between the predicted set of text for the text line and known text for the text line.

### Claim 17 (depends on 16)

The non-transitory computer-readable medium of claim 16, further including instructions that, when executed by one or more processors, cause:
performing a text localization process to identify each of the multiple text lines in the image-based document, wherein each of the multiple text lines are bounded by a bounding box specifying a position of each of the multiple text lines in the image-based document.

### Claim 18 (depends on 16)

The non-transitory computer-readable medium of claim 16, further including instructions that, when executed by one or more processors, cause:
aggregating predicted languages for each of the multiple text lines to derive a predicted language for the image-based document.

### Claim 19 (depends on 16)

The non-transitory computer-readable medium of claim 16, wherein the RNN includes a series of Bi-Directional Long Short Term Memory (Bi-LSTM) layers.

### Claim 20 (depends on 16)

The non-transitory computer-readable medium of claim 16, further including instructions that, when executed by one or more processors, cause:
jointly training the CNN and the RNN using both the CE loss and CTC loss.
