Online handwriting document layout analysis system

An online handwriting document layout analysis system;

• the system serving for segmenting a document into a plurality of coarse-grained objects for determining whether a type of each of the coarse-grained objects is a paragraph, a list, a table, a chart or an annotation;
• each of the coarse-grained objects being spilt into a plurality of fine-grained objects;
• the system serving for determining whether a type of each of the fine-grained objects is a text line, a formula, a basic shape, a graffiti, or a set of mistyped strokes;
• the coarse-grained objects and the fine-grained objects forming a tree structure;
• the system comprising a mainframe which includes a processor and a memory;
• and the processor serving for executing required operations of the system and the memory serving for storing data, programs and related operation results of the system;
• the system further comprising:
• a preprocess unit serving to receive the document for performing a preprocessing on the document;
• the document being an online handwriting document and being composed of a plurality of strokes which are close in time or space;
• wherein in the preprocessing of the preprocess unit, the preprocess unit serves to generate an undirected graph for representing relations between different strokes of the document;
• the undirected graph includes a plurality of nodes and a plurality of edges;
• each of the nodes is corresponded to a respective one stroke, which is a directed sequence composed of the points of the stroke according to a writing order;
• each of the edges is corresponded to a pair of respective strokes which are close in time or space;
• the pair of respective strokes close in time or space is corresponded to a respective one edge;
• each of the strokes and itself further form a set to form a respective one edge which is called a loop;
• that is, the loop is an edge connecting one stroke to itself;
• wherein in the undirected graph, each of the strokes is considered to be close in time to at most NT strokes written after it;
• for each of the strokes, a closest point distance between the stroke and the other strokes is calculated for determining at most NS strokes spatially closest in space to the stroke;
• each of NT and NS is a predetermined value;
• a bidirectional recursive neural network unit connected to the preprocess unit for initializing a feature vector of each of the nodes in the undirected graph and initializing a feature vector of each of edges in the undirected graph by using recursive neural networks (RNN);
• wherein in the bidirectional recursive neural network unit, the feature vector of each of the edges is initialized with zero values;
• a graphic neural network unit connected to the bidirectional recursive neural network unit;
• the graphic neural network unit serving to update the feature vector of each of the nodes and the feature vector of each of the edges for obtaining an updated feature vector of each of the nodes and an updated feature vector of each of the edges by using a graphic neural network (GNN) which is based on a message passing;
• a fully connected neural network unit connected to the graphic neural network unit;
• the fully connected neural network unit serving for predicting the type of the coarse-grained object and the fine-grained object for the strokes corresponded to each of the nodes and for predicting that whether the pair of strokes corresponding to each of the edges belong to a same coarse-grained object or a same fine-grained object;
• wherein in the predicting of the fully connected neural network unit, a coarse-grained object classifying and a fine-grained object classifying are performed for each of the nodes and the edges by using fully connected neural networks (FNN) based on the updated feature vectors of the nodes and the edges from the graphic neural network unit;
• a document restoration unit connected to the fully connected neural network unit for restoring the tree structure of the document;
• the document restoration unit serving to group all strokes to obtain the corresponding fine-grained objects by using a connected component analysis according to predicting results of the strokes belonging to the same fine-grained object in the fully connected neural network unit;
• wherein in the document restoration unit, the type of each of the corresponding fine-grained objects is determined by a sum of the confidences that the strokes in the fine-grained object are predicted to belong to a specific type of the fine-grained object in the fully connected neural network unit;
• and
• the document restoration unit further serving to group the corresponding fine-grained objects to obtain the corresponding coarse-grained objects by using a connected component analysis according to predicting results of the strokes belonging to the same coarse-grained object in the fully connected neural network unit;
• wherein in the document restoration unit, the type of each of the corresponding coarse-grained objects is determined by a sum of the confidences that the strokes in the coarse-grained object are predicted to belong to a specific type of the coarse-grained object in the fully connected neural network unit.