System and method for vision-assisted approach for graph structure extraction in various types of documents

A system for deterministically deriving underlying graph structure and associated text information in a document, the system comprising:

• a processor;
• and
• a memory operatively connected to the processor via a communication interface, the memory storing computer readable instructions, when executed, causes the processor to:
• implement a vision-based algorithm that includes:
• converting a diagram identified from the document into a first image;
• converting the first image into a grayscale image;
• applying an adaptive threshold algorithm to the grayscale image and removing noise and smoothing the grayscale image and generating a smoothed image;
• identifying contours from the smoothed image;
• implementing a first algorithm onto the smoothed image to determine a bounded area connected to a given node among a plurality of node pixels within the identified contours;
• implementing a second algorithm to expand the node pixels to be thicker thereby expanding out node boundaries of objects themselves;
• utilizing connected components to individually label regions of nodes as a single entity;
• implementing a watershed algorithm to apply the labels onto the grayscale image;
• determining contour of each node;
• implement a network-based algorithm that includes:
• selecting, for each node, a middle pixel that is still a valid marker;
• aggregating all node pixels with same value as one object;
• combining, given the nodes, every pair;
• launching a graph search between two adjacent nodes;
• recording a path and intermediate nodes traversed;
• condensing, given the path, consecutive pixel values into a single item providing unique nodes per step;
• creating a smaller graph including only marker identifiers as a network;
• transposing the network on the first image;
• and
• deterministically derive underlying graph structure and associated text information in the document by applying the vision-based algorithm and the network-based algorithm.

A non-transitory computer readable medium configured to store instructions for deterministically deriving underlying graph structure and associated text information in a document, the instructions, when executed, cause a processor to perform the following:

• implementing a vision-based algorithm that includes:
• converting a diagram identified from the document into a first image;
• converting the first image into a grayscale image;
• applying an adaptive threshold algorithm to the grayscale image and removing noise and smoothing the grayscale image and generating a smoothed image;
• identifying contours from the smoothed image;
• implementing a first algorithm onto the smoothed image to determine a bounded area connected to a given node among a plurality of node pixels within the identified contours;
• implementing a second algorithm to expand the node pixels to be thicker thereby expanding out node boundaries of objects themselves;
• utilizing connected components to individually label regions of nodes as a single entity;
• implementing a watershed algorithm to apply the labels onto the grayscale image;
• determining contour of each node;
• implementing a network-based algorithm that includes:
• selecting, for each node, a middle pixel that is still a valid marker;
• aggregating all node pixels with same value as one object;
• combining, given the nodes, every pair;
• launching a graph search between two adjacent nodes;
• recording a path and intermediate nodes traversed;
• condensing, given the path, consecutive pixel values into a single item providing unique nodes per step;
• creating a smaller graph including only marker identifiers as a network;
• transposing the network on the first image;
• and
• deterministically deriving underlying graph structure and associated text information in the document by applying the vision-based algorithm and the network-based algorithm.