Method and apparatus for efficiently extracting and representing information from technical diagrams

A computing system:

• one or more processors;
• and
• a memory in communication with the one or more processors, wherein the memory stores instruction code that, when executed by the one or more processors, causes the computing system to perform operations comprising:
• training first machine learning logic to identify different types of features associated with an article illustrated in the diagram by providing as input into the first machine learning logic, labeled training diagrams in which features of interest are identified by a bounding box and a label is associated with the bounding box;
• identifying, via the first machine learning logic implemented by the computing system, features associated with the article illustrated in the diagram;
• creating a plurality of labeled knowledge graphs by converting information from the features identified via the first machine learning logic into nodes associated with the features and edges between the nodes that specify relationships between the features, wherein the features include structures of an aircraft and the relationships include characteristics of the structures of the aircraft;
• training second machine learning logic to generate knowledge graphs by providing as input into the second machine learning logic the plurality of labeled knowledge graphs, wherein each knowledge graph, of the plurality of labeled knowledge graphs, specifies a different type of relationship between the features, wherein a node in each knowledge graph specifies a recurrent neural network (RNN) and a relationship between nodes in each knowledge graph specifies a feed-forward neural network (FFN) so that a relationship between a first node and a second node is specified by connecting outputs of the RNN associated with the first node to inputs of the FFN and outputs of the FFN to inputs of the RNN associated with the second node;
• generating, via the second machine learning logic implemented by the computing system and based on the features, a knowledge graph that defines relationships between the features, wherein the knowledge graph specifies nodes associated with the features and edges between the nodes that specify the relationships between the features;
• receiving, by the computing system and from a user, a request for information associated with the diagram;
• searching, by the computing system, the knowledge graph for the information associated with the request;
• and
• communicating, by the computing system, the information associated with the request to the user.

A non-transitory computer-readable medium having stored thereon instruction code that, when executed by one or more processors of a computing system, causes the computing system to perform operations comprising:

• training first machine learning logic to identify different types of features associated with an article illustrated in the diagram by providing as input into the first machine learning logic, labeled training diagrams in which features of interest are identified by a bounding box and a label is associated with the bounding box;
• identifying, via the first machine learning logic implemented by the computing system, features associated with the article illustrated in the diagram;
• creating a plurality of labeled knowledge graphs by converting information from the features identified via the first machine learning logic into nodes associated with the features and edges between the nodes that specify relationships between the features, wherein the features include structures of an aircraft and the relationships include characteristics of the structures of the aircraft;
• training second machine learning logic to generate knowledge graphs by providing as input into the second machine learning logic the plurality of labeled knowledge graphs, wherein each knowledge graph, of the plurality of labeled knowledge graphs, specifies a different type of relationship between the features, wherein a node in each knowledge graph specifies a recurrent neural network (RNN) and a relationship between nodes in each knowledge graph specifies a feed-forward neural network (FFN) so that a relationship between a first node and a second node is specified by connecting outputs of the RNN associated with the first node to inputs of the FFN and outputs of the FFN to inputs of the RNN associated with the second node;
• generating, via the second machine learning logic implemented by the computing system and based on the features, a knowledge graph that defines relationships between the features, wherein the knowledge graph specifies nodes associated with the features and edges between the nodes that specify the relationships between the features;
• receiving, by the computing system and from a user, a request for information associated with the diagram;
• searching, by the computing system, the knowledge graph for the information associated with the request;
• and
• communicating, by the computing system, the information associated with the request to the user.