# US 11,443,082 B2 · Utilizing deep learning and natural language processing to convert a technical architecture diagram into an interactive technical architecture diagram > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,082 B2 - Document ID: US11443082B2 - Status: Granted - Kind: B2 - Application number: 16/884,783 - Filed: 2020-05-27 - Published: 2021-12-02 - Granted: 2022-09-13 - Earliest claimed: 2020-05-27 - Assignees: Accenture Global Solutions Limited - Applicants: Accenture Global Solutions Limited - Inventors: Balaji Janarthanam; Abhishek Patni; Anil Kumar; Vinu Varghese; Hari Kumar Karnati; Saran Prasad; Nirav Jagdish Sampat - Web page: https://patentgrep.com/patent/US11443082B2 ## CPC - G06V 30/422: Technical drawings; Geographical maps - G06F 30/18: Network design, e.g. design based on topological or interconnect aspects of utility systems, piping, heating ventilation air conditioning [HVAC] or cabling (circuit design at the physical level G06F30/39; network planning tools for wireless communication networks H04W16/18) - G06F 40/166: Editing, e.g. inserting or deleting - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06V 30/416: Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors - G06F 30/27: using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model - G06F 40/279: Recognition of textual entities - G06F 2111/12: Symbolic schematics - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N 20/00: Machine learning - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A device may receive input data identifying a technical architecture diagram, a design document, an interface specification document, and technical architecture icons, and may process the input data identifying the technical architecture diagram, with a model, to determine hierarchical objects from the technical architecture diagram. The device may perform OCR and NLP of the hierarchical objects to determine blocks of data, and may compare the blocks of data and the input data identifying the design document to identify functionalities of applications. The device may compare the blocks of data and the input data identifying the interface specification document to identify attributes, and may compare the blocks of data and the input data identifying the technical architecture icons to identify icons. The device may consolidate the blocks of data, the functionalities, the attributes, and the icons into a final document, and may perform actions based on the final document. ## Claims ### Claim 1 A method, comprising: training, by a device, a deep learning model with: historical icon data identifying icons in historical technical architecture diagrams, historical interconnection data identifying interconnections in the historical technical architecture diagrams, and historical legend data identifying legends in the historical technical architecture diagrams; receiving, by the device, input data identifying: a technical architecture diagram, a design document associated with the technical architecture diagram, an interface specification document associated with the technical architecture diagram, and technical architecture icons associated with the technical architecture diagram; processing, by the device, the input data identifying the technical architecture diagram, with the deep learning model, to determine hierarchical objects from the technical architecture diagram; performing, by the device, optical character recognition and natural language processing of the hierarchical objects to determine, from the hierarchical objects, blocks of data identifying a technical architecture of the technical architecture diagram; comparing, by the device, the blocks of data and the input data identifying the design document to identify functionalities of applications associated with the technical architecture diagram; comparing, by the device, the blocks of data and the input data identifying the interface specification document to identify attributes associated with the technical architecture diagram; comparing, by the device, the blocks of data and the input data identifying the technical architecture icons to identify icons associated with the technical architecture diagram; consolidating, by the device, the blocks of data, the identified functionalities, the identified attributes, and the identified icons into a final document; and performing, by the device, one or more actions based on the final document. ### Claim 2 (depends on 1) The method of claim 1, wherein performing the one or more actions comprises one or more of: generating an interactive technical architecture diagram based on the final document; generating a technical architecture summary document based on the final document; or generating data identifying the attributes based on the final document. ### Claim 3 (depends on 1) The method of claim 1, wherein performing the one or more actions comprises one or more of: generating snippets for the technical architecture diagram based on the final document; providing the final document to a client device; or retraining the deep learning model based on the final document. ### Claim 4 (depends on 1) The method of claim 1, wherein performing the one or more actions comprises: providing, to a client device, one or more of: an interactive technical architecture diagram generated based on the final document, a technical architecture summary document generated based on the final document, data identifying the attributes, or data identifying snippets generated based on the final document. ### Claim 5 (depends on 1) The method of claim 1, wherein performing the one or more actions comprises: generating an interactive technical architecture diagram based on the final document; providing the interactive technical architecture diagram to a client device; receiving, from the client device, an interactive input associated with the interactive technical architecture diagram; modifying the interactive technical architecture diagram based on the interactive input and to generate a modified interactive technical architecture diagram; and providing the modified interactive technical architecture diagram to the client device. ### Claim 6 (depends on 1) The method of claim 1, wherein performing the one or more actions comprises: generating an interactive technical architecture diagram based on the final document; providing the interactive technical architecture diagram to a client device; receiving, from the client device, one or more changes to the interactive technical architecture diagram; and modifying the interactive technical architecture diagram based on the one or more changes and to generate a modified interactive technical architecture diagram. ### Claim 7 (depends on 1) The method of claim 1, wherein the blocks of data are in a JavaScript Object Notation (JSON) format. ### Claim 8 A device, comprising: one or more memories; and one or more processors, communicatively coupled to the one or more memories, configured to: train a deep learning model with: historical icon data identifying icons in historical technical architecture diagrams, historical interconnection data identifying interconnections in the historical technical architecture diagrams, and historical legend data identifying legends in the historical technical architecture diagrams; receive input data identifying: a technical architecture diagram, a design document associated with the technical architecture diagram, an interface specification document associated with the technical architecture diagram, and technical architecture icons associated with the technical architecture diagram; process the input data identifying the technical architecture diagram, with the deep learning model, to determine hierarchical objects from the technical architecture diagram; perform optical character recognition and natural language processing of the hierarchical objects to determine, from the hierarchical objects, blocks of data identifying a technical architecture of the technical architecture diagram; compare the blocks of data and the input data identifying the design document to identify functionalities of applications associated with the technical architecture diagram; compare the blocks of data and the input data identifying the interface specification document to identify attributes associated with the technical architecture diagram; compare the blocks of data and the input data identifying the technical architecture icons to identify icons associated with the technical architecture diagram; consolidate the blocks of data, the identified functionalities, the identified attributes, and the identified icons into a final document; generate an interactive technical architecture diagram based on the final document; and provide the interactive technical architecture diagram to a client device. ### Claim 9 (depends on 8) The device of claim 8, wherein the one or more processors, to process the input data identifying the technical architecture diagram, with the deep learning model, to determine the hierarchical objects from the technical architecture diagram, are: process the input data identifying the technical architecture diagram, with the deep learning model, to determine a hierarchical order associated with the hierarchical objects. ### Claim 10 (depends on 8) The device of claim 8, wherein the one or more processors, to process the input data identifying the technical architecture diagram, with the deep learning model, to determine the hierarchical objects from the technical architecture diagram, are: convert the input data identifying the technical architecture diagram into formatted data in a format compatible with the deep learning model; and process the formatted data, with the deep learning model, to determine the hierarchical objects from the technical architecture diagram. ### Claim 11 (depends on 8) The device of claim 8, wherein the one or more processors are further configured to: generate, based on the final document, a technical architecture summary document that provides a description of functional components of the technical architecture diagram; and provide the technical architecture summary document to the client device. ### Claim 12 (depends on 8) The device of claim 8, wherein the hierarchical objects include one or more of: a title of the technical architecture diagram, a quantity of blocks in the technical architecture diagram, a quantity of sub-blocks in the technical architecture diagram, a legend of the technical architecture diagram, an arrow of the technical architecture diagram, a connector of the technical architecture diagram, or an icon of the technical architecture diagram. ### Claim 13 (depends on 8) The device of claim 8, wherein the blocks of data include text provided in the technical architecture diagram. ### Claim 14 (depends on 8) The device of claim 8, wherein the final document is provided in a particular format. ### Claim 15 A non-transitory computer-readable medium storing instructions, the instructions comprising: one or more instructions that, when executed by one or more processors, cause the one or more processors to: train a deep learning model with: historical icon data identifying icons in historical technical architecture diagrams, historical interconnection data identifying interconnections in the historical technical architecture diagrams, and historical legend data identifying legends in the historical technical architecture diagrams; receive input data identifying: a technical architecture diagram, a design document associated with the technical architecture diagram, an interface specification document associated with the technical architecture diagram, and technical architecture icons associated with the technical architecture diagram; process the input data identifying the technical architecture diagram, with the deep learning model, to determine hierarchical objects from the technical architecture diagram; perform optical character recognition and natural language processing of the hierarchical objects to determine, from the hierarchical objects, blocks of data identifying a technical architecture of the technical architecture diagram; compare the blocks of data and the input data identifying the design document to identify functionalities of applications associated with the technical architecture diagram; compare the blocks of data and the input data identifying the interface specification document to identify attributes associated with the technical architecture diagram; compare the blocks of data and the input data identifying the technical architecture icons to identify icons associated with the technical architecture diagram; consolidate the blocks of data, the identified functionalities, the identified attributes, and the identified icons into a final document; generate an interactive technical architecture diagram and a technical architecture summary document based on the final document; and provide the interactive technical architecture diagram and the technical architecture summary document to a client device. ### Claim 16 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the instructions further comprise: one or more instructions that, when executed by the one or more processors, cause the one or more processors to one or more of: generate data identifying the attributes based on the final document; generate snippets for the technical architecture diagram based on the final document; provide the final document to the client device; or retrain the deep learning model based on the final document. ### Claim 17 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the instructions further comprise: one or more instructions that, when executed by the one or more processors, cause the one or more processors to one or more of: receive, from the client device, an interactive input associated with the interactive technical architecture diagram; modify the interactive technical architecture diagram based on the interactive input and to generate a modified interactive technical architecture diagram; and provide the modified interactive technical architecture diagram to the client device. ### Claim 18 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the instructions further comprise: one or more instructions that, when executed by the one or more processors, cause the one or more processors to one or more of: receive, from the client device, one or more changes to the interactive technical architecture diagram; modify the interactive technical architecture diagram based on the one or more changes and to generate a modified interactive technical architecture diagram; and provide the modified interactive technical architecture diagram to the client device. ### Claim 19 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the one or more instructions, that cause the one or more processors to process the input data identifying the technical architecture diagram, with the deep learning model, to determine the hierarchical objects from the technical architecture diagram, cause the one or more processors to: convert the input data identifying the technical architecture diagram into formatted data in a format compatible with the deep learning model; and process the formatted data, with the deep learning model, to determine the hierarchical objects from the technical architecture diagram. ### Claim 20 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the hierarchical objects include one or more of: a title of the technical architecture diagram, a quantity of blocks in the technical architecture diagram, a quantity of sub-blocks in the technical architecture diagram, a legend of the technical architecture diagram, an arrow of the technical architecture diagram, a connector of the technical architecture diagram, or an icon of the technical architecture diagram.