# US 2021/0278960 A1 · Piping and instrumentation diagram extraction to human-machine interface displays > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0278960 A1 - Document ID: US20210278960A1 - Status: Published application - Kind: A1 - Application number: 16/813,421 - Filed: 2020-03-09 - Published: 2021-09-09 - Earliest claimed: 2020-03-09 - Assignees: Rockwell Automation Technologies, Inc. - Applicants: Rockwell Automation Technologies Inc. - Inventors: Itumeleng Mabote - Web page: https://patentgrep.com/patent/US20210278960A1 ## CPC - G05B 19/4183: {characterised by data acquisition, e.g. workpiece identification} - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials - G06F 8/30: Creation or generation of source code - G05B 23/0216: {Human interface functionality, e.g. monitoring system providing help to the user in the selection of tests or in its configuration} - G05B 2219/31471: Operator can select a graphical screen at his will as help diagnostic - G05B 2219/31479: Operator select part of process he wants to see, video image is displayed - Y02P 90/02: Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS] - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - Y02P: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS ## Abstract Techniques to facilitate automatic importation and placement of display objects on human-machine interface (HMI) displays are disclosed herein. In at least one implementation, a selection of a user-defined area is received that identifies at least a portion of a piping and instrumentation diagram (P&ID) associated with an industrial automation environment. The P&ID is analyzed to identify objects within the user-defined area of the P&ID and determine location data for the objects identified relative to the user-defined area. HMI objects are generated on an HMI display by mapping the objects identified within the user-defined area of the P&ID onto the HMI display based on the user-defined area of the P&ID and the location data for the objects identified relative to the user-defined area. ## Claims ### Claim 1 A method to facilitate automatic importation and placement of display objects on human-machine interface (HMI) displays, the method comprising: receiving a selection of a user-defined area that identifies at least a portion of a piping and instrumentation diagram (P&ID) associated with an industrial automation environment; analyzing the P&ID to identify objects within the user-defined area of the P&ID and determine location data for the objects, wherein the location data indicates relative positions of the identified objects within the user-defined area of the P&ID; generating an HMI-compatible file to include only the identified objects within the user-defined area and the location data associated with the user-defined area; and generating an HMI display of the user-defined area of the P&ID using the HMI-compatible file by: generating a display object for each of the identified objects to display on the HMI display, each display object having associated control information; and placing each display object and the corresponding control information at a location onto the HMI display according to the relative position of the corresponding identified object within the user-defined area of the P&ID the user-defined area of the P&ID. ### Claim 2 (depends on 1) The method of claim 1 wherein analyzing the P&ID to identify the objects within the user-defined area of the P&ID comprises performing optical character recognition on text appearing in the P&ID to identify tag names associated with the objects. ### Claim 3 (depends on 2) The method of claim 2 wherein analyzing the P&ID to identify the objects within the user-defined area of the P&ID comprises comparing object tags identified in an equipment list to the tag names associated with the objects to identify the objects within the user-defined area of the P&ID. ### Claim 4 (depends on 1) The method of claim 1 wherein analyzing the P&ID to determine the location data for the objects identified relative to the user-defined area comprises analyzing the P&ID to determine the location data for the objects identified relative to edge boundaries of the user-defined area. ### Claim 5 (depends on 1) The method of claim 1 wherein analyzing the P&ID to determine the location data for the objects identified relative to the user-defined area comprises analyzing the P&ID to determine the location data for each of the objects identified relative to each other within the user-defined area. ### Claim 6 (depends on 1) The method of claim 1 wherein generating the HMI objects on the HMI display comprises generating extensible markup language (XML) data that identifies mapping of the objects identified within the user-defined area of the P&ID onto the HMI display based on the user-defined area of the P&ID and the location data for the objects identified relative to the user-defined area. ### Claim 7 (depends on 1) The method of claim 1 wherein the control information comprises status data, current performance, historical performance, and control instructions including at least one of parameter updates, turning on/off devices, speed changes, or stop commands. ### Claim 8 One or more computer-readable storage media having program instructions stored thereon to facilitate automatic importation and placement of display objects on human-machine interface (HMI) displays, wherein the program instructions, when executed by a computing system, direct the computing system to at least: receive a selection of a user-defined area that identifies at least a portion of a piping and instrumentation diagram (P&ID) associated with an industrial automation environment; analyze the P&ID to identify objects within the user-defined area of the P&ID and determine location data for the objects, wherein the location data indicates relative positions of the identified objects within the user-defined area of the P&ID; generate an HMI-compatible file to include only the identified objects within the user-defined area and the location data associated with the user-defined area; and generate an HMI display of the user-defined area of the P&ID using the HMI-compatible file by: generating a display object for each of the identified objects to display on the HMI display, each display object having associated control information; and placing each display object and the corresponding control information at a location onto the HMI display according to the relative position of the corresponding identified object within the user-defined area of the P&ID the user-defined area of the P&ID. ### Claim 9 (depends on 8) The one or more computer-readable storage media of claim 8 wherein the program instructions direct the computing system to analyze the P&ID to identify the objects within the user-defined area of the P&ID by directing the computing system to perform optical character recognition on text appearing in the P&ID to identify tag names associated with the objects. ### Claim 10 (depends on 9) The one or more computer-readable storage media of claim 9 wherein the program instructions direct the computing system to analyze the P&ID to identify the objects within the user-defined area of the P&ID by directing the computing system to compare object tags identified in an equipment list to the tag names associated with the objects to identify the objects within the user-defined area of the P&ID. ### Claim 11 (depends on 8) The one or more computer-readable storage media of claim 8 wherein the program instructions direct the computing system to analyze the P&ID to determine the location data for the objects identified relative to the user-defined area by directing the computing system to analyze the P&ID to determine the location data for the objects identified relative to edge boundaries of the user-defined area. ### Claim 12 (depends on 8) The one or more computer-readable storage media of claim 8 wherein the program instructions direct the computing system to analyze the P&ID to determine the location data for the objects identified relative to the user-defined area by directing the computing system to analyze the P&ID to determine the location data for each of the objects identified relative to each other within the user-defined area. ### Claim 13 (depends on 8) The one or more computer-readable storage media of claim 8 wherein the program instructions direct the computing system to generate the HMI objects on the HMI display by directing the computing system to generate extensible markup language (XML) data that identifies the mapping of the objects identified within the user-defined area of the P&ID onto the HMI display based on the user-defined area of the P&ID and the location data for the objects identified relative to the user-defined area. ### Claim 14 (depends on 8) The one or more computer-readable storage media of claim 8 wherein the control information comprises status data, current performance, historical performance, and control instructions including at least one of parameter updates, turning on/off devices, speed changes, or stop commands. ### Claim 15 An apparatus to facilitate automatic importation and placement of display objects on human-machine interface (HMI) displays, the apparatus comprising: one or more computer-readable storage media; a processing system operatively coupled with the one or more computer-readable storage media; and program instructions stored on the one or more computer-readable storage media that, when executed by the processing system, direct the processing system to at least: receive a selection of a user-defined area that identifies at least a portion of a piping and instrumentation diagram (P&ID) associated with an industrial automation environment; analyze the P&ID to identify objects within the user-defined area of the P&ID and determine location data for the objects, wherein the location data indicates relative positions of the identified objects within the user-defined area of the P&ID; generate an HMI-compatible file to include only the identified objects within the user-defined area and the location data associated with the user-defined area; and generate an HMI display of the user-defined area of the P&ID using the HMI-compatible file by: generating a display object for each of the identified objects to display on the HMI display, each display object having associated control information; and placing each display object and the corresponding control information at a location onto the HMI display according to the relative position of the corresponding identified object within the user-defined area of the P&ID the user-defined area of the P&ID. ### Claim 16 (depends on 15) The apparatus of claim 15 wherein the program instructions direct the processing system to analyze the P&ID to identify the objects within the user-defined area of the P&ID by directing the processing system to perform optical character recognition on text appearing in the P&ID to identify tag names associated with the objects. ### Claim 17 (depends on 16) The apparatus of claim 16 wherein the program instructions direct the processing system to analyze the P&ID to identify the objects within the user-defined area of the P&ID by directing the processing system to compare object tags identified in an equipment list to the tag names associated with the objects to identify the objects within the user-defined area of the P&ID. ### Claim 18 (depends on 15) The apparatus of claim 15 wherein the program instructions direct the processing system to analyze the P&ID to determine the location data for the objects identified relative to the user-defined area by directing the processing system to analyze the P&ID to determine the location data for the objects identified relative to edge boundaries of the user-defined area. ### Claim 19 (depends on 15) The apparatus of claim 15 wherein the program instructions direct the processing system to analyze the P&ID to determine the location data for the objects identified relative to the user-defined area by directing the processing system to analyze the P&ID to determine the location data for each of the objects identified relative to each other within the user-defined area. ### Claim 20 (depends on 15) The apparatus of claim 15 wherein the program instructions direct the processing system to generate the HMI objects on the HMI display by directing the processing system to generate extensible markup language (XML) data that identifies the mapping of the objects identified within the user-defined area of the P&ID onto the HMI display based on the user-defined area of the P&ID and the location data for the objects identified relative to the user-defined area.