# US 12,117,780 B2 · Supplying HMI snapshots from industrial automation environments as input to machine learning models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,117,780 B2 - Document ID: US12117780B2 - Status: Granted - Kind: B2 - Application number: 17/484,364 - Filed: 2021-09-24 - Published: 2023-03-30 - Granted: 2024-10-15 - Earliest claimed: 2021-09-24 - Assignees: Rockwell Automation Technologies, Inc. - Applicants: Rockwell Automation Technologies, Inc. - Inventors: Jordan C. Reynolds; John J. Hagerbaumer; Troy W. Mahr; Jeffrey S. Sperling; Thomas K. Jacobsen; Giancarlo Scaturchio - Web page: https://patentgrep.com/patent/US12117780B2 ## CPC - G05B 13/0265: {the criterion being a learning criterion} - G06T 7/0004: {Industrial image inspection} - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Various embodiments of the present technology generally relate to integrating machine learning models into industrial automation environments. More specifically, embodiments of the present technology include systems and methods for implementing machine learning models within industrial control code to improve performance, increase productivity, and add capability to existing control programs. In an embodiment, a system comprises a screen capture component configured to capture images of a human-machine interface in an industrial automation environment, wherein the one or more images include at least one visual depiction of data collected from an industrial device. The system further comprises an input component configured to provide the images to a machine learning model configured to analyze an operating condition of the industrial device. The system also comprises a determination component configured to, based on an output of the machine learning model, identify a current operating condition of the industrial device. ## Claims ### Claim 1 A system for analyzing operating conditions in industrial automation environments, the system comprising: a memory that stores executable components; and a processor, operatively coupled to the memory, that executes the executable components, the executable components comprising: an image reception component configured to obtain: one or more images of a human-machine interface in an industrial automation environment, wherein the one or more images of the human-machine interface include at least one visual depiction of data collected from an industrial device in the industrial automation environment; and one or more images of the industrial device captured in the industrial automation environment; an input component configured to provide the one or more images of the human-machine interface and the one or more images of the industrial device to a machine learning model trained to analyze an operating condition of the industrial device in the industrial automation environment based on the one or more images of the human-machine interface and the one or more images of the industrial device; and a determination component configured to, based on an output of the machine learning model, identify a current operating condition of the industrial device, wherein the current operating condition is indicative of at least one issue related to performance of the industrial device. ### Claim 2 (depends on 1) The system of claim 1, wherein the executable components further comprise a control component configured to edit control logic for the industrial device based at least in part on the output of the machine learning model. ### Claim 3 (depends on 1) The system of claim 1, wherein the executable components further comprise a notification component configured to generate a notification in a user interface of the system indicating a presence of the at least one issue related to the performance of the industrial device. ### Claim 4 (depends on 1) The system of claim 1, wherein the executable components further comprise: an image capture component configured to capture the one or more images of the industrial device; and a screen capture component configured to capture the one or more images of the human-machine interface. ### Claim 5 (depends on 1) The system of claim 1, wherein the machine learning model comprises a convolutional neural network. ### Claim 6 (depends on 1) The system of claim 1, wherein the executable components further comprise a training component configured to train the machine learning model to identify operating conditions of the industrial device based on the one or more images of the human-machine interface and the one or more images of the industrial device. ### Claim 7 A non-transitory computer-readable medium having stored thereon instructions to support analyzing operating conditions that, in response to execution, cause a system comprising a processor to perform operations, the operations comprising: obtaining: one or more images of a human-machine interface in an industrial automation environment, wherein the one or more images of the human-machine interface include at least one visual depiction of data collected from an industrial device in the industrial automation environment; and one or more images of the industrial device captured in the industrial automation environment: providing the one or more images of the human-machine interface and the one or more images of the industrial device to a machine learning model trained to analyze an operating condition of the industrial device in the industrial automation environment based on the one or more images of the human-machine interface and the one or more images of the industrial device; and based on an output of the machine learning model, identifying a current operating condition of the industrial device, wherein the current operating condition is indicative of at least one issue related to performance of the industrial device. ### Claim 8 (depends on 7) The non-transitory computer-readable medium of claim 7, the operations further comprising editing control logic for the industrial device based at least in part on the output of the machine learning model. ### Claim 9 (depends on 7) The non-transitory computer-readable medium of claim 7, the operations further comprising generating a notification in a user interface indicating a presence of the at least one issue related to the performance of the industrial device. ### Claim 10 (depends on 7) The non-transitory computer-readable medium of claim 7, the operations further comprising: capturing the one or more images of the industrial device; and capturing the one or more images of the human-machine interface. ### Claim 11 (depends on 7) The non-transitory computer-readable medium of claim 7, wherein the machine learning model comprises a convolutional neural network. ### Claim 12 (depends on 7) The non-transitory computer-readable medium of claim 7, the operations further comprising training the machine learning model to identify operating conditions of the industrial device based on the one or more images of the human-machine interface and the one or more images of the industrial device. ### Claim 13 A method for analyzing operating conditions in industrial automation environments, the method comprising: obtaining-capturing, by a system comprising a processor: one or more images of a human-machine interface in an industrial automation environment, wherein the one or more images of the human-machine interface include at least one visual depiction of data collected from an industrial device in the industrial automation environment; and one or more images of the industrial device captured in the industrial automation environment; providing, by the system, the one or more images of the human-machine interface and the one or more images of the industrial device to a machine learning model trained to analyze an operating condition of the industrial device in the industrial automation environment based on the one or more images of the human-machine interface and the one or more images of the industrial device; and based on an output of the machine learning model, identifying, by the system, a current operating condition of the industrial device, wherein the current operating condition is indicative of at least one issue related to performance of the industrial device. ### Claim 14 (depends on 13) The method of claim 13, further comprising editing, by the system, control logic for the industrial device based at least in part on the output of the machine learning model. ### Claim 15 (depends on 13) The method of claim 13, wherein the method further comprises generating, by the system, a notification in a user interface of the system indicating a presence of the at least one issue related to performance of the industrial device. ### Claim 16 (depends on 13) The method of claim 13, further comprising: capturing, by the system, the one or more images of the industrial device; and capturing, by the system, the one or more images of the human-machine interface. ### Claim 17 (depends on 13) The method of claim 13, further comprising training the machine learning model to identify operating conditions of the industrial device based on the one or more images of the human-machine interface and the one or more images of the industrial device. ### Claim 18 (depends on 13) The method of claim 13, wherein the machine learning model comprises a convolutional neural network.