Computer-assisted conversion of video description of process into application for process control

A method comprising:

• receiving a video of a manufacturing process including one or more process steps, the video including a visual recording of the manufacturing process and an audio recording of a concurrent narration by a user during the manufacturing process;
• processing the video into a plurality of specific actions in the manufacturing process using computer vision to obtain feature data from the visual recording including one or more video descriptions and corresponding time segments for one or more action segments of the video of the manufacturing process;
• processing the audio recording by identifying one or more natural language action descriptions correlated to the one or more action segments of the video and extracting one or more text descriptions of the one or more action segments of the manufacturing process using a speech-to-text model;
• creating a description of the manufacturing process with a first language model, the description including a step description for each of the one or more action segments based on a multiplexed embedding of the one or more video descriptions of the one or more action segments from the visual recording and the one or more text descriptions of the one or more action segments from the audio recording;
• transforming the feature data into an intermediate representation for use by a low-code or no-code tool to generate an application using a summary and analysis of the feature data from an inference engine;
• transforming the intermediate representation into a machine readable description that expresses programming logic based on a structure of the manufacturing process in the video;
• and
• converting the machine readable description into data for the application to control the manufacturing process.

A system comprising:

• a manufacturing environment configured to perform a manufacturing process;
• a recording device configured to acquire a video including a visual recording of the manufacturing process and an audio recording of a concurrent narration by a user during the manufacturing process;
• a memory storing the video;
• and
• an application generator configured to generate an application to control the manufacturing process by performing the steps of:
• processing the video into a plurality of segments for each of a plurality of specific actions in the manufacturing process using computer vision to extract one or more video descriptions and corresponding time segments for one or more action segments of the video,
• processing the audio recording by identifying one or more natural language action descriptions correlated to the one or more action segments of the video and extracting one or more text descriptions of the one or more action segments of the manufacturing process using a speech-to-text model,
• creating a description of the manufacturing process with a first inference engine, the description including a step description for each of the plurality of segments based on a multiplexed embedding of the one or more video descriptions of the one or more action segments from the visual recording and the one or more text descriptions of the one or more action segments from the audio recording,
• transforming the description of the manufacturing process into an intermediate representation for use by a low-code or no-code tool to generate the application using a summary and analysis of the description from a second inference engine,
• transforming the intermediate representation into a machine readable description that expresses programming logic for the manufacturing process based on a structure of the manufacturing process contained in the intermediate representation, and
• converting the machine readable description into executable code for the application to control the manufacturing process.