Framework for machine guidance

A system comprising:

• a computer processor;
• memory;
• and
• instructions stored in the memory and executable by the computer processor to cause the computer processor to perform operations, the operations comprising:
• receiving a data stream comprising content generated by an application executing on a user device, wherein the data stream is received from a guidance service that is separate from the application;
• processing the data stream using a set of machine learning models to identify a first set of artifacts within the content;
• identifying a first state of the application based on the first set of artifacts;
• identifying first transition data in a logic flow of the application, wherein the first transition data corresponds to transitioning from the first state to a second state of the application;
• generating, based on the first transition data, first guidance data that describes user input that transitions the application from the first state to the second state;
• and
• sending the first guidance data to the user device, wherein the first guidance data is separately presented from the application by the guidance service.

A computer program product comprising non-transitory computer-readable program code that, when executed by a computer processor of a computing system, causes the computing system to perform the method of:

• receiving a data stream comprising content generated by an application executing on a user device, wherein the data stream is received from a guidance service that is separate from the application;
• processing the data stream using a set of machine learning models to identify a first set of artifacts within the content;
• identifying a first state of the application based on the first set of artifacts;
• identifying first transition data in a logic flow of the application, wherein the first transition data corresponds to transitioning from the first state to a second state of the application;
• generating, based on the first transition data, first guidance data that describes user input that transitions the application from the first state to the second state;
• and
• sending the first guidance data to the user device, wherein the first guidance data is separately presented from the application by the guidance service.