Machine learning system and method for determining or inferring user action and intent based on screen image analysis

A computer-implemented method, comprising:

• accessing, by a system operatively coupled to a processor, data comprising graphical image data associated with interactions of a user with a computing device;
• generating, by the system, an AI model that processes the graphical image data to learn user behavior;
• employing, by the system, the AI model to generate a prediction associated with future interactions of the user with the computing device in response to learning the user behavior;
• analyzing, by the system, the prediction against the future interactions of the user with the computing device to assign a reward value to the prediction;
• and
• training, by the system, the AI model according to the reward value to generate one or more new AI models directed to optimizing the future interactions of the user with the computing device, wherein the training is iterative.

A computer program product comprising a non-transitory computer readable memory having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:

• access, by the processor, data comprising graphical image data associated with interactions of a user with a computing device;
• generate, by the processor, an AI model that processes the graphical image data to learn user behavior;
• employ, by the processor, the AI model to generate a prediction associated with future interactions of the user with the computing device in response to learning the user behavior;
• analyze, by the processor, the prediction against the future interactions of the user with the computing device to assign a reward value to the prediction;
• and
• train, by the processor, the AI model according to the reward value to generate one or more new AI models directed to optimizing the future interactions of the user with the computing device, wherein the AI model is iteratively trained.