# US 12,112,526 B2 · Machine learning system and method for determining or inferring user action and intent based on screen image analysis > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,526 B2 - Document ID: US12112526B2 - Status: Granted - Kind: B2 - Application number: 18/326,756 - Filed: 2023-05-31 - Published: 2023-09-28 - Granted: 2024-10-08 - Earliest claimed: 2017-09-15 - Assignees: M37 Inc. - Applicants: M37 Inc. - Inventors: Ali Jelveh - Web page: https://patentgrep.com/patent/US12112526B2 ## CPC - G06V 10/7784: {based on feedback from supervisors} - G06F 18/2178: {based on feedback of a supervisor} - G06F 18/41: {Interactive pattern learning with a human teacher} - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 20/00: Machine learning - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06V 2201/02: Recognising information on displays, dials, clocks - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract System(s) and method(s) that analyze image data associated with a computing screen operated by a user, and learns the image data (e.g., using pattern recognition, historical information analysis, user implicit and explicit training data, optical character recognition (OCR), video information, 360°/panoramic recordings, and so on) to concurrently glean information regarding multiple states of user interaction (e.g., analyzing data associated with multiple applications open on a desktop, mobile phone or tablet). A machine learning model is trained on analysis of graphical image data associated with screen display to determine or infer user intent. An input component receives image data regarding a screen display associated with user interaction with a computing device. An analysis component employs the model to determine or infer user intent based on the image data analysis; and an action component provisions services to the user as a function of the determined or inferred user intent. In an implementation, a gaming component gamifies interaction with the user in connection with explicitly training the model. ## Claims ### Claim 1 A system, comprising: a processor that executes computer executable components stored in memory, wherein the computer executable components comprise: an input component that accesses data comprising graphical image data associated with interactions of a user with a computing device; a model generation component that generates an artificial intelligence (AI) model that processes the graphical image data to learn user behavior and in response to learning the user behavior, generates a prediction associated with future interactions of the user with the computing device; a reward component that analyzes the prediction against the future interactions of the user with the computing device to assign a reward value to the prediction; and a training component that iteratively trains the AI model according to the reward value to generate one or more new AI models directed to optimizing the future interactions of the user with the computing device. ### Claim 2 (depends on 1) The system of claim 1, wherein the data further comprises audio data, motion data, location data, weather data and temperature data associated with the user. ### Claim 3 (depends on 1) The system of claim 1, wherein the training component employs feedback from the user about accuracy of the prediction to iteratively train the AI model. ### Claim 4 (depends on 1) The system of claim 1, wherein the AI model employs a recursive learning algorithm to learn a level of relevance of the graphical image data to generate the prediction. ### Claim 5 (depends on 1) The system of claim 1, further comprising: an action component that automatically executes one or more actions based on the prediction. ### Claim 6 (depends on 1) The system of claim 1, wherein the training component employs genetic algorithms to generate the one or more new AI models, and wherein the one or more new AI models have a first fidelity that is greater than a second fidelity of the AI model. ### Claim 7 (depends on 1) The system of claim 1, wherein the one or more new AI models are further directed to minimizing an identity score that is a mathematical function that measures differences between predictions generated by the one or more new AI models and actual interactions of the user with the computing device. ### Claim 8 (depends on 1) The system of claim 1, wherein respective models of the one or more new AI models can comprise respective neural networks and Bayesian networks, and wherein the one or more new AI models interact with each other to generate new predictions associated with the future interactions of the user with the computing device. ### Claim 9 A computer-implemented method, comprising: accessing, by a system operatively coupled to a processor, data comprising graphical image data associated with interactions of a user with a computing device; generating, by the system, an AI model that processes the graphical image data to learn user behavior; employing, by the system, the AI model to generate a prediction associated with future interactions of the user with the computing device in response to learning the user behavior; analyzing, by the system, the prediction against the future interactions of the user with the computing device to assign a reward value to the prediction; and training, by the system, the AI model according to the reward value to generate one or more new AI models directed to optimizing the future interactions of the user with the computing device, wherein the training is iterative. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, wherein the data further comprises audio data, motion data, location data, weather data and temperature data associated with the user. ### Claim 11 (depends on 9) The computer-implemented method of claim 9, further comprising: employing, by the system, feedback from the user about accuracy of the prediction to iteratively train the AI model. ### Claim 12 (depends on 9) The computer-implemented method of claim 9, further comprising: employing, by the system, a recursive learning algorithm to learn a level of relevance of the graphical image data to generate the prediction. ### Claim 13 (depends on 9) The computer-implemented method of claim 9, further comprising: automatically executing, by the system, one or more actions based on the prediction. ### Claim 14 (depends on 9) The computer-implemented method of claim 9, further comprising: employing, by the system, genetic algorithms to generate the one or more new AI models, wherein the one or more new AI models have a first fidelity that is greater than a second fidelity of the AI model. ### Claim 15 (depends on 9) The computer-implemented method of claim 9, wherein the one or more new AI models are further directed to minimizing an identity score that is a mathematical function that measures differences between predictions generated by the one or more new AI models and actual interactions of the user with the computing device. ### Claim 16 (depends on 9) The computer-implemented method of claim 9, wherein respective models of the one or more new AI models can comprise respective neural networks and Bayesian networks, and wherein the one or more new AI models interact with each other to generate new predictions associated with the future interactions of the user with the computing device. ### Claim 17 A computer program product comprising a non-transitory computer readable memory having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to: access, by the processor, data comprising graphical image data associated with interactions of a user with a computing device; generate, by the processor, an AI model that processes the graphical image data to learn user behavior; employ, by the processor, the AI model to generate a prediction associated with future interactions of the user with the computing device in response to learning the user behavior; analyze, by the processor, the prediction against the future interactions of the user with the computing device to assign a reward value to the prediction; and train, by the processor, the AI model according to the reward value to generate one or more new AI models directed to optimizing the future interactions of the user with the computing device, wherein the AI model is iteratively trained. ### Claim 18 (depends on 17) The computer program product of claim 17, wherein the program instructions are further executable by the processor to cause the processor to: employ, by the processor, feedback from the user about accuracy of the prediction to iteratively train the AI model. ### Claim 19 (depends on 17) The computer program product of claim 17, wherein the program instructions are further executable by the processor to cause the processor to: employ, by the processor, a recursive learning algorithm to learn a level of relevance of the graphical image data to generate the prediction. ### Claim 20 (depends on 17) The computer program product of claim 17, wherein the program instructions are further executable by the processor to cause the processor to: employ, by the processor, genetic algorithms to generate the one or more new AI models, wherein the one or more new AI models have a first fidelity that is greater than a second fidelity of the AI model.