Suggesting activities

A computing system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• receiving a real-time image or video depicting an entity comprising at least one of a person or a system;
• receiving state inputs pertinent to at least one of a user or the entity;
• determining prospective instructions for the user to interact with the entity based on at least one of the state inputs;
• for each prospective instruction, executing a predictive model over a time horizon in the future simulating execution of the prospective instruction to predict at least one corresponding predicted outcome for execution of the prospective instruction;
• executing a plurality of evaluators, each evaluator having a corresponding objective and configured to, for each prospective instruction:
• evaluate the prospective instruction based on whether the at least one corresponding predicted outcome for execution of the prospective instruction satisfies the corresponding objective of the evaluator;
• and
• output an evaluation of the prospective instruction,
• wherein at least one evaluator comprises a cognitive computing model trained to evaluate a given prospective instruction based on whether at least one corresponding predicted outcome for execution of the given prospective instruction satisfies the corresponding objective of the evaluator, and
• wherein at least two evaluators have functionally distinct corresponding objectives, wherein a first evaluator is configured to evaluate prospective instructions based on a first criterion and a second evaluator is configured to evaluate prospective instructions based on a second criterion different from the first criterion;
• selecting a suggested instruction from the prospective instructions based on the evaluations of the prospective instructions of one or more evaluators;
• and
• in real-time:
• augmenting the real-time image or video, by:
• generating a graphical overlay illustrating the suggested instruction, the graphical overlay comprising an interactive graphical element configured to receive an input from the user in response to the suggested instruction;
• and
• superimposing the graphical overlay onto and compositing the graphical overlay with the real-time image or video depicting the entity to create an augmented reality view;
• and
• displaying, on a screen in communication with the data processing hardware, the augmented reality view.