VPA with integrated object recognition and facial expression recognition

A virtual personal assistant device, comprising:

• one or more processors;
• and
• a non-transitory computer-readable medium including instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:
• receiving, by a computing device, sensory input, wherein the sensory input includes at least two different types of information;
• determining semantic information from the sensory input, wherein the semantic information provides an interpretation of the sensory input;
• determining an action, wherein determining the action includes using the semantic information;
• identifying a context-specific framework, wherein the context-specific framework includes a cumulative sequence of one or more previous intents;
• determining a current intent, wherein determining the current intent includes using the semantic information and the context-specific framework;
• determining a current input state, wherein determining the current input state includes using the semantic information and one or more behavioral models, and wherein the behavioral models include one or more interpretations of previously-provided semantic information;
• and
• using one or more of the current intent and the current input state to modify the action prior to outputting the action to a user of the computing device, wherein to modify the action comprises changing a question type from a first type designed to elicit a first type of answer to a second type designed to elicit a second type of answer.

A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions that, when executed by one or more processors, cause the one or more processors to:

• receive, by a computing device, sensory input, wherein the sensory input includes at least two different types of information;
• determine semantic information from the sensory input, wherein the semantic information provides an interpretation of the sensory input;
• determine an action, wherein determining the action includes using the semantic information;
• determine an action, wherein determining the action includes using the semantic information;
• identify a context-specific framework, wherein the context-specific framework includes a cumulative sequence of one or more previous intents;
• determine a current intent, wherein determining the current intent includes using the semantic information and the context-specific framework;
• determine a current input state, wherein determining the current input state includes using the semantic information and one or more behavioral models, and wherein the behavioral models include one or more interpretations of previously-provided semantic information;
• and
• use one or more of the current intent and the current input state to modify the action prior to outputting the action to a user of the computing device, wherein to modify the action comprises changing the action from an open-ended dialog approach to a yes/no questioning approach or from a yes/no questioning approach to an open-ended dialog approach or from a first question not specifically designed to elicit easy to remember information to a second question designed to elicit easy to remember information.