Multi-modal model for dynamically responsive virtual characters

A method of generating a virtual character comprising:

• receiving multi-modal input information including environmental information representing a real-world environment, speech information, and facial expression information;
• implementing a first internal model of a plurality of models and a second internal model of the plurality of models to identify a first characteristic of the multi-modal input information by the first internal model and to identify a second characteristic of the multi-modal input information by the second internal model;
• determining whether the first characteristic is within a threshold similarity to the second characteristic;
• selecting a selected characteristic based on determining that the first characteristic is within the threshold similarity of the second characteristic;
• accessing a library of actions associated with the virtual character to select an action that matches the selected characteristic, the action including an animation and an associated audio to be performed by the virtual character;
• and
• causing display of the virtual character on a user device, the virtual character displayed on the user device as augmented in a display of the real-world environment, the virtual character performing the action and outputting the associated audio.

A device comprising:

• a plurality of sensors;
• a processor;
• and
• a memory storing instructions, execution of which by the processor causes the device to perform operations comprising:
• receiving multi-modal input information via the plurality of sensors, the multi-modal input information including environmental information representing a real-world environment, speech information, and facial expression information;
• implementing a first internal model of a plurality of models and a second internal model of the plurality of models to identify a first characteristic of the multi-modal input information by the first internal model and to identify a second characteristic of the multi-modal input information by the second internal model;
• determining whether the first characteristic is within a threshold similarity to the second characteristic;
• selecting a selected characteristic based on determining that the first identified characteristic is within the threshold similarity of the second identified characteristic;
• accessing a library of actions associated with a virtual character to select an action that matches the selected characteristic, the action including an animation and an associated audio to be performed by the virtual character;
• and
• displaying the virtual character on the device, the virtual character displayed on the device as augmented in a display of the real-world environment, the virtual character performing the action and outputting the associated audio.

A non-transitory computer-readable storage medium storing instructions, execution of which by a processor of a computing system causes the computing system to:

• receive, at an input layer from a plurality of sensors, a plurality of signals that each quantify a property of a physical environment being measured by the plurality of sensors;
• process the plurality of signals to produce a plurality of information streams corresponding to the plurality of signals;
• correct errors detected in a first information stream of the plurality of information streams by synthesizing the first information stream with a second information stream of the plurality of information streams;
• produce, at an output layer, a first augmented information stream based on the synthesis of the first information stream with the second information stream;
• and
• cause a virtual character to perform an action based on the first augmented information stream.