Communication in a sensory immersive motion capture simulation environment

A simulator engine, comprising:

• a position module configured to process 3D motion data of a participant in a capture volume to determine 3D motion data of a virtual representation of the participant in a simulated virtual environment, wherein the position module, further comprises:
• an avatar module configured to determine at least one of a position, an orientation and a velocity of motion of a head of the virtual representation of the participant in the simulated virtual environment based on the 3D motion data of the participant;
• a simulated character module configured to determine 3D motion data of a virtual character in the simulated virtual environment;
• and
• a vector module configured to calculate a vector between coordinates representing the position and the orientation and/or the velocity of motion of the head of the virtual representation of the participant and coordinates representing the position, the orientation and/or the velocity of motion of the head of the virtual character in the simulated virtual environment,
• wherein a length of the vector represents a distance between the virtual representation of the participant and the virtual character in the simulated virtual environment, and wherein a direction of the vector represents a direction of the virtual representation of the participant respective to the virtual character in the simulated environment;
• a prop module configured to determine a number of obstructions in the path of the vector;
• a speech module configured to process audio data of the participant to determine an attribute of the audio data of the participant at a location of the virtual character in the simulated virtual environment based on the 3D motion data of a virtual representation of the participant in a simulated virtual environment, wherein the virtual character is a non-participant simulation character that is computer generated and controlled by an artificial intelligence algorithm;
• and
• an artificial intelligence module configured to drive an action of the virtual character in the simulated virtual environment based on the audio data of the participant, when the attribute of the audio data of the participant at the location of the virtual character matches a preset condition.

A system, comprising:

• a motion capture device configured to capture a motion of a participant in a capture volume;
• a tracking device configured to transmit 3D motion data of the participant in the capture volume;
• a wearable computing device configured to transmit audio data of the participant;
• and
• a simulator engine communicatively coupled to at least one of the motion capture device, the tracking device and the wearable computing device, the simulator engine is configured to:
• receive the 3D motion data from the tracking device;
• receive the audio data of the participant from the wearable computing device over a communication link between the simulator engine and the wearable computing device, wherein, in a simulated virtual environment, the audio data of the participant is represented as audio data of a virtual representation of the participant;
• process the 3D motion data of the participant to determine 3D motion data of the virtual representation of the participant in the simulated virtual environment;
• process the audio data of the participant to determine an attribute of the audio data of the participant at a location of a virtual character in the simulated virtual environment based on a set of attenuation factors, wherein the virtual character is a non-participant simulation character that is computer generated and controlled by an artificial intelligence algorithm;
• drive an action of the virtual character in the simulated virtual environment based on the audio data of the participant, when the attribute of the audio data of the participant at the location of the virtual character matches a preset condition;
• determine at least one of a position, an orientation and a velocity of motion of a head of the virtual representation of the participant in the simulated virtual environment based on the 3D motion data of the participant,
• determine 3D motion data of the virtual character in the simulated virtual environment,
• calculate a vector between coordinates representing the position, the orientation and/or the velocity of motion of the head of the virtual representation of the participant and coordinates representing the position, the orientation and/or the velocity of motion of the head of the virtual character in the simulated virtual environment,
• wherein a length of the vector represents a distance between the virtual representation of the participant and the virtual character in the simulated virtual environment, and wherein a direction of the vector represents a direction of the virtual representation of the participant respective to the virtual character in the simulated environment;
• and
• determine a number of obstructions in the path of the vector.

A method, comprising:

• receiving, by a simulator engine, audio data of a participant in a capture volume;
• receiving, by the simulator engine, three dimensional (3D) motion data of the participant in the capture volume;
• processing, by the simulator engine, the audio data of the participant to determine an attribute of the audio data of the participant at a location of a virtual character in a simulated virtual environment based on the 3D motion data of the participant in the capture volume, wherein the virtual character is a non-participant simulation character that is computer generated and controlled by an artificial intelligence algorithm;
• probabilistically driving a change of state of the virtual character in the simulated virtual environment based on the attribute of the audio data of the participant and at least one of:
• a current state of the virtual character, and
• a current state of the simulated virtual environment
• determining, by the simulator engine, at least one of a position, an orientation and a velocity of motion of a head of the virtual representation of the participant in the simulated virtual environment based on the 3D motion data of the participant,
• determining, by the simulator engine, 3D motion data of the virtual character in the simulated virtual environment, wherein the 3D motion data of the virtual character comprises a position, an orientation and/or a velocity of motion of the virtual character's head in the simulated virtual environment;
• calculating a vector between coordinates representing the position and the orientation and/or the velocity of motion of the head of the virtual representation of the participant and coordinates representing the position, the orientation and/or the velocity of motion of the head of the virtual character in the simulated virtual environment,
• wherein a length of the vector represents a distance between the virtual representation of the participant and the virtual character in the simulated virtual environment, and wherein a direction of the vector represents a direction of the virtual representation of the participant respective to the virtual character in the simulated environment;
• and
• determining a number of obstructions in the path of the vector.

A method, comprising:

• receiving, by a simulator engine, audio data of a participant in a capture volume, wherein, in a simulated virtual environment, the audio data of the participant is represented as audio data of a virtual representation of the participant;
• processing, by the simulator engine, the audio data of the participant to determine an attribute of the audio data of the participant at a location of a virtual character in the simulated virtual environment based on a set of attenuation factors, wherein the virtual character is a non-participant simulation character that is computer generated and controlled by an artificial intelligence algorithm;
• and
• driving the virtual character in the simulated virtual environment based on the attribute of the audio data of the participant, wherein driving the action of the virtual character in the simulated virtual environment based on the audio data of the participant comprises:
• determining if the virtual representation of the participant is communicating with the virtual character when the virtual representation of the participant is outside a field of vision of the virtual character;
• and
• responsive to determining that the virtual representation of the participant is communicating with the virtual character, triggering a change in an orientation of the virtual character in the simulated virtual environment based on the audio data of the participant.

A simulator engine, comprising:

• a position module configured to process 3D motion data of a participant in a capture volume to determine 3D motion data of a virtual representation of the participant in a simulated virtual environment;
• a speech module configured to process audio data of the participant to determine an attribute of the audio data of the participant at a location of the virtual character in the simulated virtual environment based on the 3D motion data of a virtual representation of the participant in a simulated virtual environment, wherein the virtual character is a non-participant simulation character that is computer generated and controlled by an artificial intelligence algorithm;
• and
• an artificial intelligence module configured to:
• drive an action of the virtual character in the simulated virtual environment based on the audio data of the participant, when the attribute of the audio data of the participant at the location of the virtual character matches a preset condition, and
• determine if the virtual representation of the participant is communicating with the virtual character when the virtual representation of the participant is outside a field of vision of the virtual character.