Implementing shared mixed reality

A system for implementing a shared mixed reality experience to a first participant and a second participant in a same physical room comprising:

• a first VR headset worn by the first participant in the physical room;
• a second VR headset worn by the second participant in the physical room, each VR headset having a forward-facing color camera stereo pair and a memory, the memory storing an original 3D digital representation of the physical room;
• a first auxiliary processor adjacent to the first participant;
• and
• a second auxiliary processor worn by the second participant, the first auxiliary processor in wired connection to the first VR headset worn by the first participant, the second auxiliary processor in wired connection to the second VR headset worn by the second participant, the memory of the first headset worn by the first participant in communication through the wired connection with the first auxiliary processor worn by the first participant, the memory of the second headset worn by the second participant in communication through the wired connection with the second auxiliary processor worn by the second participant, the first auxiliary processor worn by the first participant calculates a rendered 3D VR scene for the first headset worn by the first participant from a point of view of the first participant's two eyes, based on the first VR headset's current position and orientation, anything in the room that is different from the original 3D digital representation of the room is visible to the first participant in the rendered 3D VR scene, any part of the room that was present in the original 3D digital representation of the room becomes visibly replaced by the rendered 3D VR scene, the second auxiliary processor worn by the second participant calculates a rendered 3D VR scene for the second headset worn by the second participant from a point of view of the second participant's two eyes, based on the second VR headset's current position and orientation, anything in the room that is different from the original 3D digital representation of the room is visible to the second participant in the rendered 3D VR scene, any part of the room that was present in the original 3D digital representation of the room becomes visibly replaced by the rendered 3D VR scene,
• wherein each headset includes a set processor and the computer wirelessly transmits back to each VR headset a silhouette image for each of the participant's right eye and left eye, which is a first color wherever the camera images from the camera should be visible, and a second color different from the first color wherever images from the rendered VR scene should be visible, the silhouette image is used by the set processor on each VR headset to selectively set each pixel of the rendered 3D scene to either a corresponding pixel of the camera image from the camera wherever the silhouette image is the first color or the corresponding pixel from the rendered 3D VR scene wherever the silhouette image is the second color.

A method for implementing a shared mixed reality experience to participants in a same physical room comprising:

• storing an original 3D digital representation of the room in a memory;
• calculating with a computer a rendered 3d VR scene for at least one VR headset worn by a participant of a plurality of headsets worn by participants in the room from a point of view of each of the participant's two eyes, based on the one VR headset's current position and orientation, anything in the room that is different from the original 3D digital representation of the room is visible to the participant in the rendered 3D VR scene, any part of the room that was present in the original 3D digital representation of the room becomes visibly replaced by the rendered 3D VR scene, each VR headset having a forward facing color camera stereo pair, the computer in communication with at least the one VR headset, the memory in communication with the computer;
• and
• displaying in the one VR headset the rendered VR scene,
• wherein each headset includes a set processor and the computer wirelessly transmits back to each VR headset a silhouette image for each of the participant's right eye and left eye, which is a first color wherever the camera images from the camera should be visible, and a second color different from the first color wherever images from the rendered VR scene should be visible, the silhouette image is used by the set processor on each VR headset to selectively set each pixel of the rendered 3D scene to either a corresponding pixel of the camera image from the camera wherever the silhouette image is the first color or the corresponding pixel from the rendered 3D VR scene wherever the silhouette image is the second color.