# US 12,120,287 B2 · Multi-person mixed reality experience, method and apparatus > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,120,287 B2 - Document ID: US12120287B2 - Status: Granted - Kind: B2 - Application number: 17/666,364 - Filed: 2022-02-07 - Published: 2022-08-18 - Granted: 2024-10-15 - Earliest claimed: 2021-02-11 - Applicants: Kenneth Perlin - Inventors: Kenneth Perlin - Web page: https://patentgrep.com/patent/US12120287B2 ## CPC - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - H04N 13/368: for two or more viewers - G06F 3/012: {Head tracking input arrangements} - G06T 7/74: {involving reference images or patches} - H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation - H04N 13/15: for colour aspects of image signals - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - H04N 13/257: Colour aspects - H04N 13/296: Synchronisation thereof; Control thereof - H04N 13/344: with head-mounted left-right displays - G06T 2207/30204: Marker - G06T 2207/30244: Camera pose - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An apparatus for viewing in a structure with objects having a first participant and at least a second participant. The apparatus includes a first VR headset to be worn by the first participant. The apparatus includes a second VR headset to be worn by the second participant. Each participant sees every other participant in the structure as every other participant physically appears in real time in a simulated world displayed about them by the respective VR headset each participant is wearing. Each participant sees the simulated world from their own correct perspective in the structure. Each participant able to interact with the simulated world and simultaneously with desired physical objects in the structure and sees the desired physical objects as they physically appear in real time in the structure. A method for a first participant and at least a second participant viewing in a structure with objects. ## Claims ### Claim 1 An apparatus for viewing in a structure having a first participant and at least a second participant comprising: a first inside-out tracked VR headset to be worn by the first participant, the first VR headset having an inertial motion unit, a first computer, and at least a first camera; a second inside-out tracked VR headset to be worn by the second participant, the second VR headset having an inertial motion unit, a second computer, and at least a second camera; a marker attached to the structure for the first and second VR headsets to determine locations of the first and second participants wearing the first and second VR headsets, respectively, in the structure and their own correct perspective in the structure; and coloring on at least a portion of the structure so the portion of the structure with coloring does not appear in the simulated world, each participant sees every other participant in the structure as every other participant physically appears in the structure in real time in a simulated world displayed about them by the respective inside-out tracked VR headset each participant is wearing. each participant sees the simulated world from their own correct perspective in the structure using only the marker, the coloring and their own VR headset they are wearing. ### Claim 2 (depends on 1) The apparatus of claim 1 including a stereo pair of forward-facing cameras mounted on each VR headset to capture an image pair corresponding to a left eye and right eye of the participant, and the at least one camera is one of the forward-facing cameras of the pair of forward-facing cameras, each forward-facing camera of the first VR headset in communication with the first computer for the first computer to receive the image pair from the pair of forward-facing cameras of the first VR headset, each forward-facing camera of the second VR headset in communication with the second computer for the second computer to receive the image pair from the pair of forward-facing cameras of the second VR headset. ### Claim 3 (depends on 2) The apparatus of claim 2 wherein the coloring includes green screen coloring on walls and floor of the structure, the green screen coloring having a brightness. ### Claim 4 (depends on 3) The apparatus of claim 3 wherein the markers include a pattern having a different brightness than the brightness of the green screen coloring on the walls and the floor, the pattern placed on the green screen coloring on the walls. ### Claim 5 (depends on 4) The apparatus of claim 4 wherein the first computer executes a software program stored in a memory of the first VR headset in communication with the first computer which uses each received stereo image pair of the first VR headset as a texture mapped overlay which is displayed on top of the simulated 3D rendered world, only those portions of the stereo image pair of the first VR headset which are not the color of the greenscreen are overlaid on top of the simulated 3D rendered world. ### Claim 6 (depends on 5) The apparatus of claim 5 wherein each VR headset includes an upward facing camera to obtain images of a ceiling of the structure. ### Claim 7 (depends on 6) The apparatus of claim 6 wherein the markers include QR patches disposed on the ceiling, images of which are obtained by the upward facing camera of the first VR headset and used by the first computer of the first VR headset to determine the position of the first participant in the structure. ### Claim 8 (depends on 7) The apparatus of claim 7 wherein the markers include a wire mesh section disposed on the wall of the structure. ### Claim 9 (depends on 8) The apparatus of claim 8 wherein the ceiling the considered by the computer to be a virtual green screen. ### Claim 10 (depends on 9) The apparatus of claim 9 including physical objects disposed in the structure, each participant is able to interact with the simulated world and simultaneously with desired physical objects in the structure and able to see the desired physical objects as they physically appear in real time in the structure in the simulated world and simultaneously how each other participant interacts with the simulated world and the physical objects in the simulated world in real time. ### Claim 11 (depends on 10) The apparatus of claim 10 wherein the first VR headset as a first transmitter and a first receiver, the second VR headset as a second transmitter and a second receiver, the first transmitter communicating with the second receiver and the second transmitter communicating with the first receiver and each participant able to interact with the simulated world. ### Claim 12 (depends on 11) The apparatus of claim 11 including an established network with a host computer having an associated transmitter and receiver which is in communication with all the VR headsets, the host computer receives transmissions from the VR headsets and transmits transmissions to the receivers of all of the VR headsets through the network. ### Claim 13 A method for a first participant and at least a second participant viewing in a structure comprising the steps of: the first participant and the second participant walking around together in a virtual world shown to the first participant by a first inside-out tracked VR headset worn by the first participant and shown to the second participant by a second inside-out tracked VR headset worn by the second participant, the virtual world is an entire world around them that is simulated and displayed in each inside-out tracked VR headset, the first participant and the second participant are in physical sight of each other in the structure and see each other in the structure in the virtual world while viewing the virtual world; and the first participant and the second participant seeing the virtual world from their own correct perspective in the structure, the first VR headset having an inertial motion unit, a first computer, and at least a first camera, the second VR headset having an inertial motion unit, a second computer, and at least a second camera, there is a marker attached to the structure for the first and second VR headsets to determine locations of the first and second participants wearing the first and second VR headsets, respectively, in the structure and their own correct perspective in the structure, there is coloring on at least a portion of the structure so the portion of the structure with coloring does not appear in the simulated world, each participant sees every other participant in the structure as every other participant physically appears in the structure in real time in a simulated world displayed about them by the respective inside-out tracked VR headset each participant is wearing, each participant sees the simulated world from their own correct perspective in the structure using only the marker, the coloring and their own VR headset they are wearing.