Method and system for fully immersive virtual reality

A computer readable non-transitory medium storing instructions for receiving a six degrees of freedom virtual reality experience at a receiver client in a client-server architecture, the instructions for:

• storing, by a server, information identifying an array of sensors distributed over a geographical area, the sensor array comprising plurality of sensors, each sensor configured to capture one or more of a 360-degree video, a range, or audio, each sensor positioned on a distinct location in the geographical area;
• receiving, by a server from a client device of a user, a set of coordinates identifying a user location and a view angle of the user corresponding to a gaze of a user, wherein the user location represents a virtual location of the user in the geographical area and the view angle represents a virtual view angle of the user in the geographical area;
• storing a viewport library comprising a plurality of precomputed viewports, each viewport corresponding to a distinct user location and a user viewing angle, the storing comprising:
• triangulating the geographical area to identify a plurality of triangular areas;
• determining one or more viewports corresponding to each triangular area;
• and
• storing the determined one or more viewports in the viewport library as the plurality of precomputed viewports, and
• wherein each viewport of the plurality of precomputed viewports is computed by:
• selecting a subset of sensors from the plurality of sensors based on a measure of distance of each sensor from the user location corresponding to the viewport;
• for each sensor from the selected subset of sensors:
• identifying a 360-degree video obtained by the sensor;
• and
• extracting a portion of the 360-degree video obtained by the sensor, wherein the portion of video extracted from the 360-degree video is determined based on the viewing angle of the viewport;
• responsive to receiving the user location and the view angle of the user, identifying a nearest stored viewport from the viewport library for sending to the client device of the user
• synthesizing the identified nearest viewport by synthesizing a video from the point of view of the user location corresponding to the identified nearest viewport in the direction of the user viewing angle corresponding to the identified nearest viewport from the extracted portions of each of the identified 360-degree videos obtained from the subset of sensors selected for the identified nearest viewport, the synthesizing based on one or more techniques comprising:
• depth/range processing, point cloud generation, mesh generation, and texture mapping;
• and
• sending the synthesized viewport to the client device of the user.

A computer system comprising:

• one or more processors;
• and
• a computer readable non-transitory medium storing instructions for providing a six degrees of freedom virtual reality experience at a client in a client-server architecture, the instructions for:
• storing, by a server, information identifying an array of sensors distributed over a geographical area, the sensor array comprising plurality of sensors, each sensor configured to capture one or more of a 360-degree video, a range, or audio, each sensor positioned on a distinct location in the geographical area;
• receiving, by a server from a client device of a user, a set of coordinates identifying a user location and a view angle of the user corresponding to a gaze of a user, wherein the user location represents a virtual location of the user in the geographical area and the view angle represents a virtual view angle of the user in the geographical area;
• storing a viewport library comprising a plurality of precomputed viewports, each viewport corresponding to a distinct user location and a user view angle, the storing comprising:
• triangulating the geographical area to identify a plurality of triangular areas;
• determining one or more viewports corresponding to each triangular area;
• and
• storing the determined one or more viewports in the viewport library as the plurality of precomputed viewports, and
• wherein each viewport of the plurality of precomputed viewports is computed by:
• selecting a subset of sensors from the plurality of sensors based on a measure of distance of each sensor from the user location corresponding to the viewport;
• for each sensor from the selected subset of sensors:
• identifying a 360-degree video obtained by the sensor;
• and
• extracting a portion of the 360-degree video obtained by the sensor, wherein the portion of video extracted from the 360-degree video is determined based on the view angle of the viewport;
• responsive to receiving the user location and the view angle of the user, identifying a nearest stored viewport from the viewport library for sending to the client device of the user;
• synthesizing the identified nearest viewport by synthesizing a video from the point of view of the user location corresponding to the identified nearest viewport in the direction of the user view angle corresponding to the identified nearest viewport from the extracted portions of each of the identified 360-degree videos obtained from the subset of sensors selected for the identified nearest viewport, the synthesizing based on one or more techniques comprising:
• depth/range processing, point cloud generation, mesh generation, and texture mapping;
• and
• sending the synthesized viewport to the client device of the user.

A computer readable non-transitory medium storing instructions for receiving a six degrees of freedom virtual reality experience at a receiver client in a client-server architecture, the instructions for:

• determining, by a client device of a user, a set of coordinates identifying a user location and a view angle of the user corresponding to a gaze of a user, the user virtually located in the geographical area;
• sending, by the client device to a server, the set of coordinates identifying the user location and the view angle of the user, wherein the server stores:
• information identifying an array of sensors distributed over a geographical area, the sensor array comprising plurality of sensors, each sensor configured to capture one or more of a 360-degree video, a range, or audio, each sensor positioned on a distinct location in the geographical area;
• a viewport library comprising a plurality of precomputed viewports, each viewport corresponding to a distinct user location and a user view angle and associated with a respective set of sensors representing a subset of the plurality of sensors of the sensor array selected based on a measure of distance of each sensor from the user position of the viewport, wherein the plurality of precomputed viewports are identified by:
• triangulating the geographical area to identify a plurality of triangular areas;
• determining one or more viewports corresponding to each triangular area;
• and
• storing the determined one or more viewports in the viewport library as the plurality of precomputed viewports;
• receiving, by the client device, from the server, point cloud/mesh/texture data based on a set of portions of videos corresponding to the set of sensors associated with a viewport of the plurality of precomputed viewported determined to be nearest to the user location of the user, each of the set of portions of videos extracted from a 360-degree video obtained by a sensor, wherein the portion of video extracted is determined based on the view angle of the identified nearest viewport;
• synthesizing, by the client device, the identified nearest viewport by synthesizing a video from the point of view of the location of the user in the direction of the gaze of the user from the extracted portions of each of the identified 360-degree videos obtained from the set of sensors, the synthesizing based on one or more techniques comprising:
• point cloud generation, mesh generation, and texture mapping;
• and
• displaying, by the client device, the synthesized viewport via a display of the client device.