# US 2020/0288113 A1 · System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0288113 A1 - Document ID: US20200288113A1 - Status: Published application - Kind: A1 - Application number: 16/653,329 - Filed: 2019-10-15 - Published: 2020-09-10 - Earliest claimed: 2011-05-27 - Assignees: Thomas Seidl - Inventors: Thomas Seidl; Ron Igra - Web page: https://patentgrep.com/patent/US20200288113A1 ## CPC - H04N 13/344: with head-mounted left-right displays - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06T 15/10: Geometric effects - H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - H04N 13/243: using three or more two-dimensional [2D] image sensors - H04N 13/383: for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes - G06T 2215/08: Gnomonic or central projection - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The present invention relates to a system and method for capturing video of a real-world scene over a field of view that may exceed the field of view of a user, manipulating the captured video, and then stereoscopically displaying the manipulated image to the user in a head mounted display to create a virtual environment having length, width, and depth in the image. By capturing and manipulating video for a field of view that exceeds the field of view of the user, the system and method can quickly respond to movement by the user to update the display allowing the user to look and pan around, i.e., navigate, inside the three-dimensional virtual environment. ## Claims ### Claim 1 A system for creating a navigable, panoramic three-dimensional virtual reality environment with stereoscopic perception of three-dimensional depth to a user, comprising: a storage device for storing first and second video image streams corresponding to two different stereoscopic viewpoints of a scene, respectively, each stream comprising image frames; a tracking device configured to track the movement of a user; a position detector disposed in communication with the tracking device to receive tracking data from the tracking device and configured to determine a direction of view of the user; an image renderer disposed in communication with the storage device to receive the first and second video image streams and disposed in communication with the position detector to receive the direction of view, the image renderer configured to create an output stream of merged stereo image frames corresponding to the direction of view of the user, the merged frames comprising image frames from the first video stream which are temporally matched to respective image frames from the second video stream; and a head mounted display disposed in communication with the image renderer to receive output stream and configured to display the output stream to the user, whereby a navigable, panoramic three-dimensional virtual reality environment having stereoscopic perception of three-dimensional depth is created. ### Claim 2 (depends on 1) The system according to claim 1, comprising an optical imaging device disposed in communication with the storage device and configured to image at least two different viewpoints of the scene, the imaging device comprising at least one optical imaging detector configured to record the first and second video image streams corresponding to the at least two different viewpoints of the scene, respectively. ### Claim 3 (depends on 2) The system according to claim 2, wherein the optical imaging device is configured to display the images of the at least two different view points on the optical imaging detector to provide the first and second video image streams. ### Claim 4 (depends on 1) The system according to claim 1, wherein the image renderer is configured to split the merged frames of the output stream and transform each of the split frames onto respective first and second models. ### Claim 5 (depends on 4) The system according to claim 4, wherein the center of origin of the each model is selected to correspond to a location within the head mounted display at which a respective one of the user's eyes is to be positioned. ### Claim 6 (depends on 4) The system according to claim 4, wherein the model comprises a wireframe model. ### Claim 7 (depends on 4) The system according to claim 4, wherein the model comprises a sphere. ### Claim 8 (depends on 1) The system according to claim 1, wherein the renderer comprises a non-transitory computer usable medium having a computer readable program code embodied therein, the computer readable program code adapted to be executed to implement a method for rendering the first and second video image streams to create a panoramic three-dimensional virtual reality environment having stereoscopic perception of three-dimensional depth comprising the steps of creating first and second wireframe models, and transforming each of the first and second video images by wrapping the first and second video streams onto the first and second wireframe models respectively. ### Claim 9 A non-transitory computer usable medium comprising a computer readable program code embodied therein, the computer readable program code adapted to be executed to implement a method for rendering video image streams to create a navigable, panoramic three-dimensional virtual reality environment having stereoscopic perception of three-dimensional depth comprising the steps of: receiving video image streams of a scene comprising first and second video image streams taken from different stereoscopic viewpoints, the first video image stream having a plurality of first image frames and the second video image stream having a plurality of second image frames; merging respective pairs of the first and second image frames in stereo format to create a video output stream of merged stereoscopic image pairs; receiving position data indicating a direction of sight of a user; calculating a first and second region of interest of the merged stereoscopic image pairs based on the position data; and displaying the first and second regions of interest on a head mounted stereoscopic display, whereby a navigable, panoramic three-dimensional virtual reality environment having stereoscopic perception of three-dimensional depth is created. ### Claim 10 (depends on 9) A non-transitory computer usable medium comprising a computer readable program code adapted to be executed to implement a method for rendering the first and second video image streams to create a navigable, panoramic three-dimensional virtual reality environment according to claim 9, comprising creating first and. second wireframe models, and wrapping the respective first and second section of the merged image pairs onto the first and second wireframe models respectively. ### Claim 11 (depends on 10) A non-transitory computer usable medium comprising a computer readable program code adapted to be executed to implement a method for rendering the first and second video image streams to create a navigable three-dimensional virtual reality environment according to claim 10, wherein the step of displaying the first and second regions of interest comprises rotating the first and second wireframe models in response to the position data. ### Claim 12 (depends on 9) A non-transitory computer usable medium comprising a computer readable program code adapted to be executed to implement a method for rendering the first and second video image streams to create a navigable three-dimensional virtual reality environment according to claim 9, comprising removing distortion from the first and second image frames. ### Claim 13 (depends on 9) A non-transitory computer usable medium comprising a computer readable program code adapted to be executed to implement a method for rendering the first and second video image streams to create a navigable, panoramic three-dimensional virtual reality environment according to claim 9, wherein the step of merging the respective pairs of the first and second image frames comprises temporally synchronizing the first and second image frames.