# US 2021/0275923 A1 · Building virtual reality (vr) gaming environments using real-world virtual reality maps > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0275923 A1 - Document ID: US20210275923A1 - Status: Published application - Kind: A1 - Application number: 17/327,655 - Filed: 2021-05-21 - Published: 2021-09-09 - Earliest claimed: 2018-03-05 - Assignees: Sony Interactive Entertainment Inc. - Applicants: Sony Interactive Entertainment LLC - Inventors: Miao Li; Shawn He - Web page: https://patentgrep.com/patent/US20210275923A1 ## CPC - A63F 13/65: automatically by game devices or servers from real world data, e.g. measurement in live racing competition - A63F 13/352: involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - A63F 2300/69: Involving elements of the real world in the game world, e.g. measurement in live races, real video - A63F 2300/8082: Virtual reality - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method for building a gaming environment. The method includes accessing a base VR model of a real-world environment from a third party mapping data store, wherein the real-world environment includes a plurality of real-world objects. The method includes augmenting a first object in the base VR model that corresponds to a first real-world object in the real-world environment. The method includes stitching in the first object that is augmented into the base VR model to generate an augmented base VR model. The method includes storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models for use as virtual environments of corresponding gaming applications storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models. The method includes using one or more of the plurality of augmented base VR models to define a virtual environment of a gaming application. ## Claims ### Claim 1 A method, comprising: determining that a surface model of an object in a base VR model of a real-world environment is incomplete, wherein the base VR model includes a plurality of surface models of a plurality of objects corresponding to real-world objects in the real-world environment; augmenting the object in the base VR model, wherein the object corresponds to a real-world object in the real-world environment; and stitching in the object that is augmented into the base VR model to generate an augmented base VR model. ### Claim 2 (depends on 1) The method of claim 1, wherein the augmenting the object includes: accessing a previously generated surface model corresponding to the real-world object, wherein the previously generated surface model is used as the object that is augmented when performing the stitching. ### Claim 3 (depends on 1) The method of claim 1, wherein the augmenting the object includes: accessing a plurality of images of the real-world object; generating a complete surface model of the object of the real-world object based on the plurality of images, wherein the complete surface model is used as the object that is augmented when performing the stitching. ### Claim 4 (depends on 1) The method of claim 1, wherein the augmenting the object includes: accessing a filler object, wherein the filler object is used as the object that is augmented when performing the stitching. ### Claim 5 (depends on 1) The method of claim 1, further comprising: detecting a missing portion of the surface model, wherein the augmenting the object includes generating an augmented surface model of the object using a surface segment corresponding to the missing portion of the surface model; and wherein the stitching in the object includes generating an augmented base VR model by stitching the augmented surface model into the base VR model. ### Claim 6 (depends on 1) The method of claim 1, further comprising: detecting the real-world object based on one or more images or videos of the real-world environment using a deep learning engine including a plurality of learned models of objects. ### Claim 7 (depends on 1) The method of claim 1, further comprising: detecting a plurality of real-world objects of the real-world environment based on one or more images or videos of the real-world environment; detecting a gap region in the base VR model including one or more missing surface models corresponding to one or more of the plurality of real-world objects; accessing a filler surface model of a filler object; and stitching in the filler surface model into the base VR model in the gap region to generate the augmented base VR model. ### Claim 8 A method, comprising: determining that a surface model of an object in a base VR model of a real-world environment is incomplete, wherein the base VR model includes a plurality of surface models of a plurality of objects corresponding to real-world objects in the real-world environment; generating an augmented surface model of the object using a surface segment corresponding to a missing portion of the surface model, wherein the object corresponds to a real-world object in the real-world environment; and generating an augmented base VR model by stitching the augmented surface model into the base VR model. ### Claim 9 (depends on 8) The method of claim 8, further comprising: accessing the base VR model of a real-world environment from a third party mapping data store. ### Claim 10 (depends on 8) The method of claim 8, wherein the generating an augmented surface model includes: accessing a plurality of images of the real-world object; building the surface segment from the plurality of images; and augmenting the surface model using the surface segment to generate the augmented surface model. ### Claim 11 (depends on 8) The method of claim 8, further comprising: detecting the real-world object based on one or more images or videos of the real-world environment a deep learning engine including a plurality of learned models of objects. ### Claim 12 (depends on 8) The method of claim 8, wherein the surface segment is a generic surface segment. ### Claim 13 A non-transitory computer-readable medium storing a computer program, the computer-readable medium comprising: program instructions for determining that a surface model of an object in a base VR model of a real-world environment is incomplete, wherein the base VR model includes a plurality of surface models of a plurality of objects corresponding to real-world objects in the real-world environment; program instructions for augmenting the object in the base VR model, wherein the object corresponds to a real-world object in the real-world environment; and program instructions for stitching in the object that is augmented into the base VR model to generate an augmented base VR model. ### Claim 14 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein the program instructions for augmenting the object includes: program instructions for accessing a previously generated surface model corresponding to the real-world object, wherein the previously generated surface model is used as the object that is augmented when performing the stitching. ### Claim 15 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein the program instructions for augmenting the object includes: program instructions for accessing a plurality of images of the real-world object; program instructions for generating a complete surface model of the object of the real-world object based on the plurality of images, wherein the complete surface model is used as the object that is augmented when performing the stitching. ### Claim 16 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein the program instructions for augmenting the object includes: program instructions for accessing a filler object, wherein the filler object is used as the object that is augmented when performing the stitching. ### Claim 17 (depends on 13) The non-transitory computer-readable medium of claim 13, further comprising: program instructions for detecting a missing portion of the surface model, wherein the augmenting the object includes generating an augmented surface model of the object using a surface segment corresponding to the missing portion of the surface model; and wherein the stitching in the object includes generating an augmented base VR model by stitching the augmented surface model into the base VR model. ### Claim 18 (depends on 13) The non-transitory computer-readable medium of claim 13, further comprising: program instructions for detecting the real-world object based on one or more images or videos of the real-world environment using a deep learning engine including a plurality of learned models of objects. ### Claim 19 (depends on 13) The non-transitory computer-readable medium of claim 13, further comprising: program instructions for detecting a plurality of real-world objects of the real-world environment based on one or more images or videos of the real-world environment; program instructions for detecting a gap region in the base VR model including one or more missing surface models corresponding to one or more of the plurality of real-world objects; program instructions for accessing a filler surface model of a filler object; and program instructions for stitching in the filler surface model into the base VR model in the gap region to generate the augmented base VR model.