# US 2021/0279957 A1 · Systems and methods for building a virtual representation of a location > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0279957 A1 - Document ID: US20210279957A1 - Status: Published application - Kind: A1 - Application number: 17/194,075 - Filed: 2021-03-05 - Published: 2021-09-09 - Earliest claimed: 2020-03-06 - Assignees: Yembo, Inc. - Applicants: Yembo, Inc. - Inventors: Marc EDER; Siddharth MOHAN; Maciej HALBER; Anoop JAKKA; Devin WALTMAN; Zachary RATTNER - Web page: https://patentgrep.com/patent/US20210279957A1 ## CPC - G06Q 30/0205: {based on location or geographical consideration} - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 16/29: Geographical information databases - G06F 18/2413: based on distances to training or reference patterns - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06Q 10/06395: {Quality analysis or management} - G06Q 10/0875: Itemisation or classification of parts, supplies or services, e.g. bill of materials - G06Q 30/0206: {Price or cost determination based on market factors} - G06Q 40/08: Insurance - G06T 7/0002: {Inspection of images, e.g. flaw detection} - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 7/50: Depth or shape recovery - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60) - G06V 20/64: Three-dimensional [3D] objects - H04N 23/631: {Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters} - H04N 23/80: Camera processing pipelines; Components thereof - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G01S 17/06: Systems determining position data of a target - G01S 17/89: for mapping or imaging - G01S 19/42: Determining position - G06F 18/24: Classification techniques - G06Q 50/16: Real estate - G06T 2207/20081: Training; Learning - G06T 2219/004: Annotating, labelling - Y02A 10/40: Controlling or monitoring, e.g. of flood or hurricane; Forecasting, e.g. risk assessment or mapping - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - Y02A: TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE ## Abstract Systems, methods, and non-transitory computer readable media are disclosed that include operations to generate a virtual representation of a physical location with spatially localized information related to elements within the location being embedded in the virtual representation. The operations includes receiving description data (e.g., a plurality of images and videos) of the location, the description data being generated via at least one of a camera, a user interface; receive metadata associated elements within the location; generating (e.g., offline or in real-time), via a machine learning model and/or a geometric model, a 3-dimensional (3D) model of the location and elements therein; and generating, based on the 3D model of the location, an information-rich virtual representation of the location by annotating the 3D model with spatially localized metadata associated with the elements within the location and semantic information of the elements. ## Claims ### Claim 1 A system configured to generate a virtual representation of a location with spatially localized information of elements within the location being embedded in the virtual representation, the system comprising one or more hardware processors configured by machine-readable instructions to: receive description data of a location, the description data being generated via at least one of a camera, a user interface, an environment sensor, and an external location information database, the description data comprising a plurality of images, and pose matrices; receive metadata associated elements within the location; generate, in real-time, via a machine learning model and/or a geometric model, a 3-dimensional (3D) representation of the location and elements therein, the machine learning model being configured to receive the plurality of image and pose matrices as inputs and predict geometry of the location and the elements therein to form the 3D model; and generate, based on the 3D model of the location, a virtual representation of the location by annotating the 3D model with spatially localized metadata associated with the elements within the location, and semantic information of the elements within the location, the virtual representation being editable by a user to allow modifications to the spatially localized metadata. ### Claim 2 (depends on 1) The system of claim 1, wherein generating the 3D model comprises: encoding each image of the plurality of images with the machine learning model; adjusting, based on the encoded images of the plurality of images, an intrinsics matrix associated with the camera; using the intrinsics matrix and pose matrices to back-project the encoded images into a predefined voxel grid volume; providing the voxel grid with the features as input to a neural network to predicts a 3D model of the location for each voxel in the voxel grid; and extracting a 2D surface of the predicted 3D model. ### Claim 3 (depends on 2) The system of claim 2, wherein the intrinsics matrix represents physical attributes of a camera, the physical attributes comprising: focal length, principal point, and skew. ### Claim 4 (depends on 2) The system of claim 2, wherein the pose matrix represents a camera's relative or absolute orientation in a virtual world, the pose matrix comprising 3-degrees-of-freedom rotation of the camera and a 3-degrees-of-freedom position in the virtual representation. ### Claim 5 (depends on 1) The system of claim 1, wherein generating the virtual representation with spatially localized metadata comprises: spatially localizing the metadata using a geometric estimation model, or manual entry of the metadata via a graphical user interface configured to allow a user to hover over or select a particular element and edit the metadata. ### Claim 6 (depends on 5) The system of claim 5, wherein spatially localizing of the metadata comprises: receiving additional images of the location and associating the additional images to the 3D model of the location; computing camera poses associated with the additional images with respect to the existing plurality of images and the 3D model; and relocalizing, via the geometric estimation model and the camera poses, the additional images and associating the metadata. ### Claim 7 (depends on 1) The system of claim 1, further comprising instructions to: display metadata about an element when a user hovers over or selects the element within virtual representation of the location. ### Claim 8 (depends on 1) The system of claim 1, wherein the metadata associated with the element comprises at least one of: geometric properties of the element; material specifications of the element; a condition of the element; receipts related to the element; invoices related to the element; spatial measurements captured through the virtual representation or physically at the location; details about insurance coverage; audio, visual, or natural language notes; or 3D shapes and objects including geometric primitives and CAD models. ### Claim 9 (depends on 1) The system of claim 1, wherein generating the virtual representation with the semantic information comprises: identifying elements from the plurality of image or the 3D model by a semantically trained machine learning model, the semantically trained machine learning model configured to perform semantic or instance segmentation and 3D object detection and localization of each object in an input image. ### Claim 10 (depends on 1) The system of claim 1, wherein generating the virtual representation further comprises: estimating, via a pose estimation method and the plurality of images, pose data based on the plurality of images and intrinsics of the camera, obtaining heuristic information associated with one or more standard elements detected within the location, and estimating, based on the heuristic information and geometric correlation between the plurality of images, a scale factor to determine dimensions of the elements within the location. ### Claim 11 (depends on 1) The system of claim 1, further comprising instructions to: generate feedback information configured to guide the user to collect additional description data at a particular portion of the location; and update the virtual representation based on the additional description data collected in response to the feedback information to cause improvements in accuracy and confidence level of the virtual representation. ### Claim 12 (depends on 11) The system of claim 11, wherein generating the feedback information comprises: generating visual indicators representing confidence levels associated with the elements within the location, wherein visual indicators having a relatively low confidence level guide the user to collect additional description data at a particular portion of the location. ### Claim 13 (depends on 12) The system of claim 12, wherein generating the visual indicators comprises: generating, via a machine learning model or based on a number of instances of each element in the plurality of images, a probability map indicating how accurately a particular element is represented in the virtual representation or a 2D representation of the virtual representation. ### Claim 14 (depends on 11) The system of claim 11, wherein updating the virtual representation based on the additional description data comprises: inputting the additional data to the machine learning model to update the corresponding portion of the virtual representation. ### Claim 15 (depends on 11) The system of claim 11, wherein the feedback information is generated in real-time, and the additional data is collected in real-time to update the virtual representation in real-time. ### Claim 16 (depends on 1) The system of claim 1, further comprising instructions to: specify points of interest within the virtual representation displayed on a graphical user interface; generate, based on the points of interest, a floor plan; and spatially localize the floor plan on to the virtual representation. ### Claim 17 (depends on 1) The system of claim 1, further comprising instructions to: generate a floor plan of the location by a layout estimation algorithm configured to use lines in the 3D model, plane information related to the elements in the 3D model, vanishing points of the location, or geometric indicators related to the 3D model to estimate the floor plan of the location. ### Claim 18 (depends on 17) The system of claim 17, wherein the layout estimation algorithm is further configured to generate or update the floor plan based on constraints and heuristics, the heuristics comprising orthogonal junctions of the location, planar assumptions about a composition of the location, and cuboidal layout assumptions related to the location. ### Claim 19 (depends on 18) The system of claim 18, wherein the layout estimation algorithm includes a neural network configured to generate the floor plan using the lines, planes, points, and geometric indicators as inputs. ### Claim 20 (depends on 1) The system of claim 1, wherein the description data comprises one or more media types, the media types comprising at least one or more of video data, image data, audio data, text data, user interface/display data, and/or sensor data. ### Claim 21 (depends on 1) The system of claim 1, wherein receiving description data comprises receiving sensor data from the one or more environment sensors, the one or more environment sensors comprising at least one of a GPS, an accelerometer, a gyroscope, a barometer, or a microphone. ### Claim 22 (depends on 1) The system of claim 1, wherein description data is captured by a mobile computing device associated with a user and transmitted to the one or more processors with or without a first user and/or other user interaction. ### Claim 23 (depends on 1) The system of claim 1, wherein the description data of the location comprises receiving a real-time video stream of the location. ### Claim 24 (depends on 23) The system of claim 23, wherein generating the virtual representation comprises: generating or updating the 3D model based on the real-time video stream of the location. ### Claim 25 (depends on 1) The system of claim 1, wherein the generating, in real-time, the virtual representation comprises: receiving, at a user device, the description data of the location, transmitting the description data to a server configured to execute the machine learning model to generate the 3D model of the location, generating, at the server based on the machine learning model and the description data, the virtual representation of the location, and transmitting the virtual representation to the user device. ### Claim 26 A system configured to generate a virtual representation of a location with spatially localized information of elements within the location being embedded in the virtual representation, the system comprising one or more hardware processors configured by machine-readable instructions to: receive description data of a location, the description data being generated via at least one of a camera, a user interface, an environment sensor, and an external location information database, the description data comprising a plurality of images; generate, via a machine learning model and/or a geometric model, a 3-dimensional (3D) representation of the location and elements therein, the machine learning model being configured to receive the plurality of image as input and predict geometry of the location and the elements therein to form the 3D model; obtain, via a database or a user, heuristic information associated with one or more elements detected within the location, the heuristic information comprising dimension data associated with a element; estimate, based on the heuristic information and geometric correlation between the plurality of images, a scale factor for adjusting sizes of the elements in the images; estimate, based on the scale factor, dimensions of the elements within the location; and update, based on the estimated scale factor and the estimated dimensions, the virtual representation of the location by adjusting sizes of the elements within the location and annotating the 3D model with estimated dimensions of the elements of the location. ### Claim 27 (depends on 26) The system of claim 26, further comprising instructions to: estimate pose matrices and intrinsics for each image of the plurality of images by a geometric reconstruction framework configured to triangulate 3D points based on the plurality of images to estimate both camera poses up to scale and camera intrinsics, and input the pose matrices and intrinsics to the machine learning model to accurately predict the 3D model of the location. ### Claim 28 (depends on 27) The system of claim 27, wherein the geometric reconstruction framework comprises at least one of: structure-from-motion (SFM), multi-view stereo (MVS), or simultaneous localization and mapping (SLAM). ### Claim 29 (depends on 26) The system of claim 26, further comprising instructions to: estimate metric depth maps for each image of the plurality of images, and input the depth maps to the machine learning model to accurately predict the 3D model of the location. ### Claim 30 (depends on 26) The system of claim 26, further comprising instructions to: obtain metadata associated with the elements within the location; and generate, based on the 3D model of the location, a virtual representation of the location by annotating the 3D model with spatially localized metadata associated with the elements within the location, and semantic information of the elements within the location, the virtual representation being editable by a user to allow modifications to the spatially localized metadata. ### Claim 31 (depends on 30) The system of claim 30, further comprising instructions to:: specify points of interest within the virtual representation displayed on a graphical user interface; generate, based on the points of interest, a floor plan; and spatially localize the floor plan on to the virtual representation. ### Claim 32 (depends on 30) The system of claim 30, further comprising instructions to: generate a floor plan of the location by a layout estimation algorithm configured to use lines in the 3D model, plane information related to the elements in the 3D model, vanishing points of the location, or geometric indicators related to the 3D model to estimate the floor plan of the location. ### Claim 33 (depends on 32) The system of claim 32, wherein the layout estimation algorithm is further configured to generate or update the floor plan based on constraints and heuristics, the heuristics comprising orthogonal junctions of the location, planar assumptions about a composition of the location, and cuboidal layout assumptions related to the location. ### Claim 34 (depends on 33) The system of claim 33, wherein the layout estimation algorithm includes a neural network configured to generate the floor plan using the lines, planes, points, and geometric indicators as inputs. ### Claim 35 (depends on 26) The system of claim 26, wherein the description data comprises one or more media types, the media types comprising at least one or more of video data, image data, audio data, text data, user interface/display data, and/or sensor data. ### Claim 36 A system configured to generate a virtual representation of a location with spatially localized information of elements within the location being embedded in the virtual representation, the system comprising one or more hardware processors configured by machine-readable instructions to: receive description data of a location, the description data being generated via at least one of a camera, a user interface, an environment sensor, and an external location information database, the description data comprising a plurality of images, and pose matrices; receive metadata associated elements within the location; generate, via a machine learning model and/or a geometric model, a 3-dimensional (3D) representation of the location and elements therein, the machine learning model being configured to receive the plurality of image and pose matrices as inputs and predict geometry of the location and the elements therein to form the 3D model; and generate, based on the 3D model of the location, a virtual representation of the location by annotating the 3D model with spatially localized metadata associated with the elements within the location, and semantic information of the elements within the location, the virtual representation being editable by a user to allow modifications to the spatially localized metadata. ### Claim 37 (depends on 36) The system of claim 36, wherein generating the virtual representation comprises: generating or updating the 3D model based on the real-time video stream of the location. ### Claim 38 (depends on 37) The system of claim 37, wherein the generating, in real-time, the virtual representation comprises: receiving, at a user device, the description data of the location, transmitting the description data to a server configured to execute the machine learning model to generate the 3D model of the location, generating, at the server based on the machine learning model and the description data, the virtual representation of the location, and transmitting the virtual representation to the user device. ### Claim 39 (depends on 36) The system of claim 36, wherein generating the 3D model comprises: encoding each image of the plurality of images with the machine learning model; adjusting, based on the encoded images of the plurality of images, an intrinsics matrix associated with the camera; using the intrinsics matrix and pose matrices to back-project the encoded images into a predefined voxel grid volume; provide the voxel grid with the features as input to a neural network to predicts a 3D model of the location for each voxel in the voxel grid; and extract a 2D surface of the predicted 3D model. ### Claim 40 (depends on 36) The system of claim 36, further comprising estimating intrinsics and extrinsics camera parameters using structure from motion (SFM) algorithm.