# US 10,872,464 B2 · Aligned augmented reality views > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,872,464 B2 - Document ID: US10872464B2 - Status: Granted - Kind: B2 - Application number: 16/176,510 - Filed: 2018-10-31 - Published: 2019-05-02 - Granted: 2020-12-22 - Earliest claimed: 2017-10-31 - Assignees: Outward, Inc. - Applicants: Outward, Inc. - Inventors: Manu Parmar; Brook Seaton; Clarence Chui - Web page: https://patentgrep.com/patent/US10872464B2 ## CPC - G06T 7/55: from multiple images - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/73: using feature-based methods - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 2207/20096: Interactive definition of curve of interest - G06T 2207/30244: Camera pose - G06T 2210/04: Architectural design, interior design - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Techniques for generating aligned, augmented reality views are disclosed. In some embodiments, an estimate of a floorplan of a three-dimensional space is first generated based on one or more views of the three-dimensional space from a camera of a device located in the three-dimensional space. A cost function is next minimized with respect to the estimate of the floorplan to generate a corrected floorplan having less error than the estimated floorplan, and a camera pose corresponding to the corrected floorplan is determined. Display of a rendering having the determined camera pose is facilitated in an actual view of the camera, resulting in an aligned, augmented reality view. ## Claims ### Claim 1 A method, comprising: receiving user input from a user comprising one or more lines specifying where one or more walls of a three-dimensional space meet a floor of the three-dimensional space; generating an estimate of a floorplan of the three-dimensional space based on the received user input and one or more views of the three-dimensional space from a camera of a client device of the user located in the three-dimensional space; minimizing a cost function with respect to the estimate of the floorplan to generate a corrected floorplan having less error than the estimated floorplan; determining a pose of the camera with respect to the corrected floorplan; and facilitating display of a rendering provided by a server that has the determined camera pose in an actual view of the camera of the client device of the user. ### Claim 2 (depends on 1) The method of claim 1, wherein facilitating display of the rendering in the actual view comprises facilitating real time display of the rendering in the actual view. ### Claim 3 (depends on 1) The method of claim 1, wherein the rendering and the actual view are aligned. ### Claim 4 (depends on 1) The method of claim 1, wherein the rendering and the actual view comprise a composite view. ### Claim 5 (depends on 1) The method of claim 1, wherein the rendering and the actual view comprise an augmented reality view. ### Claim 6 (depends on 1) The method of claim 1, wherein the rendering is displayed on top of the actual view. ### Claim 7 (depends on 1) The method of claim 1, wherein the rendering is overlaid on the actual view. ### Claim 8 (depends on 1) The method of claim 1, wherein the rendering is superimposed on the actual view. ### Claim 9 (depends on 1) The method of claim 1, wherein the cost function includes a set of one or more constraints that are based on foreknowledge of floorplan properties. ### Claim 10 (depends on 1) The method of claim 1, wherein the cost function includes a set of one or more constraints that are based on foreknowledge of floorplan geometries. ### Claim 11 (depends on 1) The method of claim 1, wherein the cost function includes a set of one or more constraints that are based on a known floorplan specification of the three-dimensional space. ### Claim 12 (depends on 1) The method of claim 1, wherein the cost function comprises a constraint that floorplan walls meet at right angles. ### Claim 13 (depends on 1) The method of claim 1, wherein the cost function comprises an error function. ### Claim 14 (depends on 1) The method of claim 1, wherein the three-dimensional space comprises an enclosed, indoor space. ### Claim 15 (depends on 1) The method of claim 1, wherein the three-dimensional space comprises a room. ### Claim 16 (depends on 1) The method of claim 1, wherein the three-dimensional space comprises a building. ### Claim 17 (depends on 1) The method of claim 1, wherein the one or more lines are drawn by the user on a display of the client device displaying the actual view of the camera. ### Claim 18 (depends on 1) The method of claim 1, further comprising extracting textures from camera images for three-dimensional model rendering. ### Claim 19 A system, comprising: a processor configured to: receive user input from a user comprising one or more lines specifying where one or more walls of a three-dimensional space meet a floor of the three-dimensional space; generate an estimate of a floorplan of the three-dimensional space based on the received user input and one or more views of the three-dimensional space from a camera of a client device of the user located in the three-dimensional space; minimize a cost function with respect to the estimate of the floorplan to generate a corrected floorplan having less error than the estimated floorplan; determine a pose of the camera with respect to the corrected floorplan; and facilitate display of a rendering provided by a server that has the determined camera pose in an actual view of the camera of the client device of the user; and a memory coupled to the processor and configured to provide the processor with instructions. ### Claim 20 (depends on 19) The system of claim 19, wherein to facilitate display of the rendering in the actual view comprises to facilitate real time display of the rendering in the actual view. ### Claim 21 (depends on 19) The system of claim 19, wherein the rendering and the actual view are aligned. ### Claim 22 (depends on 19) The system of claim 19, wherein the rendering and the actual view comprise a composite view. ### Claim 23 (depends on 19) The system of claim 19, wherein the rendering and the actual view comprise an augmented reality view. ### Claim 24 (depends on 19) The system of claim 19, wherein the rendering is displayed on top of the actual view. ### Claim 25 (depends on 19) The system of claim 19, wherein the rendering is overlaid on the actual view. ### Claim 26 (depends on 19) The system of claim 19, wherein the rendering is superimposed on the actual view. ### Claim 27 (depends on 19) The system of claim 19, wherein the cost function includes a set of one or more constraints that are based on foreknowledge of floorplan properties. ### Claim 28 (depends on 19) The system of claim 19, wherein the cost function includes a set of one or more constraints that are based on foreknowledge of floorplan geometries. ### Claim 29 (depends on 19) The system of claim 19, wherein the cost function includes a set of one or more constraints that are based on a known floorplan specification of the three-dimensional space. ### Claim 30 (depends on 19) The system of claim 19, wherein the cost function comprises a constraint that floorplan walls meet at right angles. ### Claim 31 (depends on 19) The system of claim 19, wherein the cost function comprises an error function. ### Claim 32 (depends on 19) The system of claim 19, wherein the three-dimensional space comprises an enclosed, indoor space. ### Claim 33 (depends on 19) The system of claim 19, wherein the three-dimensional space comprises a room. ### Claim 34 (depends on 19) The system of claim 19, wherein the three-dimensional space comprises a building. ### Claim 35 (depends on 19) The system of claim 19, wherein the one or more lines are drawn by the user on a display of the client device displaying the actual view of the camera. ### Claim 36 (depends on 19) The system of claim 19, wherein the processor is further configured to extract textures from camera images for three-dimensional model rendering. ### Claim 37 A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for: receiving user input from a user comprising one or more lines specifying where one or more walls of a three-dimensional space meet a floor of the three-dimensional space; generating an estimate of a floorplan of the three-dimensional space based on the received user input and one or more views of the three-dimensional space from a camera of a client device of the user located in the three-dimensional space; minimizing a cost function with respect to the estimate of the floorplan to generate a corrected floorplan having less error than the estimated floorplan; determining a pose of the camera with respect to the corrected floorplan; and facilitating display of a rendering provided by a server that has the determined camera pose in an actual view of the camera of the client device of the user. ### Claim 38 (depends on 37) The computer program product of claim 37, wherein facilitating display of the rendering in the actual view comprises facilitating real time display of the rendering in the actual view. ### Claim 39 (depends on 37) The computer program product of claim 37, wherein the rendering and the actual view are aligned. ### Claim 40 (depends on 37) The computer program product of claim 37, wherein the rendering and the actual view comprise a composite view. ### Claim 41 (depends on 37) The computer program product of claim 37, wherein the rendering and the actual view comprise an augmented reality view. ### Claim 42 (depends on 37) The computer program product of claim 37, wherein the rendering is displayed on top of the actual view. ### Claim 43 (depends on 37) The computer program product of claim 37, wherein the rendering is overlaid on the actual view. ### Claim 44 (depends on 37) The computer program product of claim 37, wherein the rendering is superimposed on the actual view. ### Claim 45 (depends on 37) The computer program product of claim 37, wherein the cost function includes a set of one or more constraints that are based on foreknowledge of floorplan properties. ### Claim 46 (depends on 37) The computer program product of claim 37, wherein the cost function includes a set of one or more constraints that are based on foreknowledge of floorplan geometries. ### Claim 47 (depends on 37) The computer program product of claim 37, wherein the cost function includes a set of one or more constraints that are based on a known floorplan specification of the three-dimensional space. ### Claim 48 (depends on 37) The computer program product of claim 37, wherein the cost function comprises a constraint that floorplan walls meet at right angles. ### Claim 49 (depends on 37) The computer program product of claim 37, wherein the cost function comprises an error function. ### Claim 50 (depends on 37) The computer program product of claim 37, wherein the three-dimensional space comprises an enclosed, indoor space. ### Claim 51 (depends on 37) The computer program product of claim 37, wherein the three-dimensional space comprises a room. ### Claim 52 (depends on 37) The computer program product of claim 37, wherein the three-dimensional space comprises a building. ### Claim 53 (depends on 37) The computer program product of claim 37, wherein the one or more lines are drawn by the user on a display of the client device displaying the actual view of the camera. ### Claim 54 (depends on 37) The computer program product of claim 37, further comprising computer instructions for extracting textures from camera images for three-dimensional model rendering.