# US 10,482,659 B2 · System and method for superimposing spatially correlated data over live real-world images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,659 B2 - Document ID: US10482659B2 - Status: Granted - Kind: B2 - Application number: 14/799,859 - Filed: 2015-07-15 - Published: 2017-01-19 - Granted: 2019-11-19 - Earliest claimed: 2015-07-15 - Assignees: Oracle International Corporation - Applicants: Oracle International Corporation - Inventors: Xin Li; John R. Punin; Rashmi Raja - Web page: https://patentgrep.com/patent/US10482659B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Systems, methods, and other embodiments are disclosed that augment a visually displayed portion of a facility with superimposed virtual elements. In one embodiment, mobile position data is generated based on non-optical sensor readings taken by a mobile computing device. The mobile position data represents a location and an orientation of the mobile computing device in a three-dimensional (3D) space of the facility. Projection parameters are generated based on the mobile position data and are applied to modeled facility data to generate rendered image data. The modeled facility data represents hidden and unhidden elements of the facility within the 3D space. The rendered image data is superimposed on live real-time image data acquired by the mobile computing device and displayed by the mobile computing device. The projection parameters promote spatial alignment of the rendered image data with the live image data with respect to positions within the 3D space. ## Claims ### Claim 1 A method implemented by a mobile computing device, the method comprising: reading modeled facility data stored by a remote database device that is external to the mobile computing device and forms part of a computerized facility system, over a wireless communication network, into a memory of the mobile computing device, wherein the modeled facility data represents a hidden element and an unhidden element as virtual elements of a facility that are spatially correlated to positions in a three-dimensional space; displaying a live real-time image on a display screen of the mobile computing device, as acquired by an optical sensor of the mobile computing device, showing at least a live image of the unhidden element of the facility represented by the modeled facility data within the three-dimensional space; generating mobile position data, based on non-optical sensor readings taken by the mobile computing device, representing at least a location and an orientation of the mobile computing device in the three-dimensional space; generating projection parameters based at least in part on: (i) the mobile position data, (ii) optical scaling data, derived from characteristics of an optical lens of the optical sensor of the mobile computing device, and (iii) a size of the display screen of the mobile computing device, wherein the projection parameters are data relating a spatial arrangement of the virtual element representing the unhidden element of the facility in the modeled facility data relative to a spatial arrangement of the live image of the unhidden element of the facility appearing in the live real-time image, according to a view of the optical sensor, for projecting and spatially aligning the virtual elements representing the hidden element of the facility and the unhidden element of the facility from the modeled facility data onto locations corresponding to the hidden element and the unhidden element in the live real-time image; generating a rendered image, at least in part by applying the projection parameters to the modeled facility data, showing the virtual elements representing the hidden element and the unhidden element of the facility from the view of the optical sensor within the three-dimensional space; and superimposing the rendered image onto the live real-time image on the display screen based on at least the applied projection parameters for spatially aligning the virtual elements representing the unhidden element and the hidden element of the rendered image with the locations of the unhidden element and the hidden element in the live real-time image. ### Claim 2 (depends on 1) The method of claim 1, further comprising improving the spatial alignment in response to an input that interactively aligns the rendered image to the live real-time image on the display screen via a user interface of the mobile computing device. ### Claim 3 (depends on 1) The method of claim 1, further comprising: generating annotation data in response to user interaction with a user interface of the mobile computing device; associating the annotation data with one or more of the virtual elements representing the hidden element and the unhidden element of the facility from the modeled facility data in response to user interaction with the user interface; and displaying the annotation data on the display screen proximate a displayed virtual element representing one or more of the hidden element and the unhidden element. ### Claim 4 (depends on 1) The method of claim 1, further comprising: generating annotation data in response to user interaction with a user interface of the mobile computing device; associating the annotation data with one or more of the virtual elements representing the hidden element and the unhidden element of the facility represented by the modeled facility data in response to user interaction with the user interface; and storing the annotation data, as being associated with the one or more of the virtual elements representing the hidden element and the unhidden element of the facility, in a non-transitory computer storage medium of a computerized facility system that is external to the mobile computing device. ### Claim 5 (depends on 1) The method of claim 1, wherein the non-optical sensor readings include at least one of global positioning system sensor readings, cell tower triangulation sensor readings, barometric sensor readings, radio frequency parametric sensor readings, inertial sensor readings, compass sensor readings, and gyroscopic sensor readings. ### Claim 6 (depends on 1) The method of claim 1, wherein the characteristics of the optical lens of the optical sensor includes at least one of a focal length, a field-of-view, a lens distortion, and a zoom level. ### Claim 7 (depends on 1) The method of claim 1, further comprising generating the projection parameters further based on contextual default data to improve the spatial alignment. ### Claim 8 (depends on 7) The method of claim 7, wherein the contextual default data improves an accuracy of latitude and longitude data of the mobile position data. ### Claim 9 (depends on 7) The method of claim 7, wherein the contextual default data improves an accuracy of altitude data of the mobile position data. ### Claim 10 A mobile computing device, comprising: a display screen; an optical sensor that acquires a live real-time image to be displayed by the display screen; a non-optical sensor; at least one computer processor; and a non-transitory computer readable medium storing instructions that, when executed by the at least one computer processor, cause the mobile computing device to: read modeled facility data stored by a remote database device that is external to the mobile computing device and forms part of a computerized facility system, over a wireless communication network, into a memory of the mobile computing device, wherein the modeled facility data includes a virtual element representing a hidden element of the facility and a virtual element representing an unhidden element of the facility that are spatially correlated to positions in a three-dimensional space; generate mobile position data based on non-optical sensor readings taken by the non-optical sensor of the mobile computing device, wherein the mobile position data represents at least a location and an orientation of the mobile computing device in a three-dimensional space of a facility; generate projection parameters based at least in part on mobile position data stored in a database, wherein the projection parameters are data representing a spatial arrangement of the virtual element representing the unhidden element of the facility represented by modeled facility data, relative to a spatial arrangement of a live image of the unhidden element of the facility appearing in the live real-time image acquired by the optical sensor, according to a view of the optical sensor, for projecting and spatially aligning the virtual element representing the unhidden element and the virtual element representing the hidden element from the modeled facility data onto locations corresponding to the live image of the unhidden element and a position of the hidden element in the live real-time image, generate a rendered image at least in part by applying the projection parameters to the modeled facility data representing the virtual element representing the unhidden element and the virtual element representing the hidden element of the facility within the three-dimensional space, and superimpose the rendered image onto the live real-time image acquired by the optical sensor of the mobile computing device, wherein the live real-time image includes the live image of the unhidden element of the facility within the three-dimensional space, and wherein the virtual element representing the hidden element and the virtual element representing the unhidden element of the modeled facility in the rendered image are spatially aligned with respective locations of the position of the hidden element and the live image of the unhidden element in the live real-time image within the three-dimensional space. ### Claim 11 (depends on 10) The mobile computing device of claim 10, further comprising instructions stored in the non-transitory computer-readable medium that, when executed by the at least one computer processor, cause the mobile computing device to: generate contextual default data based at least in part on the mobile position data and the modeled facility data; and use the generated contextual default data to improve spatial alignment of the rendered image with the live real-time image. ### Claim 12 (depends on 10) The mobile computing device of claim 10, further comprising instructions stored in the non-transitory computer-readable medium that, when executed by the at least one computer processor, cause the mobile computing device to improve spatial alignment of the rendered image with the live real-time image in response to user interaction with a user interface of the mobile computing device. ### Claim 13 (depends on 10) The mobile computing device of claim 10, further comprising instructions stored in the non-transitory computer-readable medium that, when executed by the at least one processor, cause the mobile computing device to: generate annotation data in response to user interaction with a user interface of the mobile computing device; and associate the annotation data with one or more of the virtual element representing the hidden element and the virtual element representing the unhidden element of the facility represented by the modeled facility data in response to user interaction with the user interface of the mobile computing device. ### Claim 14 (depends on 10) The mobile computing device of claim 10, wherein the display screen is configured to display the rendered image data superimposed onto the live real-time image data. ### Claim 15 (depends on 10) The mobile computing device of claim 10, wherein the projection parameters are further based on at least optical scaling data, derived from: (i) characteristics of an optical lens of the optical sensor of the mobile computing device, and (ii) a size of the display screen of the mobile computing device, to improve the spatial alignment. ### Claim 16 A non-transitory computer-readable medium storing computer-executable instructions that, when executed by at least one computer processor of a mobile computing device, cause the mobile computing device to: read modeled facility data stored by a remote database device that is external to the mobile computing device and forms part of a computerized facility system, over a wireless communication network, and store the modeled facility data into a memory of the mobile computing device, wherein the modeled facility data includes a virtual element representing a hidden element of the facility and a virtual element representing an unhidden element of the facility that are spatially correlated to positions in a three-dimensional space; generate mobile position data based on non-optical sensor readings taken by the mobile computing device, wherein the mobile position data represents at least a location and an orientation of the mobile computing device in a three-dimensional space of a facility; generate projection parameters based at least in part on the mobile position data, wherein the projection parameters are data representing a spatial arrangement of the virtual element representing the an unhidden element of the facility represented by the modeled facility data read from the remote database, relative to a spatial arrangement of a live image of the unhidden element of the facility appearing in a live real-time image acquired by an optical sensor of the mobile computing device, according to a view of the optical sensor, for projecting and spatially aligning a virtual element representing the hidden element and the virtual element representing the unhidden element from the modeled facility data of the facility onto locations corresponding to the unhidden element and the hidden element in the live real-time image; generate a rendered image at least in part by applying the projection parameters to the modeled facility data representing the virtual element representing the unhidden element and the virtual element representing the hidden element of the facility within the three-dimensional space; and superimpose the rendered image onto the live real-time image acquired by the optical sensor of the mobile computing device, wherein the live real-time image includes the live image of the unhidden element of the facility within the three-dimensional space, and wherein the virtual element representing the hidden element and the virtual element representing the unhidden element of the modeled facility in the rendered image are spatially aligned with respective locations of the position of the hidden element and the live image of the unhidden element in the live real-time image within the three-dimensional space. ### Claim 17 (depends on 16) The non-transitory computer-readable medium of claim 16, further storing instructions that, when executed by the at least one processor, cause the mobile computing device to: generate contextual default data based at least in part on the mobile position data and the modeled facility data; and improve a spatial alignment of the rendered image with the live real-time image based on the contextual default data. ### Claim 18 (depends on 16) The non-transitory computer-readable medium of claim 16, further storing instructions that, when executed by the at least one processor, cause the mobile computing device to improve a spatial alignment of the rendered image with the live real-time image in response to user interaction with a user interface of the mobile computing device. ### Claim 19 (depends on 16) The non-transitory computer-readable medium of claim 16, further storing instructions that, when executed by the at least one processor, cause the mobile computing device to: generate annotation data in response to user interaction with a user interface of the mobile computing device; and associate the annotation data with one or more of the virtual element representing the hidden element and the virtual element representing the unhidden element of the facility represented by the modeled facility data in response to user interaction with the user interface of the mobile computing device. ### Claim 20 (depends on 16) The non-transitory computer-readable medium of claim 16, further storing instructions that, when executed by the at least one processor, cause the mobile computing device to: generate the projection parameters further based on at least optical scaling data, derived from: (i) characteristics of an optical lens of the optical sensor of the mobile computing device, and (ii) a size of the display screen of the mobile computing device, to improve the spatial alignment.