Image and point cloud based tracking and in augmented reality systems

A device comprising:

• a memory;
• and
• one or more processors coupled to the memory and configured to perform the operations comprising:
• generating data of a point cloud in relation to an environment associated with a first device;
• determining a first position estimate for an image sensor of a second device associated with the first device, the second device is separate from the first device and associated with a different location than the first position estimate;
• receiving a two-dimensional surface of an image by the image sensor of the second device, the image comprising an optical graphic located at a portion of the image;
• retrieving map data comprising at least one map location associated with the environment;
• processing the two-dimensional surface of the image and the optical graphic to determine a match of at least a portion of a set of key points of the point cloud to the two-dimensional surface of the image and the optical graphic;
• determining, based on the match of the at least a portion of the set of key points of the point cloud to the two-dimensional surface of the image and optical graphic, a position error associated with the first position estimate and a second position estimate for the image sensor of the second device;
• generating a model of a virtual object within the point cloud;
• generating a first augmented reality image comprising the virtual object in the environment using the second position estimate;
• and
• causing the model of the virtual object within the point cloud and the match of the at least a portion of the set of key points of the point cloud to the two-dimensional surface of the image and the optical graphic to be displayed in a map interface based on the map data.

The device of 11, wherein the one or more processors are configured to perform the operations further comprising:

• accessing a set of structure facade data describing one or more structure facades associated with the environment;
• identifying, using the set of structure facade data, a first structure facade portion of the two-dimensional surface of the image and the optical graphic of the environment, wherein the first structure facade portion comprises position and elevation information of the optical graphic, wherein the optical graphic matches first structure facade data of the set of structure facade data;
• and
• determining, based at least in part on the first structure facade portion of the two-dimensional surface of the image and the optical graphic of the environment, a third position estimate;
• wherein the first augmented reality image is generated using the third position estimate along with the second position estimate to align the virtual object within the two-dimensional surface of the image and the optical graphic of the environment.

A non-transitory computer readable medium comprising instructions that, when performed by a device, cause the device to perform operations comprising:

• generating data of a point cloud of an environment associated with a first device;
• determining a first position estimate for an image sensor of a second device associated with the first device, the second device is separate from the first device and associated with a different location than the first position estimate;
• receiving a two-dimensional surface of an image by the image sensor of the second device, the image comprising an optical graphic located at a portion of the image;
• retrieving map data comprising at least one map location associated with the environment;
• processing the two-dimensional surface of the image and the optical graphic to determine a match of at least a portion of a set of key points of the point cloud to the two-dimensional surface of the image and the optical graphic;
• determining, based on the match of the at least a portion of the set of key points of the point cloud to the two-dimensional surface of the image and optical graphic, a position error associated with the first position estimate and a second position estimate for the image sensor of the second device;
• generating a model of a virtual object within the point cloud;
• generating a first augmented reality image comprising the virtual object in the environment using the second position estimate;
• and
• causing the model of the virtual object within the point cloud and the match of the at least a portion of the set of key points of the point cloud to the two-dimensional surface of the image and the optical graphic to be displayed in a map interface based on the map data.