Computer-vision based positioning for augmented reality navigation

A computer-implemented method for computer vision based positioning for Augmented Reality (AR) navigation, the method comprising:

• receiving a live camera image of a scene, the live camera image captured by a device at a viewpoint;
• scaling and transforming the live camera image for display on the device;
• reading, by the device, a current Global Positioning System (GPS) location;
• performing computer vision scene understanding on the live camera image to identify one or more semantic objects in the live camera image;
• obtaining navigation instructions from a navigation system;
• generating a route overlay and a point of interest from the navigation instructions;
• performing a computer vision-based positioning algorithm on the live camera image, comprising:
• feature matching a plurality of features in successive frames of the live camera image using a machine learning model,
• computing, based on the feature matching, an estimated motion of the viewpoint;
• predicting the movement of one or more landmarks in the live camera image based on the estimated motion of the viewpoint, and
• determining, based on the estimated motion and the actual movement of the landmarks in the live camera image, the relative position between the viewpoint and the landmarks;
• generating a 3D model of a local environment around the device based on the computer vision-based positioning algorithm;
• generating a visual display of the route overlay;
• transforming, via an affine transformation and based on the relative position between the viewpoint and the one or more landmarks determined via the computer vision-based position algorithm, the shape and size of the visual display of the route overlay such that the visual display appears textured on a surface of the 3D model of the local environment;
• determining coordinates in the live camera image corresponding to the transformed visual display of the route overlay in the 3D model of the local environment;
• and
• displaying the scene with an augmented reality overlay comprising the transformed visual display of the route overlay appearing textured on a surface of the scene and a visual display of the point of interest, wherein the location or shape of the visual display of the point of interest is transformed based on the computer vision-based positioning algorithm.