# US 10,871,377 B1 · Computer-vision based positioning for augmented reality navigation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,871,377 B1 - Document ID: US10871377B1 - Status: Granted - Kind: B1 - Application number: 16/536,270 - Filed: 2019-08-08 - Granted: 2020-12-22 - Earliest claimed: 2019-08-08 - Assignees: Phiar Technologies, Inc. - Applicants: Phiar Technologies, Inc. - Inventors: Chen-Ping Yu; Xinyu Li; Jim Zin; Aaditya Chandrasekhar - Web page: https://patentgrep.com/patent/US10871377B1 ## CPC - G01C 21/32: Structuring or formatting of map data - G06T 7/579: from motion - G01C 21/34: Route searching; Route guidance - G01C 21/3632: {Guidance using simplified or iconic instructions, e.g. using arrows (G01C21/365 takes precedence)} - G01C 21/3644: {Landmark guidance, e.g. using POIs or conspicuous other objects} - G01C 21/3647: {Guidance involving output of stored or live camera images or video streams} - G01C 21/365: {Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself} - G06T 7/248: {involving reference images or patches} - G06T 7/74: {involving reference images or patches} - G06T 19/003: {Navigation within 3D models or images} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 20/20: in augmented reality scenes - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - G06V 30/274: {Syntactic or semantic context, e.g. balancing} - G06T 2207/10016: Video; Image sequence - G06T 2207/30241: Trajectory - G06T 2219/004: Annotating, labelling - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Systems and methods for a more usable Augmented Reality (AR) display of navigation indications is described. A live camera image of a scene may be captured from a device. Navigation instructions may be generated from a navigation system and a navigation indication may be generated for display. A computer vision-based positioning algorithm may be performed on the camera image to determine the relative position between the viewpoint of the device and one or more landmarks in the live camera image. The location or shape of the visual display of the navigation indication may be determined based on the computer vision-based positioning algorithm. ## Claims ### Claim 1 A computer-implemented method for computer vision based positioning for Augmented Reality (AR) navigation, the method comprising: receiving a live camera image of a scene, the live camera image captured by a device at a viewpoint; performing computer vision scene understanding on the live camera image; obtaining navigation instructions from a navigation system; generating a navigation indication from the navigation instructions, wherein the navigation indication is a route overlay; performing a computer vision-based positioning algorithm on the live camera image, comprising: feature matching a plurality of features in successive frames of the live camera image using a machine learning model, computing, based on the feature matching, an estimated motion of the viewpoint; predicting the movement of one or more landmarks in the live camera image based on the estimated motion of the viewpoint, and determining, based on the estimated motion and the actual movement of the landmarks in the live camera image, the relative position between the viewpoint and the landmarks; generating a 3D model of a local environment around the device based on the computer vision-based positioning algorithm; generating a visual display of the navigation indication; transforming, via an affine transformation and based on the relative position between the viewpoint and the one or more landmarks determined via the computer vision-based positioning algorithm, the shape and size of the visual display of the navigation indication such that the visual display appears textured on a surface of the 3D model of the local environment; determining coordinates in the live camera image corresponding to the transformed visual display of the navigation indication in the 3D model of the local environment; and displaying the scene with an augmented reality overlay comprising the transformed visual display of the navigation indication appearing textured on a surface of the scene. ### Claim 2 (depends on 1) The method of claim 1, wherein: the navigation indication is displayed to appear on top of the ground. ### Claim 3 (depends on 1) The method of claim 1, wherein: the navigation indication is a world-locked point of interest. ### Claim 4 (depends on 1) The method of claim 1, wherein: the computer vision scene understanding identifies one or more semantic objects in the live camera image. ### Claim 5 (depends on 1) The method of claim 1, wherein: the computer vision-based positioning algorithm is visual odometry. ### Claim 6 (depends on 5) The method of claim 5, further comprising: receiving the successive frames of the live camera image; detecting the plurality of features in one or more of the successive frames; and performing local optimization of the estimated motion. ### Claim 7 (depends on 1) The method of claim 1, wherein: the computer vision-based positioning algorithm is visual inertial odometry. ### Claim 8 (depends on 7) The method of claim 7, further comprising: receiving the successive frames of the live camera image; detecting the plurality of features in one or more of the successive frames; reading data from an inertial measurement unit and using the data for computing the estimated motion; and performing local optimization of the estimated motion. ### Claim 9 (depends on 1) The method of claim 1, wherein: the computer vision-based positioning algorithm is Simultaneous Localization and Mapping (SLAM). ### Claim 10 (depends on 9) The method of claim 9, further comprising: receiving the successive frames of the live camera image; extracting, for each of the successive frames, one or more SLAM landmarks from the frame; associating each SLAM landmark with one or more corresponding SLAM landmarks in prior frames of the live camera image; storing the estimated location; and updating one or more stored estimated locations of the SLAM landmarks, wherein determining the relative position between the viewpoint and the landmarks comprises using the SLAM landmarks as the one or more landmarks in the live camera image. ### Claim 11 A computer-implemented method for computer vision based positioning for Augmented Reality (AR) navigation, the method comprising: receiving a live camera image of a scene, the live camera image captured by a device at a viewpoint; scaling and transforming the live camera image for display on the device; reading, by the device, a current Global Positioning System (GPS) location; performing computer vision scene understanding on the live camera image to identify one or more semantic objects in the live camera image; obtaining navigation instructions from a navigation system; generating a route overlay and a point of interest from the navigation instructions; performing a computer vision-based positioning algorithm on the live camera image, comprising: feature matching a plurality of features in successive frames of the live camera image using a machine learning model, computing, based on the feature matching, an estimated motion of the viewpoint; predicting the movement of one or more landmarks in the live camera image based on the estimated motion of the viewpoint, and determining, based on the estimated motion and the actual movement of the landmarks in the live camera image, the relative position between the viewpoint and the landmarks; generating a 3D model of a local environment around the device based on the computer vision-based positioning algorithm; generating a visual display of the route overlay; transforming, via an affine transformation and based on the relative position between the viewpoint and the one or more landmarks determined via the computer vision-based position algorithm, the shape and size of the visual display of the route overlay such that the visual display appears textured on a surface of the 3D model of the local environment; determining coordinates in the live camera image corresponding to the transformed visual display of the route overlay in the 3D model of the local environment; and displaying the scene with an augmented reality overlay comprising the transformed visual display of the route overlay appearing textured on a surface of the scene and a visual display of the point of interest, wherein the location or shape of the visual display of the point of interest is transformed based on the computer vision-based positioning algorithm. ### Claim 12 (depends on 11) The method of claim 11, wherein: the shape of the visual display of the route overlay is transformed to appear on top of the ground. ### Claim 13 (depends on 11) The method of claim 11, wherein: the location of the visual display of the point of interest is transformed to appear locked to a location in a local environment. ### Claim 14 (depends on 11) The method of claim 11, wherein: the semantic objects comprise road lanes, moving objects, non-moving objects, environmental objects, and a horizon line. ### Claim 15 (depends on 11) The method of claim 11, wherein: the computer vision-based positioning algorithm is visual odometry. ### Claim 16 (depends on 11) The method of claim 11, further comprising: receiving the successive frames of the live camera image; detecting the plurality of features in one or more of the successive frames; and performing local optimization of the estimated motion. ### Claim 17 (depends on 11) The method of claim 11, wherein: the computer vision-based positioning algorithm is visual inertial odometry. ### Claim 18 (depends on 11) The method of claim 11, further comprising: receiving the successive frames of the live camera image; detecting the plurality of features in one or more of the successive frames; reading data from an inertial measurement unit and using the data for estimating motion; and performing local optimization of the estimated motion. ### Claim 19 (depends on 11) The method of claim 11, wherein: the computer vision-based positioning algorithm is Simultaneous Localization and Mapping (SLAM). ### Claim 20 (depends on 19) The method of claim 19, further comprising: receiving the successive frames of the live camera image; extracting, for each of the successive frames, one or more SLAM landmarks from the frame; associating each SLAM landmark with one or more corresponding SLAM landmarks in prior frames of the live camera image; storing the estimated location; and updating one or more stored estimated locations of the SLAM landmarks, wherein determining the relative position between the viewpoint and the landmarks comprises using the SLAM landmarks as the one or more landmarks in the live camera image.