# US 10,769,440 B1 · Visual-inertial positional awareness for autonomous and non-autonomous tracking > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,769,440 B1 - Document ID: US10769440B1 - Status: Granted - Kind: B1 - Application number: 16/550,143 - Filed: 2019-08-23 - Granted: 2020-09-08 - Earliest claimed: 2016-08-29 - Assignees: Trifo, Inc. - Applicants: Trifo, Inc. - Inventors: Zhe Zhang; Grace Tsai; Shaoshan Liu - Web page: https://patentgrep.com/patent/US10769440B1 ## CPC - G06T 17/05: Geographic models - G01C 21/1656: {with passive imaging devices, e.g. cameras} - G01C 21/206: {specially adapted for indoor navigation} - G01C 21/3807: {characterised by the type of data} - G01C 21/3848: {Data obtained from both position sensors and additional sensors} - G06F 18/29: {Graphical models, e.g. Bayesian networks} - G06T 7/74: {involving reference images or patches} - G06V 20/20: in augmented reality scenes - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06T 2207/10016: Video; Image sequence - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/30244: Camera pose - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The described positional awareness techniques employing visual-inertial sensory data gathering and analysis hardware with reference to specific example implementations implement improvements in the use of sensors, techniques and hardware design that can enable specific embodiments to provide positional awareness to machines with improved speed and accuracy. ## Claims ### Claim 1 A system including: two or more mobile autonomous units, including a first autonomous unit and a second autonomous unit, each having a mobile platform and disposed thereon: a visual sensor comprising cameras providing capturing images including at least two frames, thereby providing a 360-degrees view about a centerline of the mobile platform; and at least one of: a multi-axis inertial measuring unit (IMU) sensor capable of providing measurement of at least acceleration using one or more accelerometers; and a global positioning system (GPS) receiver; and a map server, including a processor and a coupled memory storing instructions to build 3D maps of surrounding scenery encountered by the autonomous units using information sourced by the autonomous units, which instructions when executed by the processor perform: receiving a set of keyrigs from a first autonomous unit, each keyrig comprising visual information of surrounding scenery captured by the visual sensor of the first autonomous unit and a position of the first autonomous unit where the visual information was captured, the position of the autonomous unit generated using combinations of global positioning system (GPS) receiver, the multi-axis inertial measurement unit (IMU), and visual information of the surrounding scenery by the first autonomous unit during travel from a starting point to an end point; classifying at least one of one or more objects from the visual information of the surrounding scenery from the set of keyrigs into a set of moving objects and a set of non-moving objects; selecting a subset of keyrig from the set of keyrigs and determining a sparse 3D mapping of object feature points taken from the visual information of the surrounding scenery from the subset of keyrigs for the set of non-moving objects; building a 3D map of object feature points from the sparse 3D mapping of object feature points using visual information of the surrounding scenery from the set of keyrigs for the set of non-moving objects, wherein the 3D map is accurate within 20 centimeters; and providing the 3D map via a communications link to one or more additional autonomous units to guide the one or more additional autonomous units at a future time. ### Claim 2 (depends on 1) The system of claim 1, further comprising instructions that when executed by the processor perform: receiving from the second autonomous unit a set of keyrigs while hosting a first 3D map, each keyrig comprising visual information of the surrounding scenery and a position of the first autonomous unit where the visual information was captured, the position of the first autonomous unit generated using combinations of global positioning system, inertial measurement unit, and visual information of the surrounding scenery by the second autonomous unit; generating a second 3D map using the set of keyrigs from the second autonomous unit; and merging the second 3D map with the first 3D map. ### Claim 3 (depends on 2) The system of claim 2, wherein the first 3D map and the second 3D map both cover common location and further comprising instructions that when executed by the processor perform: updating the first 3D map using the second 3D map. ### Claim 4 (depends on 1) The system of claim 1, further comprising instructions that when executed by the processor perform: generating several hundred thousand images during one hour of operation by an autonomous unit. ### Claim 5 (depends on 1) The system of claim 1, further comprising instructions that when executed by the processor perform: substantially contemporaneously tracking a position of moving autonomous units against 3D maps. ### Claim 6 (depends on 1) The system of claim 1, further comprising instructions that when executed by the processor perform: storing a time of day with at least one 3D map. ### Claim 7 (depends on 1) The system of claim 1, further comprising instructions that when executed by the processor perform: storing a weather condition with at least one 3D map. ### Claim 8 (depends on 1) The system of claim 1, further comprising instructions that when executed by the processor perform: identifying an object feature point to be included in the sparse 3D mapping of object feature points: extracting a first set 2D features of the object from a first 360-degrees image in a keyrig selected from a subset of keyrigs; extracting a second set of 2D features of the object from a second 360-degrees image in the selected keyrig; receiving a position of the autonomous unit when the 360-degrees images were captured including longitude and latitude as input; triangulating the first set of 2D features from the first 360-degrees image and the second set of 2D features from the 360-degrees second image to derive location for feature points of the object relative to the position of the autonomous unit; and generating for at least one feature point of the object, a global position, including longitude, latitude, and height and adding the global position and feature descriptors of the object to the sparse mapping of object feature points. ### Claim 9 (depends on 1) The system of claim 1, further comprising instructions that when executed by the processor perform identifying an object feature point to be included in the 3D map of object feature points, by: extracting a first set 2D features of the object from a first 360-degrees image in a keyrig selected from a set of keyrigs provided by an autonomous unit; extracting a second set of 2D features of the object from a second 360-degrees image in the selected keyrig; receiving a position of the autonomous unit when the 360-degrees images were captured including longitude and latitude as input; triangulating the first set of 2D features from the first 360-degrees image and the second set of 2D features from the 360-degrees second image to derive location for feature points of the object relative to the position of the autonomous unit; and generating for at least one feature point of the object, a global position, including longitude, latitude, and height and adding the global position and feature descriptors of the object to the 3D map of object feature points. ### Claim 10 (depends on 1) The system of claim 1, wherein accuracy is determined by a difference between a location of an object depiction on the 3D map and an actual location of the object in space. ### Claim 11 (depends on 10) The system of claim 10, further including at least one 3D map with an accuracy within 10 centimeters. ### Claim 12 (depends on 1) The system of claim 1, wherein the starting point of travel of the first autonomous unit is the end point of the travel of the first autonomous unit. ### Claim 13 (depends on 1) The system of claim 1, wherein the multi-axis inertial measuring unit (IMU) sensor further includes one or more gyroscopes for reporting a rotational rate, and wherein the position of the autonomous unit generated using combinations of global positioning system (GPS) receiver, the multi-axis inertial measurement unit (IMU), and visual information of the surrounding scenery by the first autonomous unit during travel from a starting point to an end point further includes: generating position of the autonomous unit using the rotational rate from the one or more gyroscopes. ### Claim 14 (depends on 13) The system of claim 13, wherein the multi-axis inertial measuring unit (IMU) sensor further including three gyroscopes and three accelerometers mounted substantially orthogonal to each other, and the coupled memory further storing instructions for performing: determining misalignment between the IMU and the mobile unit by performing a sighting estimation to determine an offset between an IMU measurement frame and a sensor (objective) frame. ### Claim 15 (depends on 14) The system of claim 14, wherein the coupled memory further storing instructions for performing: correcting misalignment correction using the offset determined between the IMU measurement frame and the sensor (objective) frame. ### Claim 16 (depends on 14) The system of claim 14, wherein the coupled memory further storing instructions for performing continuous estimation and correction during system operation to minimize effect of the offset determined between the IMU measurement frame and the sensor (objective) frame. ### Claim 17 (depends on 14) The system of claim 14, wherein the offset determined between the IMU measurement frame and the sensor (objective) frame is a transformation stored by the autonomous unit. ### Claim 18 A method for building 3D maps from a surrounding scenery using information sourced by one or more moving autonomous units that include at least a camera visual sensor and at least one selected from a global positioning system and an inertial measurement unit, including: receiving a set of keyrigs from a first autonomous unit, each keyrig comprising visual information of the surrounding scenery and a position of the first autonomous unit where the visual information was captured, the position of the autonomous unit generated using combinations of global positioning system, inertial measurement unit, and visual information of the surrounding scenery by the first autonomous unit during travel from a starting point to an end point; classifying at least one of one or more objects from the visual information of the surrounding scenery from the set of keyrigs into a set of moving objects and a set of non-moving objects; selecting a subset of keyrig from the set of keyrigs and determining a sparse 3D mapping of object feature points taken from the visual information of the surrounding scenery from the subset of keyrigs for the set of non-moving objects; building a 3D map of object feature points from the sparse 3D mapping of object feature points using visual information of the surrounding scenery from the set of keyrigs for the set of non-moving objects, wherein the 3D map is accurate within 20 centimeters; and providing the 3D map via a communications link to one or more additional autonomous units to guide the one or more additional autonomous units at a future time. ### Claim 19 A non-transitory computer readable medium storing instructions for building 3D maps from surrounding scenery using information sourced by one or more moving autonomous units, which instructions when executed by a processor perform: receiving a set of keyrigs from a first autonomous unit, each keyrig comprising visual information from surrounding scenery and a position of the first autonomous unit where the visual information was captured, the position of the autonomous unit generated using combinations of global positioning system, inertial measurement unit, and visual information of the surrounding scenery by the first autonomous unit during travel from a starting point to an end point; classifying at least one of one or more objects from the visual information of the surrounding scenery from the set of keyrigs into a set of moving objects and a set of non-moving objects; selecting a subset of keyrig from the set of keyrigs and determining a sparse 3D mapping of object feature points taken from the visual information of the surrounding scenery from the subset of keyrigs for the set of non-moving objects; building a 3D map of object feature points from the sparse 3D mapping of object feature points using visual information of the surrounding scenery from the set of keyrigs for the set of non-moving objects, wherein the 3D map is accurate within 20 centimeters; and providing the 3D map via a communications link to one or more additional autonomous units to guide the one or more additional autonomous units at a future time. ### Claim 20 A method for building 3D maps from a surrounding scenery using information sourced by one or more moving autonomous units that include at least a camera visual sensor and at least one selected from a global positioning system and an inertial measurement unit, including: receiving a set of keyrigs from a first autonomous unit, each keyrig comprising visual information of the surrounding scenery and a position of the first autonomous unit where the visual information was captured, the position of the autonomous unit generated using combinations of global positioning system, inertial measurement unit, and visual information of the surrounding scenery by the first autonomous unit during travel from a starting point to an end point; classifying at least one of one or more objects from the visual information of the surrounding scenery from the set of keyrigs into a set of moving objects and a set of non-moving objects; selecting a subset of keyrig from the set of keyrigs and determining a sparse 3D mapping of object feature points taken from the visual information of the surrounding scenery from the subset of keyrigs for the set of non-moving objects; building a 3D map of object feature points from the sparse 3D mapping of object feature points using visual information of the surrounding scenery from the set of keyrigs for the set of non-moving objects, wherein the 3D map is accurate within 20 centimeters; and providing the 3D map via a communications link to a map server that stores and distributes 3D maps to guide the autonomous unit at a future time and at least one other autonomous unit.