# US 10,816,346 B2 · Occupancy map updates based on sensor data collected by autonomous vehicles > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,816,346 B2 - Document ID: US10816346B2 - Status: Granted - Kind: B2 - Application number: 15/859,014 - Filed: 2017-12-29 - Published: 2018-07-05 - Granted: 2020-10-27 - Earliest claimed: 2016-12-30 - Assignees: Deepmap Inc. - Applicants: DeepMap Inc. - Inventors: Mark Damon Wheeler; Xiaqing Wu - Web page: https://patentgrep.com/patent/US10816346B2 ## CPC - G06V 10/764: using classification, e.g. of video objects - B60W 30/0956: {the prediction being responsive to traffic or environmental parameters} - B60W 40/02: related to ambient conditions - G01C 21/30: Map- or contour-matching - G01C 21/3635: {Guidance using 3D or perspective road maps} - G01C 21/3811: {Point data, e.g. Point of Interest [POI]} - G01C 21/3822: {Road feature data, e.g. slope data} - G01C 21/3841: {Data obtained from two or more sources, e.g. probe vehicles} - G01C 21/3848: {Data obtained from both position sensors and additional sensors} - G01C 21/3867: {Geometry of map features, e.g. shape points, polygons or for simplified maps} - G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00) - G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)} - G05D 1/0214: {in accordance with safety or protection criteria, e.g. avoiding hazardous areas (monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13)} - G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)} - G06F 16/29: Geographical information databases - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/24: Classification techniques - G06T 17/05: Geographic models - G06V 10/75: Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries - G06V 10/98: Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/582: {of traffic signs} - G06V 20/584: {of vehicle lights or traffic lights} - G06V 20/64: Three-dimensional [3D] objects - G08G 1/096827: {where the route is computed onboard} - H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) - H04L 67/12: specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks - H04L 67/52: specially adapted for the location of the user terminal - H04L 69/40: for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection (management of faults, events, alarms or notifications in data switching networks H04L41/06) - B60W 2554/00: Input parameters relating to objects - B60W 2556/50: of positioning data, e.g. GPS [Global Positioning System] data - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - H04L 67/53: using third party service providers - Y04S 40/18: Network protocols supporting networked applications, e.g. including control of end-device applications over a network - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - Y04S: SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS ## Abstract An online system builds a high definition (HD) map for a geographical region based on sensor data captured by a plurality of autonomous vehicles driving through a geographical region. The autonomous vehicles detect map discrepancies based on differences in the surroundings observed using sensor data compared to the high definition map and send messages describing these map discrepancies to the online system. The online system updates existing occupancy maps to improve the accuracy of the occupancy maps (OMaps), and to thereby improve passenger and pedestrian safety. While vehicles are in motion, they can continuously collect data about their surroundings. When new data is available from the various vehicles within a fleet, this can be updated in a local representation of the occupancy map and can be passed to the online HD map system (e.g., in the cloud) for updating the master occupancy map shared by all of the vehicles. ## Claims ### Claim 1 A method of maintaining a high definition map, comprising: receiving, from an online system by an autonomous vehicle, a high definition map for a geographical region, the high definition map comprising an occupancy map comprising a three-dimensional representation of the geographical region; receiving, from sensors of the autonomous vehicles, sensor data comprising images of the surroundings of the autonomous vehicle; determining a current location of the autonomous vehicle based on the sensor data; identifying a first set of objects in the three dimensional representation of the geographical region, wherein the one or more objects are expected to be visible from the current location of the autonomous vehicle; identifying a second set of objects based on the sensor data; comparing the first set of objects with the second set of objects; identifying a map discrepancy based on the comparison, the map discrepancy comprising, at least one of: an object identified by sensors that is not identifiable in the high definition map, an object in the high definition map that is not identifiable by the sensor data, and an object's appearance identified by sensors different from the object's representation in the high definition map; responsive to determining a significance value of the map discrepancy greater than a threshold, sending, by the autonomous vehicle, the map discrepancy to the online system; receiving a request from the online system for additional data associated with the map discrepancy; and sending the additional data associated with the map discrepancy to the online system. ### Claim 2 (depends on 1) The method of claim 1, further comprising registering the images of the surroundings in a three-dimensional coordinate system corresponding to the occupancy map to create a three-dimensional representation of the surroundings. ### Claim 3 (depends on 2) The method of claim 2, wherein identifying the second set of objects based on the sensor data comprises applying a machine learning model to the three-dimensional representation of the surroundings. ### Claim 4 (depends on 2) The method of claim 2, further comprising: identifying a plurality of objects based on the three-dimensional representation of the surroundings; and classifying each of the objects into a moving object group or a still object group; wherein each of the second set of objects is in the still object group. ### Claim 5 (depends on 4) The method of claim 4, further comprising removing the group of moving objects from the three-dimensional representation of the surroundings. ### Claim 6 (depends on 2) The method of claim 2, wherein identifying the second set of objects based on the sensor data comprises: identifying an object based on the sensor data; determining a likelihood of the identified object moving; and responsive to the likelihood of the identified object moving exceeding a threshold value, excluding the identified object from the second set of objects. ### Claim 7 (depends on 6) The method of claim 6, further comprising removing the identified object from the three-dimensional representation of the surroundings. ### Claim 8 (depends on 1) The method of claim 1, further comprising: receiving, from the sensors, newly-generated sensor data; and verifying the map discrepancy using the newly-generated sensor data. ### Claim 9 (depends on 1) The method of claim 1, further comprising: providing the map discrepancy to a user for verification. ### Claim 10 (depends on 2) The method of claim 2, wherein comparing the first set of objects with the second set of objects comprises: updating the high definition map based on the three-dimensional representation of the surroundings to create an updated high definition map; and comparing the updated high definition map to the high definition map. ### Claim 11 (depends on 1) The method of claim 1, further comprising: determining the significance value associated with the map discrepancy; and responsive to determining the significance value associated with the map discrepancy greater than a second threshold, sending the map discrepancy to the online system within a predetermined time interval. ### Claim 12 (depends on 1) The method of claim 1, further comprising: removing, from the sensor data, a set of sensor data unrelated to the map discrepancy; and sending only a set of sensor data related to the map discrepancy to the online system. ### Claim 13 (depends on 1) The method of claim 1, wherein the request for the additional data comprises an indication of a level of urgency, further comprising: responsive to the level of urgency indicating high urgency, sending the requested additional data within a threshold time interval. ### Claim 14 (depends on 13) The method of claim 13, wherein the request for additional data comprises an indication of a level of urgency, the method further comprising: responsive to the level of urgency indicating low urgency, waiting for a high speed connection; and responsive to detecting a high speed connection sending the requested additional data. ### Claim 15 (depends on 1) The method of claim 1, wherein the sensor data comprises images captured by camera, LIDAR scan data, GPS data, and EVIU data. ### Claim 16 (depends on 1) The method of claim 1, further comprising: detecting a failure in determining the current location of the autonomous vehicle; and responsive to detecting the failure, determining whether the failure was caused by sensor failure. ### Claim 17 A non-transitory computer readable storage medium storing instructions for: receiving, from an online system by an autonomous vehicle, a high definition map for a geographical region, the high definition map comprising an occupancy map comprising a three-dimensional representation of the geographical region; receiving, from sensors of the autonomous vehicles, sensor data comprising images of the surroundings of the autonomous vehicle; determining a current location of the autonomous vehicle based on the sensor data; identifying a first set of objects in the three dimensional representation of the geographical region, wherein the one or more objects are expected to be visible from the current location of the autonomous vehicle; identifying a second set of objects based on the sensor data; comparing the first set of objects with the second set of objects; identifying a map discrepancy based on the comparison, the map discrepancy comprising, at least one of: an object identified by sensors that is not identifiable in the high definition map, an object in the high definition map that is not identifiable by the sensor data, and an object's appearance identified by sensors different from the object's representation in the high definition map; responsive to determining a significance value of the map discrepancy greater than a threshold, sending, by the autonomous vehicle, the map discrepancy to the online system; receiving a request from the online system for additional data associated with the map discrepancy; and sending the additional data associated with the map discrepancy to the online system. ### Claim 18 (depends on 17) The non-transitory computer readable storage medium of claim 17, wherein the instructions comprise instructions for registering the images of the surroundings in a three-dimensional coordinate system corresponding to the occupancy map to create a three-dimensional representation of the surroundings. ### Claim 19 A computer system, comprising: an electronic processor; and a non-transitory computer readable storage medium storing instructions for: receiving, from an online system by an autonomous vehicle, a high definition map for a geographical region, the high definition map comprising an occupancy map comprising a three-dimensional representation of the geographical region; receiving, from sensors of the autonomous vehicles, sensor data comprising images of the surroundings of the autonomous vehicle; determining a current location of the autonomous vehicle based on the sensor data; identifying a first set of objects in the three dimensional representation of the geographical region, wherein the one or more objects are expected to be visible from the current location of the autonomous vehicle; identifying a second set of objects based on the sensor data; comparing the first set of objects with the second set of objects; identifying a map discrepancy based on the comparison, the map discrepancy comprising, at least one of: an object identified by sensors that is not identifiable in the high definition map, an object in the high definition map that is not identifiable by the sensor data, and an object's appearance identified by sensors different from the object's representation in the high definition map; responsive to determining a significance value of the map discrepancy greater than a threshold, sending, by the autonomous vehicle, the map discrepancy to the online system; receiving a request from the online system for additional data associated with the map discrepancy; and sending the additional data associated with the map discrepancy to the online system.