# US 10,870,351 B2 · Method and apparatus for augmented reality based on localization and environmental conditions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,870,351 B2
- Document ID: US10870351B2
- Status: Granted
- Kind: B2
- Application number: 16/217,538
- Filed: 2018-12-12
- Published: 2020-06-18
- Granted: 2020-12-22
- Earliest claimed: 2018-12-12
- Assignees: HERE Global B.V.
- Applicants: HERE Global B.V.
- Inventors: Anirudh Viswanathan
- Web page: https://patentgrep.com/patent/US10870351B2

## CPC

- G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera}
- G01S 17/89: for mapping or imaging
- B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
- B60K 35/22: Display screens
- B60K 35/23: Head-up displays [HUD] (optical aspects of head-up displays G02B27/01)
- B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
- B60K 35/29: Instruments characterised by the way in which information is handled, e.g. showing information on plural displays or prioritising information according to driving conditions
- B60K 35/85: Arrangements for transferring vehicle- or driver-related data
- G01C 21/365: {Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself}
- G01C 21/3658: {Lane guidance}
- G01C 21/3679: {Retrieval, searching and output of POI information, e.g. hotels, restaurants, shops, filling stations, parking facilities (G01C21/3611 takes precedence)}
- G01C 21/3697: {Output of additional, non-guidance related information, e.g. low fuel level (G01C21/3679 takes precedence)}
- G01C 21/3815: {Road data}
- G01C 21/3837: {Data obtained from a single source}
- G06V 20/582: {of traffic signs}
- G06V 20/64: Three-dimensional [3D] objects
- B60K 2360/186: Displaying information according to relevancy
- B60K 2360/334: Projection means
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods described herein relate to using vision-based mapping to provide augmented reality of objects in the environment of the vehicle in response to objects being obscured. Methods may include: receiving sensor data from a vehicle traveling along a road segment; identifying a location of the vehicle on the road segment; retrieving map data from a map database of an environment of the road segment at the location of the vehicle on the road segment; identifying an object in the sensor data representing an obscured object in an environment of the vehicle; and providing for display of at least one object from the map data in an augmented reality of the environment, where the at least one object appears overlaid on the obscured object to a driver of the vehicle. The object in the sensor data may include a road sign having information relating to driving along the road segment.

## Claims

### Claim 1

An apparatus comprising at least one processor and at least one non-transitory memory including computer program code instructions, the computer program code instructions configured to, when executed, cause the apparatus to at least:
receive sensor data from a vehicle traveling along a road segment;
identify, based on the sensor data, a location of the vehicle on the road segment;
retrieve map data from a map database of an environment of the road segment at the location of the vehicle on the road segment, wherein the map data comprises a plurality of geo-referenced images associated with the road segment that are amenable to visual place-recognition;
identify an object in the sensor data representing an at least partially obscured object in an environment of the vehicle;
compare the sensor data to the plurality of geo-referenced images associated with the road segment;
identify an image of the plurality of geo-referenced images associated with the road segment as corresponding to the sensor data based on the comparison; and
provide for display of at least one object from the map data in an augmented reality of the environment, wherein the at least one object appears overlaid on the at least partially obscured object to an occupant of the vehicle.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the object in the sensor data comprises a road sign including information relating to driving along the road segment, wherein causing the apparatus to provide for display of at least one object from the map data in an augmented reality of the environment comprises causing the apparatus to provide for display of an image of the road sign stored in map data, wherein the image of the road sign appears, to a driver of the vehicle, to be overlaid on the road sign in the environment.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein causing the apparatus to identify, based on the sensor data, a location of the vehicle on the road segment comprises causing the apparatus to process the sensor data through a localization framework and receive, from the localization framework, an indication of the location of the vehicle on the road segment.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the sensor data comprises image sensor data, and wherein causing the apparatus to identify, based on the sensor data, a location of the vehicle on the road segment further comprises causing the apparatus to:
confirm a location associated with the image of the plurality of geo-referenced images as the location of the vehicle on the road segment.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein causing the apparatus to identify an object in the sensor data representing an at least partially obscured object in an environment of the vehicle comprises causing the apparatus to:
identify an object in the sensor data;
identify a corresponding object in the map data;
determine a proportion of the object in the sensor data that is obscured relative to the corresponding object in the map data; and
identify the object in the sensor data as an obscured object in response to the proportion of the object in the sensor data that is obscured satisfying a predetermined value.

### Claim 6 (depends on 5)

The apparatus of claim 5, wherein the predetermined value is dependent upon a type of object in the sensor data.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the computer program code instructions are configured to, when executed, further cause the apparatus to at least:
receive a first plurality of sensor data from at least a first plurality of vehicles travelling on one or more road segments;
compile the first plurality of sensor data into a platform specification format;
generate a first plurality of map data from at least the first plurality of sensor data compiled in the platform specification format;
generate the map database based on at least the platform specification format;
store the first plurality of map data in the map database;
receive a second plurality of sensor data from at least a second plurality of vehicles travelling on the one or more road segments; and
update the map database based on the second plurality of sensor data.

### Claim 8

A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising program code instructions to:
receive sensor data from a vehicle traveling along a road segment;
identify, based on the sensor data, a location of the vehicle on the road segment;
retrieve map data from a map database of an environment of the road segment at the location of the vehicle on the road segment, wherein the map data comprises a plurality of geo-referenced images associated with the road segment that are amenable to visual place-recognition;
identify an object in the sensor data representing an at least partially obscured object in an environment of the vehicle;
compare the sensor data to the plurality of geo-referenced images associated with the road segment;
identify an image of the plurality of geo-referenced images associated with the road segment as corresponding to the sensor data based on the comparison; and
provide for display of at least one object from the map data in an augmented reality of the environment, wherein the at least one object appears overlaid on the at least partially obscured object to an occupant of the vehicle.

### Claim 9 (depends on 8)

The computer program product of claim 8, wherein the object in the sensor data comprises a road sign including information relating to driving along the road segment, wherein the program code instructions to provide for display of at least one object from the map data in an augmented reality of the environment comprises program code instructions to provide for display of an image of the road sign stored in map data, wherein the image of the road sign appears, to a driver of the vehicle, to be overlaid on the road sign in the environment.

### Claim 10 (depends on 8)

The computer program product of claim 8, wherein the program code instructions to identify, based on the sensor data, a location of the vehicle on the road segment comprises program code instructions to process the sensor data through a localization framework and receive, from the localization framework, an indication of the location of the vehicle on the road segment.

### Claim 11 (depends on 8)

The computer program product of claim 8, wherein the sensor data comprises image sensor data, and wherein the program code instructions to identify, based on the sensor data, a location of the vehicle on the road segment further comprises program code instructions to:
confirm a location associated with the image of the plurality of geo-referenced images as the location of the vehicle on the road segment.

### Claim 12 (depends on 8)

The computer program product of claim 8, wherein the program code instructions to identify an object in the sensor data representing an at least partially obscured object in an environment of the vehicle comprises program code instructions to:
identify an object in the sensor data;
identify a corresponding object in the map data;
determine a proportion of the object in the sensor data that is obscured relative to the corresponding object in the map data; and
identify the object in the sensor data as an obscured object in response to the proportion of the object in the sensor data that is obscured satisfying a predetermined value.

### Claim 13 (depends on 12)

The computer program product of claim 12, wherein the predetermined value is dependent upon a type of object in the sensor data.

### Claim 14 (depends on 8)

The computer program product of claim 8, wherein the computer-executable program code instructions further comprise program code instructions to:
receive a first plurality of sensor data from at least a first plurality of vehicles travelling on one or more road segments;
compile the first plurality of sensor data into a platform specification format;
generate a first plurality of map data from at least the first plurality of sensor data compiled in the platform specification format;
generate the map database based on at least the platform specification format;
store the first plurality of map data in the map database;
receive a second plurality of sensor data from at least a second plurality of vehicles travelling on the one or more road segments; and
update the map database based on the second plurality of sensor data.

### Claim 15

A method comprising:
receiving sensor data from a vehicle traveling along a road segment;
identifying, based on the sensor data, a location of the vehicle on the road segment;
retrieving map data from a map database of an environment of the road segment at the location of the vehicle on the road segment, wherein the map data comprises a plurality of geo-referenced images associated with the road segment that are amenable to visual place-recognition;
identifying an object in the sensor data representing an at least partially obscured object in an environment of the vehicle;
comparing the sensor data to the plurality of geo-referenced images associated with the road segment;
identifying an image of the plurality of geo-referenced images associated with the road segment as corresponding to the sensor data based on the comparison; and
providing for display of at least one object from the map data in an augmented reality of the environment, wherein the at least one object appears overlaid on the at least partially obscured object to an occupant of the vehicle.

### Claim 16 (depends on 15)

The method of claim 15, wherein the object in the sensor data comprises a road sign including information relating to driving along the road segment, providing for display of at least one object from the map data in an augmented reality of the environment comprises providing for display of an image of the road sign stored in map data, wherein the image of the road sign appears, to a driver of the vehicle, to be overlaid on the road sign in the environment.

### Claim 17 (depends on 15)

The method of claim 15, wherein identifying, based on the sensor data, a location of the vehicle on the road segment comprises processing the sensor data through a localization framework and receiving, from the localization framework, an indication of the location of the vehicle on the road segment.

### Claim 18 (depends on 15)

The method of claim 15, wherein the sensor data comprises image sensor data, and wherein identifying, based on the sensor data, a location of the vehicle on the road segment further comprises:
confirming a location associated with the image of the plurality of geo-referenced images as the location of the vehicle on the road segment.

### Claim 19 (depends on 15)

The method of claim 15, wherein identifying an object in the sensor data representing an at least partially obscured object in an environment of the vehicle comprises:
identifying an object in the sensor data;
identifying a corresponding object in the map data;
determining a proportion of the object in the sensor data that is obscured relative to the corresponding object in the map data; and
identifying the object in the sensor data as an obscured object in response to the proportion of the object in the sensor data that is obscured satisfying a predetermined value.

### Claim 20 (depends on 15)

The method of claim 15, further comprising:
receiving a first plurality of sensor data from at least a first plurality of vehicles travelling on one or more road segments;
compiling the first plurality of sensor data into a platform specification format;
generating a first plurality of map data from at least the first plurality of sensor data compiled in the platform specification format;
generating the map database based on at least the platform specification format;
storing the first plurality of map data in the map database;
receiving a second plurality of sensor data from at least a second plurality of vehicles travelling on the one or more road segments; and
updating the map database based on the second plurality of sensor data.