# US 11,041,730 B2 · Reverse rendering of an image based on high definition map data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,041,730 B2 - Document ID: US11041730B2 - Status: Granted - Kind: B2 - Application number: 16/290,489 - Filed: 2019-03-01 - Published: 2019-09-05 - Granted: 2021-06-22 - Earliest claimed: 2018-03-02 - Assignees: Deepmap Inc. - Applicants: DeepMap Inc. - Inventors: Gil Colgate; Mark Damon Wheeler - Web page: https://patentgrep.com/patent/US11041730B2 ## CPC - G06T 11/00: Two-dimensional [2D] image generation - G01C 21/28: with correlation of data from several navigational instruments - G01C 21/3415: {Dynamic re-routing, e.g. recalculating the route when the user deviates from calculated route or after detecting real-time traffic data or accidents} - G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera} - G01C 21/3638: {including 3D objects and buildings (three dimensional [3D] modelling, e.g. data description of 3D objects G06T17/00; geographic models G06T17/05)} - G01C 21/3667: {Display of a road map (G01C21/3614 takes precedence; guidance using 3D or perspective road maps G01C21/3635)} - G01C 21/367: {Details, e.g. road map scale, orientation, zooming, illumination, level of detail, scrolling of road map or positioning of current position marker} - G01C 21/3811: {Point data, e.g. Point of Interest [POI]} - G01C 21/3841: {Data obtained from two or more sources, e.g. probe vehicles} - G01C 21/3848: {Data obtained from both position sensors and additional sensors} - G01C 21/3881: {Tile-based structures} - G01C 21/3889: {Transmission of selected map data, e.g. depending on route} - G01C 21/3896: {Transmission of map data from central databases} - G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00) - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 11/65: on geographic maps - G06V 10/82: using neural networks - G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 30/274: {Syntactic or semantic context, e.g. balancing} - G05D 1/0231: {using optical position detecting means (position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16)} - G05D 1/027: {comprising intertial navigation means, e.g. azimuth detector (inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165)} - G06T 2207/10004: Still image; Photographic image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20221: Image fusion; Image merging - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The autonomous vehicle generates an overlapped image by overlaying HD map data over sensor data and rendering the overlaid images. The visualization process is repeated as the vehicle drives along the route. The visualization may be displayed on a screen within the vehicle or at a remote device. The system performs reverse rendering of a scene based on map data from a selected point. For each line of sight originating at the selected point, the system identifies the farthest object in the map data. Accordingly, the system eliminates objects obstructing the view of the farthest objects in the HD map as viewed from the selected point. The system further allows filtering of objects using filtering criteria based on semantic labels. The system generates a view from the selected point such that 3D objects matching the filtering criteria are eliminated from the view. ## Claims ### Claim 1 A non-transitory computer readable storage medium storing instructions encoded thereon that, when executed by a processor, cause the processor to perform steps including: receiving, by a vehicle, sensor data comprising information about a route of the vehicle; performing localization based on the sensor data to determine a location of the vehicle; accessing map data corresponding to a region surrounding the determined location of the vehicle; identifying a reference point in the map data, the reference point being based on the location of the vehicle; for each respective angle of a plurality of angles in the region originating at the reference point and being with respect to a ground plane and a corresponding line passing through the reference point, selecting a corresponding point on a first object from the map data, the corresponding point being located from the reference point along the corresponding line, wherein at least some of the angles are associated with a second object located between the reference point and the first object and wherein the second object at least partially obstructs the first object as viewed from the reference point at one or more of the respective viewing angles; generating an image displaying the selected points from the reference point, wherein the generated image shows the first object and excludes the second object based on the second object at least partially obstructing the first object as viewed from the reference point; and transmitting the generated image for display on a client device. ### Claim 2 (depends on 1) The non-transitory computer readable storage medium of claim 1, wherein a height of the reference point is at a threshold height above a ground level that corresponds to the ground plane, the threshold height being greater than a height of objects in the region surrounding the determined location of the vehicle. ### Claim 3 (depends on 1) The non-transitory computer readable storage medium of claim 1, wherein the stored instructions further cause the processor to perform steps comprising: overlaying an image of the vehicle on the generated image. ### Claim 4 (depends on 1) The non-transitory computer readable storage medium of claim 1, wherein the steps are repeated to dynamically update the generated image as the vehicle drives along the route. ### Claim 5 (depends on 1) The non-transitory computer readable storage medium of claim 1, wherein the generated image shows a perspective view looking down upon the vehicle from the reference point. ### Claim 6 (depends on 1) The non-transitory computer readable storage medium of claim 1, wherein the stored instructions further cause the processor to perform steps comprising: moving the reference point as the vehicle moves; and transmitting the generated image to a remote display device for displaying a video, the video showing the region as observed from the moving reference point as the vehicle moves. ### Claim 7 (depends on 1) The non-transitory computer readable storage medium of claim 1, wherein the generated image is generated using orthographic projection. ### Claim 8 (depends on 1) The non-transitory computer readable storage medium of claim 1, wherein the steps are repeated for a plurality of locations of the vehicle to produce a plurality of pre-rendered images. ### Claim 9 A computer-implemented method comprising: receiving, by a vehicle, sensor data comprising information about a route of the vehicle; performing localization based on the sensor data to determine a location of the vehicle; accessing map data corresponding to a region surrounding the determined location of the vehicle; identifying a reference point in the map data, the reference point being based on the location of the vehicle; for each respective angle of a plurality of angles in the region originating at the reference point and being with respect to a ground plane and a corresponding line passing through the reference point, selecting a corresponding point on a first object from the map data, the corresponding point being located from the reference point along the corresponding line, wherein at least some of the angles are associated with a second object located between the reference point and the first object and wherein the second object at least partially obstructs the first object as viewed from the reference point at one or more of the respective viewing angles; generating an image displaying the selected points from the reference point, wherein the generated image shows the first object and excludes the second object based on the second object at least partially obstructing the first object as viewed from the reference point; and transmitting the generated image for display on a client device. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, wherein a height of the reference point is at a threshold height above a ground level that corresponds to the ground plane, the threshold height being greater than a height of objects in the region surrounding the determined location of the vehicle. ### Claim 11 (depends on 9) The computer-implemented method of claim 9, further comprising: overlaying an image of the vehicle on the generated image. ### Claim 12 (depends on 9) The computer-implemented method of claim 9, wherein the steps are repeated to dynamically update the generated image as the vehicle drives along the route. ### Claim 13 (depends on 9) The computer-implemented method of claim 9, wherein the generated image shows a perspective view looking down upon the vehicle from the reference point. ### Claim 14 (depends on 9) The computer-implemented method of claim 9, further comprising: moving the reference point as the vehicle moves; and transmitting the generated image to a remote display device for displaying a video, the video showing the region as observed from the moving reference point as the vehicle moves. ### Claim 15 (depends on 9) The computer-implemented method of claim 9, wherein the generated image is generated using orthographic projection. ### Claim 16 (depends on 9) The computer-implemented method of claim 9, wherein the steps are repeated for a plurality of locations of the vehicle to produce a plurality of pre-rendered images. ### Claim 17 A computer system comprising: one or more processors; and one or more non-transitory computer readable storage media storing instructions encoded thereon that, when executed by the one or more processors a, cause the system to perform steps comprising: receiving by a vehicle, sensor data comprising information about a route of the vehicle; performing localization based on the sensor data to determine a location of the vehicle; accessing map data corresponding to a region surrounding the determined location of the vehicle; identifying a reference point in the map data, the reference point being based on the location of the vehicle; for each respective angle of a plurality of angles in the region originating at the reference point and being with respect to a ground plane and a corresponding line passing through the reference point, selecting a corresponding point on a first object from the map data, the corresponding point being located from the reference point along the corresponding line, wherein at least some of the angles are associated with a second object located between the reference point and the first object and wherein the second object at least partially obstructs the first object as viewed from the reference point at one or more of the respective viewing angles; generating an image displaying the selected points from the reference point, wherein the generated image shows the first object and excludes the second object based on the second object at least partially obstructing the first object as viewed from the reference point; and transmitting the generated image for display on a client device. ### Claim 18 (depends on 17) The computer system of claim 17, wherein a height of the reference point is at a threshold height above a ground level that corresponds to the ground plane, the threshold height being greater than a height of objects in the region surrounding the determined location of the vehicle. ### Claim 19 (depends on 17) The computer system of claim 17, wherein the stored instructions further cause the system to perform steps comprising: overlaying an image of the vehicle on the generated image. ### Claim 20 (depends on 17) The computer system of claim 17, wherein the steps are repeated to dynamically update the generated image as the vehicle drives along the route. ### Claim 21 (depends on 17) The computer system of claim 17, wherein the generated image shows a perspective view looking down upon the vehicle from the reference point. ### Claim 22 (depends on 17) The computer system of claim 17, wherein the stored instructions further cause the system to perform steps comprising: moving the reference point as the vehicle moves; and transmitting the generated image to a remote display device for displaying a video, the video showing the region scene as observed from the moving reference point as the vehicle moves. ### Claim 23 (depends on 17) The computer system of claim 17, wherein the generated image is generated using orthographic projection. ### Claim 24 (depends on 17) The computer system of claim 17, wherein the steps are repeated for a plurality of locations of the vehicle to produce a plurality of pre-rendered images. ### Claim 25 (depends on 1) The non-transitory computer readable storage medium of claim 1, wherein the reference point is located at a height above the ground plane.