# US 12,602,452 B2 · Filtering of dynamic objects from vehicle generated map

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,602,452 B2
- Document ID: US12602452B2
- Status: Granted
- Kind: B2
- Application number: 17/645,620
- Filed: 2021-12-22
- Published: 2023-06-22
- Granted: 2026-04-14
- Earliest claimed: 2021-12-22
- Assignees: AUMOVIO Autonomous Mobility US, LLC
- Applicants: Julien Ip; Eduardo Jose Ramirez Llanos; Matthew Donald Berkemeier
- Inventors: Julien Ip; Eduardo Jose Ramirez Llanos; Matthew Donald Berkemeier
- Web page: https://patentgrep.com/patent/US12602452B2

## CPC

- G01S 13/931: of land vehicles
- G06F 18/251: {of input or preprocessed data}
- B60W 60/001: {Planning or execution of driving tasks}
- G01C 21/1652: {with ranging devices, e.g. LIDAR or RADAR}
- G01C 21/1656: {with passive imaging devices, e.g. cameras}
- G01C 21/30: Map- or contour-matching
- G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera}
- G01C 21/3807: {characterised by the type of data}
- G01C 21/3833: {characterised by the source of data}
- G01S 13/582: {adapted for simultaneous range and velocity measurements}
- G01S 13/584: {adapted for simultaneous range and velocity measurements}
- G01S 13/867: {Combination of radar systems with cameras}
- G01S 13/89: for mapping or imaging
- G06T 7/55: from multiple images
- G06T 7/579: from motion
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2420/408: Radar; Laser, e.g. lidar
- B60W 2554/404: Characteristics
- B60W 2556/50: of positioning data, e.g. GPS [Global Positioning System] data
- G01S 2013/932: {using own vehicle data, e.g. ground speed, steering wheel direction}
- G06T 2200/08: involving all processing steps from image acquisition to 3D model generation
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30241: Trajectory
- G06T 2207/30244: Camera pose
- G06T 2207/30264: Parking
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method and system for a vehicle control system generates maps utilized for charting a path of a vehicle through an environment. The method performed by the system obtains information indicative of vehicle movement from at least one vehicle system and images including objects within an environment from a camera mounted on the vehicle. The system uses the gathered information and images to create a depth map of the environment. The system also generates an image point cloud map from images taken with a vehicle camera and a radar point cloud map with velocity information from a radar sensor mounted on the vehicle. The depth map and the point cloud maps are fused together and any dynamic objects filtered out from the final map used for operation of the vehicle.

## Claims

### Claim 1

A method of creating a map of an environment surrounding a vehicle comprising:
obtaining, by a controller processor, information indicative of vehicle movement from at least one vehicle system, wherein the vehicle system comprises at least one of a wheel speed sensor, an acceleration sensor, an inertial measurement unit or a global positioning system;
obtaining, by the controller processor, images including objects within the environment surrounding the vehicle from a mono-camera mounted on the vehicle;
determining a pose of the mono-camera in a local reference frame;
creating, by the controller processor, a depth map of the environment surrounding the vehicle based on images obtained from the mono-camera, the pose of the mono-camera, and the information indicative of vehicle movement, the depth map associating a distance to the vehicle for each pixel in the images;
creating, by the controller processor, an image point cloud map of the environment surrounding the moving vehicle based on the depth map and based on the images obtained from the mono-camera mounted to the moving vehicle;
creating, by the controller processor, a radar point cloud map of object velocities with information from a radar sensor mounted on the vehicle and the information indicative of vehicle movement, wherein the radar point cloud map includes a plurality of points that are indicative of a relative velocity between the moving vehicle and the objects around the moving vehicle;
creating, by the controller processor, a first occupancy grid of the image point cloud map, and creating a second occupancy grid of the radar point cloud map;
creating, by the controller processor, a fused map by combining the first occupancy grid with the second occupancy grid and removing any dynamic objects from the fused map;
at least one of using, by the controller processor or another processor in the vehicle, the fused map for navigating the vehicle autonomously or semi-autonomously, or communicating, by the controller processor, the fused map to a vehicle driving control system of the vehicle for autonomously or semi-autonomously controlling the vehicle.

### Claim 2 (depends on 1)

The method as recited in claim 1, further comprising identifying, by the controller processor, an object as a dynamic object in response to a cluster of points within the radar point cloud map having a velocity that indicates movement relative to static features within the environment.

### Claim 3 (depends on 1)

The method as recited in claim 1, further comprising using, by the controller processor, the pose of the mono-camera in the creation of at least one of the image point cloud map or the radar point cloud map.

### Claim 4 (depends on 1)

The method as recited in claim 1, wherein the depth map includes points that are indicative of a distance between the vehicle and objects surrounding the vehicle.

### Claim 5 (depends on 1)

The method as recited in claim 1, wherein the object velocities comprise a Doppler velocity.

### Claim 6 (depends on 1)

The method as recited in claim 1, wherein the information indicative of vehicle movement is generated with a vehicle dynamic model.

### Claim 7 (depends on 1)

The method as recited in claim 1, wherein the image point cloud map and the radar point cloud map are each in three dimensions, and wherein the occupancy grid of the image point cloud map and the occupancy grid of the radar point cloud map are each in two dimensions.

### Claim 8

An autonomous vehicle system for creating a map of static objects within an environment surrounding the autonomous vehicle, the system comprising:
a controller processor configured to:
obtain information indicative of vehicle movement from vehicle navigation system comprising at least one of a wheel speed sensor, an acceleration sensor, an inertial measurement unit or a global positioning system;
obtain images of objects within the environment surrounding vehicle from a mono-camera mounted on the vehicle;
determine a pose of the mono-camera in a local reference frame;
create a depth map of the environment surrounding the vehicle based on images obtained from the mono-camera, the pose of the mono-camera and the information indicative of vehicle movement, the depth map associating a distance to the vehicle for each pixel in the images;
create an image point cloud map of the environment surrounding the moving vehicle based on the depth map and based on the images obtained from the mono-camera mounted to the moving vehicle;
create a radar point cloud map of object velocities with information from a radar sensor mounted on the vehicle and the information indicative of vehicle movement, wherein the radar point cloud map includes a plurality of points that are indicative of a relative velocity between the moving vehicle and the objects around the moving vehicle;
create a fused map by combining the occupancy grid of the image point cloud map with the occupancy grid of the radar point cloud map, and remove any dynamic objects; and
at least one of use the fused map to provide navigation for the vehicle or communicate the fused map to a vehicle driving control system of the vehicle for autonomously or semi-autonomously controlling the vehicle.

### Claim 9 (depends on 8)

The autonomous vehicle system as recited in claim 8, wherein the controller processor is further configured to identify an object as a dynamic object in response to a cluster of points within the radar point cloud map having a velocity that indicates movement relative to static features within the environment.

### Claim 10 (depends on 8)

The autonomous vehicle system as recited in claim 8, including a data storage medium that includes instructions executable by the controller processor.

### Claim 11

A non-transitory computer readable medium comprising stored instructions executable by a controller processor for creating a map of an environment surrounding a vehicle, the instructions comprising:
instructions prompting a controller processor to obtain images including objects within the environment surrounding the vehicle from a mono-camera mounted on the vehicle;
instructions prompting a controller processor to determine a pose of the mono-camera in a local reference frame;
instructions prompting a controller processor to create a depth map of the environment surrounding the vehicle based on the images obtained from the mono-camera, on information indicative of the vehicle movement from a navigation system of the vehicle comprising at least one of a wheel speed sensor, an acceleration sensor, a inertial measurement unit or a global positioning system, and on the pose of the mono-camera, the depth map associating a distance to the vehicle for each pixel in the images;
instructions prompting a controller processor to create an image point cloud map of the environment surrounding the moving vehicle based on the depth map and based on the images obtained from the mono-camera mounted to the moving vehicle;
instructions prompting the controller processor to create a radar point cloud map of object velocities with information from a radar sensor and the information indicative of vehicle movement, wherein the radar point cloud map includes a plurality of points that are indicative of a relative velocity between the moving vehicle and the objects around the moving vehicle;
instructions prompting the controller processor to create an occupancy grid of the image point cloud map, and create an occupancy grid of the radar point cloud map;
instructions prompting the controller processor to create a fused map by combining the occupancy grid of the image point cloud map with the occupancy grid of the radar point cloud map, and to remove any dynamic objects from the fused map; and
instructions prompting the controller processor to at least one of use the fused map to provide navigation for the vehicle or communicate the fused map to a vehicle driving control system of the vehicle for autonomously or semi-autonomously controlling the vehicle.

### Claim 12 (depends on 11)

The non-transitory computer readable medium as recited in claim 11, further including instructions for prompting the controller processor to identify an object as a dynamic object in response to a cluster of points within the radar point cloud map having a velocity that indicates movement relative to static features within the environment.

### Claim 13 (depends on 11)

The non-transitory computer readable medium as recited in claim 11, wherein the non-transitory computer readable medium comprises at least one of a volatile memory or nonvolatile memory.

### Claim 14 (depends on 11)

The non-transitory computer readable medium as recited in claim 11, wherein the image point cloud map and the radar point cloud map are each in three dimensions, and wherein the occupancy grid of the image point cloud map and the occupancy grid of the radar point cloud map are each in two dimensions.