# US 11,112,237 B2 · Using map information to smooth objects generated from sensor data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,112,237 B2
- Document ID: US11112237B2
- Status: Granted
- Kind: B2
- Application number: 15/350,473
- Filed: 2016-11-14
- Published: 2018-05-17
- Granted: 2021-09-07
- Earliest claimed: 2016-11-14
- Assignees: Waymo LLC
- Applicants: Waymo LLC
- Inventors: Nirmal Patel
- Web page: https://patentgrep.com/patent/US11112237B2

## CPC

- G01B 11/24: for measuring contours or curvatures
- B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
- B60K 35/22: Display screens
- B60K 35/29: Instruments characterised by the way in which information is handled, e.g. showing information on plural displays or prioritising information according to driving conditions
- B60K 35/60: Instruments characterised by their location or relative disposition in or on vehicles (arrangements of lighting devices on dashboards B60Q3/10)
- B60K 35/80: Arrangements for controlling instruments
- G01B 11/02: for measuring length, width or thickness (G01B11/08 takes precedence)
- G01B 11/04: specially adapted for measuring length or width of objects while moving
- G01C 21/30: Map- or contour-matching
- G01C 21/3697: {Output of additional, non-guidance related information, e.g. low fuel level (G01C21/3679 takes precedence)}
- G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46)
- G01S 17/931: of land vehicles
- G05D 1/024: {in combination with a laser (lasers per se H01S)}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- G06V 20/64: Three-dimensional [3D] objects
- B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
- B60K 2360/16: Type of output information
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The technology relates to generating a representation of an object detected by a perception system of a vehicle. For instance, a set of data points corresponding to an object is received from the perception system. Map information identifying shapes and locations of lanes as well as locations of center points for the lanes is retrieved. A representative point of the center points is selected based on the set of data points. A first position is determined based on the location of the representative point. A second position is determined based on the set of data points. The first and second positions are used to determine a third position for a representation of the object based on a distance the first position and the second position. The he representation of the object is generated using the third position. The representation of the object is displayed on a display of the vehicle.

## Claims

### Claim 1

A method for generating a smoothed representation of an object detected by a perception system of a vehicle, the method comprising:
receiving, by one or more processors, a set of data points from the perception system, the data points of the set of data points corresponding to the object detected during a sweep of a laser of a laser sensor;
retrieving, by the one or more processors, map information identifying shapes and locations of lanes as well as locations of center points for the lanes;
selecting, by the one or more processors, a representative point from the center points based on the set of data points;
determining, by the one or more processors, a first position for the smoothed representation of the object based on the location of the representative point;
determining, by the one or more processors, a second position for the smoothed representation of the object based on the set of data points;
using, by the one or more processors, the first position and the second position in order to determine a third position for the smoothed representation of the object based on a distance between the first position and the second position;
generating, by the one or more processors, the smoothed representation of the object using the third position; and
displaying, by the one or more processors, the smoothed representation of the object on a display of the vehicle.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first position and the second position are used such that:
when the distance is a first distance, the first position has a first contribution to the third position, and
when the distance is a second distance greater than the first distance, the second position has a second contribution to the third position that is greater than the first contribution.

### Claim 3 (depends on 1)

The method of claim 1, wherein the map information further includes heading information identifying headings for the lanes, and the method further comprising:
determining a first orientation based on the heading of the lane associated with the representative point;
determining a second orientation based on the set of data points; and
combining the first orientation and the second orientation in order to determine a third orientation for the smoothed representation of the object based on the distance, and wherein the smoothed representation for the object is generated using the third orientation.

### Claim 4 (depends on 3)

The method of claim 3, wherein the first orientation and the second orientation are used such that:
when the distance is a first distance, the first orientation has a first contribution to the third orientation, and
when the distance is a second distance greater than the first distance, the second orientation has a second contribution to the third orientation that is greater than the first contribution.

### Claim 5 (depends on 3)

The method of claim 3, further comprising determining a mix value based on a distance between the first position and the second position, and wherein the third orientation is determined using the equation: first_orientation*(1−mix value)+second_orientation*mix value, where first_orientation is the first orientation and the second_orientation is the second orientation.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
determining a first width for the smoothed representation of the object based on a width of the lane associated with the representative point;
determining a second width for the smoothed representation of the object based on the set of data points; and
combining the first width and the second width in order to determine a third width for the smoothed representation of the object based on the distance, and wherein the smoothed representation for the object is generated using the third width.

### Claim 7 (depends on 6)

The method of claim 6, wherein the first width and the second width are combined such that:
when the distance is a first distance, the first width has a first contribution to the third width, and
when the distance is a second distance greater than the first distance, the second width has a second contribution to the third width that is greater than the first contribution.

### Claim 8 (depends on 6)

The method of claim 6, further comprising determining a mix value based on a distance between the first position and the second position, and wherein the third width is determined using the equation: first_width*(1−mix value)+second_width*mix value, where first_width is the first width and the second_width is the second width.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
prior to receiving the set of data points, receiving an earlier set of data points corresponding to the object detected during a first sweep of the laser prior to the sweep of the laser; and
determining a length based on the earlier set of data points, and
wherein generating the smoothed representation of the object is further based on the length.

### Claim 10 (depends on 9)

The method of claim 9, wherein the length is determined further based on an angular width of a side of the object corresponding to the length from a perspective of the laser.

### Claim 11 (depends on 10)

The method of claim 10, wherein the length is determined further based on whether the angular width is less than a minimum threshold angular width value.

### Claim 12 (depends on 1)

The method of claim 1, further comprising determining a mix value based on a distance between the first position and the second position, and wherein determining the third position includes using the mix value in conjunction with the first position and the second position.

### Claim 13 (depends on 12)

The method of claim 12, further comprising determining the third position includes using the mix value to determine a first contribution by the first position and a second contribution by the second position.

### Claim 14 (depends on 13)

The method of claim 13, wherein determining the third position includes summing the first contribution and the second contribution.

### Claim 15 (depends on 12)

The method of claim 12, wherein the mix value is further determined based on a value representative of half a width of a lane.

### Claim 16 (depends on 12)

The method of claim 12, wherein the mix value is further based on a maximum distance value corresponding to one of half of an actual width of the lane at the location of the representative point, half an average width of the lane, or half an average width or default width value for lanes less a predetermined distance.

### Claim 17 (depends on 16)

The method of claim 16, wherein determining the mix value includes dividing the distance between the first position and the second position by the maximum distance value.

### Claim 18 (depends on 1)

The method of claim 1, further comprising, using information from the perception system regarding the object in order to control the vehicle in an autonomous driving mode, and wherein the smoothed representation of the object is not used to control the vehicle in the autonomous driving mode.

### Claim 19 (depends on 1)

The method of claim 1, wherein the smoothed representation reduces lateral shifts in the location of the object when displayed on the display.

### Claim 20 (depends on 1)

The method of claim 1, further comprising determining a mix value based on a distance between the first position and the second position, and wherein the third position is determined using the equation: first_position*(1−mix value)+second_position*mix value, where first_position is the first position and the second_position is the second position.

### Claim 21

A method for generating a smoothed representation of an object detected by a perception system of a vehicle, the method comprising:
receiving, by one or more processors, a set of data points from the perception system, the data points of the set of data points corresponding to the object detected during a sweep of a laser of a laser sensor;
determining a location of the object from the set of data points;
accessing map information identifying center points of traffic lanes to identify a center point of a traffic lane that is closest to the location;
determining a location for the smoothed representation of the object based on a distance between the identified center point and the location of the object; and
displaying, by the one or more processors, the smoothed representation of the object on a display of the vehicle based on the location for the smoothed representation of the object.

### Claim 22 (depends on 21)

The method of claim 21, wherein the map information further includes heading information identifying headings for the lanes, and the method further comprises:
determining an orientation of the object based on the set of data points; and
determining an orientation for the smoothed representation of the object based on the orientation of the object and the distance between the identified center point and the location of the object, and
wherein displaying the smoothed representation is further based on the orientation for the smoothed representation.

### Claim 23 (depends on 21)

The method of claim 21, further comprising:
determining a width of the object based on the set of data points; and
determining a width for the smoothed representation of the object based on the width of the object and the distance between the identified center point and the location of the object, and
wherein displaying the smoothed representation is further based on the width for the representation.