# US 2018/0321686 A1 · Systems and methods for projecting a location of a nearby object into a map according to a camera image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0321686 A1
- Document ID: US20180321686A1
- Status: Published application
- Kind: A1
- Application number: 15/585,878
- Filed: 2017-05-03
- Published: 2018-11-08
- Earliest claimed: 2017-05-03
- Assignees: Toyota Research Institute, Inc.
- Applicants: Toyota Research Institute, Inc.
- Inventors: Yusuke Kanzawa; Kuan-Hui Lee
- Web page: https://patentgrep.com/patent/US20180321686A1

## CPC

- G05D 1/0246: {using a video camera in combination with image processing means}
- B60W 30/00: Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
- G05D 1/027: {comprising intertial navigation means, e.g. azimuth detector (inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165)}
- G05D 1/0272: {comprising means for registering the travel distance, e.g. revolutions of wheels (measuring distance traversed on the ground by vehicles, e.g. using odometers G01C22/00)}
- G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)}
- G05D 1/0278: {using satellite positioning signals, e.g. GPS}
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes}
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

System, methods, and other embodiments described herein relate to locating an object within an image and projecting the object into a map. In one embodiment, a method includes, responsive to identifying a nearby vehicle in an image of an external environment surrounding a scanning vehicle, determining a relative location of the nearby vehicle in relation to the scanning vehicle using the image. The method includes projecting the nearby vehicle into a map according to at least the relative location determined from the image. The method includes controlling the scanning vehicle according to the map.

## Claims

### Claim 1

An object projection system associated with a scanning vehicle, comprising:
one or more processors;
a memory communicably coupled to the one or more processors and storing:
a projection module including instructions that when executed by the one or more processors cause the one or more processors to, responsive to identifying a nearby vehicle in an image of an external environment surrounding the scanning vehicle, determine a relative location of the nearby vehicle in relation to the scanning vehicle using the image,
wherein the projection module includes instructions to project the nearby vehicle into a map according to at least the relative location determined from the image; and
a control module including instructions that when executed by the one or more processors cause the one or more processors to control the scanning vehicle according to the map.

### Claim 2 (depends on 1)

The object projection system of claim 1, wherein the projection module further includes instructions to identify the nearby vehicle by analyzing the image to distinguish the nearby vehicle from the external environment and outlining the nearby vehicle using a bounding box.

### Claim 3 (depends on 1)

The object projection system of claim 1, wherein the projection module further includes instructions to determine the relative location of the nearby vehicle by (i) projecting a current centerline of a lane of the scanning vehicle along a roadway in the image, and (ii) determining a closest point of the current centerline to a bounding box that outlines the nearby vehicle.

### Claim 4 (depends on 3)

The object projection system of claim 3, wherein the projection module further includes instructions to determine the relative location by generating the relative location according to a calculated distance of the nearby vehicle from the closest point to the current centerline, and wherein the relative location is a position that is relative to the scanning vehicle.

### Claim 5 (depends on 3)

The object projection system of claim 3, wherein the projection module further includes instructions to determine the closest point by (i) extending an edge of the bounding box to intersect the current centerline at an intersection point and (ii) searching for the closest point to the bounding box among proximate points around the intersection point by calculating distances of the proximate points to the bounding box.

### Claim 6 (depends on 3)

The object projection system of claim 3, wherein the projection module further includes instructions to determine operating characteristics of the scanning vehicle by retrieving associated data from one or more sensors of the scanning vehicle, wherein the one or more sensors include a global position system (GPS) sensor, a tachometer, and an inertial monitoring unit (IMU) sensor, and wherein the operating characteristics include at least a current location of the scanning vehicle.

### Claim 7 (depends on 3)

The object projection system of claim 3, wherein the projection module further includes instructions to project the nearby vehicle into the map by localizing the nearby vehicle using the relative location and one or more operating characteristics of the scanning vehicle, wherein the projection module further includes instructions to project the current centerline by filtering secondary centerlines of additional travel lanes from the image that interfere with determining the closest point.

### Claim 8 (depends on 1)

The object projection system of claim 1, wherein the projection module further includes instructions to acquire the image of the external environment, wherein the image is from a camera and includes a field-of-view along a roadway around the scanning vehicle, and wherein the map is an obstacle map that identifies locations of obstacles in the external environment.

### Claim 9

A non-transitory computer-readable medium storing instructions that when executed by one or more processors cause the one or more processors to:
responsive to identifying a nearby vehicle in an image of an external environment surrounding a scanning vehicle, determine a relative location of the nearby vehicle in relation to the scanning vehicle as a function of the image;
project the nearby vehicle into a map according to at least the relative location determined from the image; and
control the scanning vehicle according to the map.

### Claim 10 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the instructions to determine the relative location of the nearby vehicle include instructions to (i) project a current centerline of a lane of the scanning vehicle along a roadway in the image, and (ii) determine a closest point of the current centerline to a bounding box that outlines the nearby vehicle.

### Claim 11 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein the instructions to determine the relative location include instructions to generate the relative location according to a calculated distance of the nearby vehicle from the closest point to the current centerline, and
wherein the instructions to determine the closest point include instructions to (i) extend an edge of the bounding box to intersect the current centerline at an intersection point and (ii) search for the closest point to the bounding box among proximate points around the intersection point by calculating distances of the proximate points to the bounding box.

### Claim 12 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein the instructions to determine operating characteristics of the scanning vehicle include instructions to retrieve associated data from one or more sensors of the scanning vehicle, wherein the one or more sensors include a global position system (GPS) sensor, a tachometer, and an inertial monitoring unit (IMU) sensor, and wherein the operating characteristics include at least a current location of the scanning vehicle.

### Claim 13 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein the instructions to project the nearby vehicle into the map include instructions to localize the nearby vehicle using the relative location and one or more operating characteristics of the scanning vehicle, wherein the instructions to project the current centerline include instructions to filter secondary centerlines of additional travel lanes from the image that interfere with determining the closest point.

### Claim 14

A method of localizing objects around a scanning vehicle using an image, comprising:
responsive to identifying a nearby vehicle in the image of an external environment surrounding the scanning vehicle, determining a relative location of the nearby vehicle in relation to the scanning vehicle using the image;
projecting the nearby vehicle into a map according to at least the relative location determined from the image; and
controlling the scanning vehicle according to the map.

### Claim 15 (depends on 14)

The method of claim 14, wherein identifying the nearby vehicle includes analyzing the image to distinguish the nearby vehicle from the external environment and outlining the nearby vehicle using a bounding box.

### Claim 16 (depends on 14)

The method of claim 14, wherein determining the relative location of the nearby vehicle includes (i) projecting a current centerline of a lane of the scanning vehicle along a roadway in the image, and (ii) determining a closest point of the current centerline to a bounding box that outlines the nearby vehicle.

### Claim 17 (depends on 16)

The method of claim 16, wherein determining the relative location includes generating the relative location according to a calculated distance of the nearby vehicle from the closest point to the current centerline, and wherein projecting the nearby vehicle into the map includes localizing the nearby vehicle using the relative location and one or more operating characteristics of the scanning vehicle, wherein the one or more operating characteristics include at least a current location of the scanning vehicle, and wherein the relative location is a position that is relative to the scanning vehicle.

### Claim 18 (depends on 16)

The method of claim 16, further comprising:
determining operating characteristics of the scanning vehicle by retrieving associated data from one or more sensors of the scanning vehicle, wherein the one or more sensors include a global position system (GPS) sensor, a tachometer, and an inertial monitoring unit (IMU) sensor, and
wherein projecting the current centerline includes filtering secondary centerlines of additional travel lanes from the image that interfere with determining the closest point.

### Claim 19 (depends on 16)

The method of claim 16, wherein determining the closest point includes (i) extending an edge of the bounding box to intersect the current centerline at an intersection point and (ii) searching for the closest point to the current centerline among proximate points around the intersection point by calculating distances of the proximate points to the bounding box.

### Claim 20 (depends on 14)

The method of claim 14, further comprising:
acquiring, by the scanning vehicle, the image of the external environment, wherein the image is from a camera and includes a field-of-view along a roadway around the scanning vehicle.
