# US 11,908,200 B2 · System and method in the prediction of target vehicle behavior based on image frame and normalization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,200 B2
- Document ID: US11908200B2
- Status: Granted
- Kind: B2
- Application number: 17/305,701
- Filed: 2021-07-13
- Published: 2023-01-19
- Granted: 2024-02-20
- Earliest claimed: 2021-07-13
- Assignees: Canoo Technologies Inc.
- Applicants: Canoo Technologies Inc.
- Inventors: Kilsoo Kim; Jongmoo Choi; Mayukh Sattiraju; Siddharth Agarwal
- Web page: https://patentgrep.com/patent/US11908200B2

## CPC

- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- B60W 30/12: Lane keeping
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2552/53: Road markings, e.g. lane marker or crosswalk
- G06V 2201/08: Detecting or categorising vehicles
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT

## Abstract

An apparatus includes at least one camera configured to capture a series of image frames for traffic lanes in front of an ego vehicle, where each of the series of image frames is captured at a different one of a plurality of times. A target object detection and tracking controller is configured to process each of the image frames using pixel measurements extracted from the respective image frame to determine, from the pixel measurements, a predicted time to line crossing for a target vehicle detected in the respective image frame at a time corresponding to capture of the respective image frame.

## Claims

### Claim 1

A method, comprising:
capturing a series of image frames for traffic lanes in front of an ego vehicle using at least one camera, wherein each of the series of image frames is captured at a different one of a plurality of times, and wherein the traffic lanes include an ego traffic lane occupied by the ego vehicle and an adjacent traffic lane for traffic traveling in a same direction as traffic within the ego traffic lane;
processing each of the image frames to detect a target vehicle in the adjacent traffic lane within the respective image frame at a time corresponding to capture of the respective image frame;
determining, using pixel measurements extracted from the image frames, a predicted time to line crossing for the detected target vehicle to cross into the ego traffic lane from the adjacent traffic lane;
determining a rate of arrival R(k) of the target vehicle from the ratio Ln(k) at time k and the ratio Ln(k−1) at time k−1; and
controlling the ego vehicle based on the predicted time to line crossing by the target vehicle into the ego traffic lane from the adjacent traffic lane.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the rate of arrival R(k) is determined from:
R
⁡
(
k
)
=
L
n
(
k
)
-
L
n
(
k
-
1
)
Δ
⁢
t
,
where Δt is a difference between the time k and the time k−1.

### Claim 3 (depends on 1)

The method according to claim 1, further comprising:
determining a time to line crossing TTLC (k) for the target vehicle at the time k.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the time to line crossing TTLC (k) is determined from:
TTLC
⁡
(
k
)
=
L
n
(
k
)
R
⁡
(
k
)
.

### Claim 5

An apparatus, comprising:
at least one camera configured to capture a series of image frames for traffic lanes in front of an ego vehicle, wherein each of the series of image frames is captured at a different one of a plurality of times, and wherein the traffic lanes include an ego traffic lane occupied by the ego vehicle and an adjacent traffic lane for traffic traveling in a same direction as traffic within the ego traffic lane; and
a target object detection and tracking controller configured to:
process each of the image frames to detect a target vehicle in the adjacent traffic lane within the respective image frame at a time corresponding to capture of the respective image frame,
determine, using pixel measurements extracted from the image frames, a predicted time to line crossing for the detected target vehicle to cross into the ego traffic lane from the adjacent traffic lane, and
control the ego vehicle based on the predicted time to line crossing by the target vehicle into the ego traffic lane from the adjacent traffic lane.

### Claim 6 (depends on 5)

The apparatus according to claim 5, wherein the at least one camera is positioned and configured to capture images of at least a portion of the ego traffic lane occupied by the ego vehicle and at least an adjoining portion of the adjacent traffic lane.

### Claim 7 (depends on 5)

The apparatus according to claim 5, wherein, to process each of the image frames, the target object detection and tracking controller is configured to determine a width in pixels of an ego traffic lane and a lateral offset in pixels of the target vehicle.

### Claim 8 (depends on 7)

The apparatus according to claim 7, wherein, to process each of the image frames, the target object detection and tracking controller is further configured to determine a ratio Ln of the lateral offset of the target vehicle Lt the width of the ego traffic lane Le.

### Claim 9 (depends on 5)

The ego vehicle comprising the apparatus according to claim 5, the ego vehicle further comprising:
a motor configured to drive wheels of the ego vehicle;
a chassis supporting axles on which the wheels are mounted, the wheels coupled to a steering mechanism configured to receive a steering angle control signal based on the predicted time to line crossing TTLC (k) for the target vehicle from the target object detection and tracking controller; and
a brake actuator configured to actuate brakes for one or more of the wheels, the brake actuator configured to receive a braking control signal based on the predicted time to line crossing TTLC (k) for the target vehicle from the target object detection and tracking controller.

### Claim 10 (depends on 9)

The ego vehicle according to claim 9, wherein the ego vehicle is an electric vehicle and the motor is an electric motor.

### Claim 11

An apparatus, comprising:
at least one camera configured to capture a series of image frames for traffic lanes in front of an ego vehicle, wherein each of the series of image frames is captured at a different one of a plurality of times; and
a target object detection and tracking controller configured to:
process each of the image frames using pixel measurements extracted from the respective image frame to determine, from the pixel measurements, a predicted time to line crossing for a target vehicle detected in the respective image frame at a time corresponding to capture of the respective image frame, wherein, to process each of the image frames, the target object detection and tracking controller is further configured to
determine a width in pixels of an ego traffic lane and a lateral offset in pixels of the target vehicle,
determine a ratio Ln of the lateral offset of the target vehicle Lt the width of the ego traffic lane Le, and
determine a rate of arrival R(k) of the target vehicle from the ratio Ln(k) at time k and the ratio Ln(k−1) at time k−1, and control the ego vehicle based on the predicted time to line crossing.

### Claim 12 (depends on 11)

The apparatus according to claim 11, wherein the rate of arrival R(k) is determined from:
R
⁡
(
k
)
=
L
n
(
k
)
-
L
n
(
k
-
1
)
Δ
⁢
t
,
where Δt is a difference between the time k and the time k−1.

### Claim 13 (depends on 11)

The apparatus according to claim 11, wherein, to process each of the image frames, the target object detection and tracking controller is further configured to determine a time to line crossing TTLC(k) for the target vehicle at the time k.

### Claim 14 (depends on 13)

The apparatus according to claim 13, wherein the time to line crossing TTLC(k) is determined from:
TTLC
⁡
(
k
)
=
L
n
(
k
)
R
⁡
(
k
)
.

### Claim 15

A method, comprising:
capturing a series of image frames for traffic lanes in front of an ego vehicle using at least one camera, wherein each of the series of image frames is captured at a different one of a plurality of times, and wherein the traffic lanes include an ego traffic lane occupied by the ego vehicle and an adjacent traffic lane for traffic traveling in a same direction as traffic within the ego traffic lane;
processing each of the image frames to detect a target vehicle in the adjacent traffic lane within the respective image frame at a time corresponding to capture of the respective image frame;
determining, using pixel measurements extracted from the image frames, a predicted time to line crossing for the detected target vehicle to cross into the ego traffic lane from the adjacent traffic lane; and
controlling the ego vehicle based on the predicted time to line crossing by the target vehicle into the ego traffic lane from the adjacent traffic lane.

### Claim 16 (depends on 15)

The method according to claim 15, further comprising:
capturing images of at least a portion of the ego traffic lane occupied by the ego vehicle and at least an adjoining portion of the adjacent traffic lane.

### Claim 17 (depends on 15)

The method according to claim 15, further comprising:
determining a width in pixels of an ego traffic lane and a lateral offset in pixels of the target vehicle.

### Claim 18 (depends on 17)

The method according to claim 17, further comprising:
determining a ratio Ln of the lateral offset of the target vehicle Lt the width of the ego traffic lane Le.

### Claim 19 (depends on 15)

The method according to claim 15, further comprising:
driving wheels of the ego vehicle with a motor;
supporting axles on which the wheels are mounted with a chassis, the wheels coupled to a steering mechanism configured to receive a steering angle control signal based on the predicted time to line crossing TTLC(k) for the target vehicle; and
actuating brakes for one or more of the wheels based on the predicted time to line crossing TTLC (k) for the target vehicle.

### Claim 20 (depends on 19)

The method according to claim 19, wherein the ego vehicle is an electric vehicle and the motor is an electric motor.