# US 2022/0277647 A1 · Systems and methods for analyzing the in-lane driving behavior of a road agent external to a vehicle

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0277647 A1
- Document ID: US20220277647A1
- Status: Published application
- Kind: A1
- Application number: 17/186,401
- Filed: 2021-02-26
- Published: 2022-09-01
- Earliest claimed: 2021-02-26
- Assignees: Toyota Motor Engineering & Manufacturing North America, Inc.
- Applicants: Toyota Motor Engineering & Manufacturing North America, Inc.
- Inventors: Rui Guo; Kentaro Oguchi; Takamasa Higuchi; Xuewei Qi; Seyhan Ucar; Haritha Muralidharan
- Web page: https://patentgrep.com/patent/US20220277647A1

## CPC

- G08G 1/0125: {Traffic data processing}
- G08G 1/167: {Driving aids for lane monitoring, lane changing, e.g. blind spot detection}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 7/269: using gradient-based methods
- G06T 7/55: from multiple images
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06V 10/82: using neural networks
- G06V 20/54: of traffic, e.g. cars on the road, trains or boats
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- G06V 20/64: Three-dimensional [3D] objects
- G08G 1/0112: {from the vehicle, e.g. floating car data [FCD]}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30241: Trajectory
- G06T 2207/30256: Lane; Road marking
- G06T 2219/004: Annotating, labelling
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for analyzing the in-lane driving behavior of an external road agent are disclosed herein. One embodiment generates a sequence of sparse 3D point clouds based on a sequence of depth maps corresponding to a sequence of images of a scene; performs flow clustering based on the sequence of depth maps and a sequence of flow maps to identify points across the sequence of sparse 3D point clouds that belong to a detected road agent; generates a dense 3D point cloud by combining at least some of the points across the sequence of sparse 3D point clouds that belong to the detected road agent; detects one or more lane markings and projects them into the dense 3D point cloud to generate an annotated 3D point cloud; and analyzes the in-lane driving behavior of the detected road agent based, at least in part, on the annotated 3D point cloud.

## Claims

### Claim 1

A system for analyzing in-lane driving behavior of a road agent external to a vehicle, the system comprising:
one or more processors; and
a memory communicably coupled to the one or more processors and storing:
a three-dimensional (3D) rendering module including instructions that when executed by the one or more processors cause the one or more processors to:
generate a sequence of sparse 3D point clouds based on a sequence of depth maps corresponding to a sequence of images of a scene external to the vehicle;
perform flow clustering based on the sequence of depth maps and a sequence of flow maps corresponding to the sequence of images to identify points across the sequence of sparse 3D point clouds that belong to a detected road agent in the scene; and
generate a dense 3D point cloud that includes the detected road agent by combining at least some of the points across the sequence of sparse 3D point clouds that belong to the detected road agent;
a lane markings module including instructions that when executed by the one or more processors cause the one or more processors to detect one or more lane markings in the scene and project the one or more lane markings into the dense 3D point cloud to generate an annotated 3D point cloud; and
an analysis module including instructions that when executed by the one or more processors cause the one or more processors to analyze in-lane driving behavior of the detected road agent based, at least in part, on the annotated 3D point cloud.

### Claim 2 (depends on 1)

The system of claim 1, further comprising:
a monocular depth estimation module including instructions that when executed by the one or more processors cause the one or more processors to process the sequence of images using a first deep neural network that performs monocular depth estimation to generate the sequence of depth maps; and
an optical flow module including instructions that when executed by the one or more processors cause the one or more processors to process the sequence of images using a second deep neural network that generates the sequence of flow maps;
wherein the instructions in the lane markings module to detect one or more lane markings in the scene include instructions to process the sequence of images using a third deep neural network.

### Claim 3 (depends on 2)

The system of claim 2, wherein the depth maps in the sequence of depth maps represent respective distances of image pixels from a camera as grayscale intensities and the flow maps in the sequence of flow maps represent a speed of the detected road agent as pixel intensities and a direction of movement of the detected road agent as pixel colors.

### Claim 4 (depends on 1)

The system of claim 1, wherein the instructions in the 3D rendering module to perform flow clustering include instructions to pair the flow maps in the sequence of flow maps with corresponding depth maps in the sequence of depth maps to generate a clustering mask that identifies the points across the sequence of sparse 3D point clouds that belong to the detected road agent in the scene.

### Claim 5 (depends on 1)

The system of claim 1, wherein the instructions in the analysis module to analyze the in-lane driving behavior of the detected road agent include instructions to detect deviations of a trajectory of the detected road agent from a reference trajectory within a lane defined by the one or more lane markings projected into the annotated 3D point cloud.

### Claim 6 (depends on 5)

The system of claim 5, wherein the instructions in the analysis module to analyze the in-lane driving behavior of the detected road agent include instructions to determine that the detected deviations indicate anomalous driving behavior.

### Claim 7 (depends on 6)

The system of claim 6, wherein the anomalous driving behavior is substance-impaired driving.

### Claim 8 (depends on 1)

The system of claim 1, wherein the detected road agent is one of another vehicle, a motorcycle, and a bicycle and the vehicle is one of an autonomous vehicle, a semi-autonomous vehicle, a vehicle equipped with an Advanced Driver-Assistance System, and a vehicle manually driven by a human driver.

### Claim 9

A non-transitory computer-readable medium for analyzing in-lane driving behavior of a road agent external to a vehicle and storing instructions that when executed by one or more processors cause the one or more processors to:
generate a sequence of sparse three-dimensional (3D) point clouds based on a sequence of depth maps corresponding to a sequence of images of a scene external to the vehicle;
perform flow clustering based on the sequence of depth maps and a sequence of flow maps corresponding to the sequence of images to identify points across the sequence of sparse 3D point clouds that belong to a detected road agent in the scene;
generate a dense 3D point cloud that includes the detected road agent by combining at least some of the points across the sequence of sparse 3D point clouds that belong to the detected road agent;
detect one or more lane markings in the scene and project the one or more lane markings into the dense 3D point cloud to generate an annotated 3D point cloud; and
analyze in-lane driving behavior of the detected road agent based, at least in part, on the annotated 3D point cloud.

### Claim 10 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the instructions include further instructions that when executed by the one or more processors cause the one or more processors to:
process the sequence of images using a first deep neural network that performs monocular depth estimation to generate the sequence of depth maps; and
process the sequence of images using a second deep neural network that generates the sequence of flow maps;
wherein the instructions to detect one or more lane markings in the scene include instructions to process the sequence of images using a third deep neural network.

### Claim 11 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the instructions to perform flow clustering include instructions to pair the flow maps in the sequence of flow maps with corresponding depth maps in the sequence of depth maps to generate a clustering mask that identifies the points across the sequence of sparse 3D point clouds that belong to the detected road agent in the scene.

### Claim 12 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the instructions in the analysis module to analyze the in-lane driving behavior of the detected road agent include instructions to detect deviations of a trajectory of the detected road agent from a reference trajectory within a lane defined by the one or more lane markings projected into the annotated 3D point cloud.

### Claim 13

A method of analyzing in-lane driving behavior of a road agent external to a vehicle, the method comprising:
generating a sequence of sparse three-dimensional (3D) point clouds based on a sequence of depth maps corresponding to a sequence of images of a scene external to the vehicle;
performing flow clustering based on the sequence of depth maps and a sequence of flow maps corresponding to the sequence of images to identify points across the sequence of sparse 3D point clouds that belong to a detected road agent in the scene;
generating a dense 3D point cloud that includes the detected road agent by combining at least some of the points across the sequence of sparse 3D point clouds that belong to the detected road agent;
detecting one or more lane markings in the scene and projecting the one or more lane markings into the dense 3D point cloud to generate an annotated 3D point cloud; and
analyzing in-lane driving behavior of the detected road agent based, at least in part, on the annotated 3D point cloud.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
processing the sequence of images using a first deep neural network that performs monocular depth estimation to generate the sequence of depth maps;
processing the sequence of images using a second deep neural network that generates the sequence of flow maps; and
processing the sequence of images using a third deep neural network to detect the one or more lane markings.

### Claim 15 (depends on 14)

The method of claim 14, wherein the depth maps in the sequence of depth maps represent respective distances of image pixels from a camera as grayscale intensities and the flow maps in the sequence of flow maps represent a speed of the detected road agent as pixel intensities and a direction of movement of the detected road agent as pixel colors.

### Claim 16 (depends on 13)

The method of claim 13, wherein performing flow clustering includes pairing the flow maps in the sequence of flow maps with corresponding depth maps in the sequence of depth maps to generate a clustering mask that identifies the points across the sequence of sparse 3D point clouds that belong to the detected road agent in the scene.

### Claim 17 (depends on 13)

The method of claim 13, wherein the analyzing in-lane driving behavior of the detected road agent includes detecting deviations of a trajectory of the detected road agent from a reference trajectory within a lane defined by the one or more lane markings projected into the annotated 3D point cloud.

### Claim 18 (depends on 17)

The method of claim 17, wherein the analyzing in-lane driving behavior of the detected road agent includes determining that the detected deviations indicate anomalous driving behavior.

### Claim 19 (depends on 18)

The method of claim 18, wherein the anomalous driving behavior is substance-impaired driving.

### Claim 20 (depends on 13)

The method of claim 13, wherein the detected road agent is one of another vehicle, a motorcycle, and a bicycle and the vehicle is one of an autonomous vehicle, a semi-autonomous vehicle, a vehicle equipped with an Advanced Driver-Assistance System, and a vehicle manually driven by a human driver.
