# US 2023/0399027 A1 · Method for classifying a behavior of a road user and method for controlling an ego vehicle

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0399027 A1
- Document ID: US20230399027A1
- Status: Published application
- Kind: A1
- Application number: 18/313,843
- Filed: 2023-05-08
- Published: 2023-12-14
- Earliest claimed: 2022-05-23
- Assignees: Robert Bosch GmbH
- Applicants: Robert Bosch GmbH
- Inventors: Andreas Schmidt; Koba Natroshvili; Maxim Dolgov; Steffen Knoop
- Web page: https://patentgrep.com/patent/US20230399027A1

## CPC

- B60W 60/0027: {using trajectory prediction for other traffic participants}
- B60W 60/0015: {specially adapted for safety}
- G01S 7/417: {involving the use of neural networks}
- G01S 13/931: of land vehicles
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/931: of land vehicles
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- B60W 2554/4041: Position
- B60W 2554/4046: Behavior, e.g. aggressive or erratic
- B60W 2554/80: Spatial relation or speed relative to objects
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method for classifying a driving behavior of a road user in the environment of an ego vehicle. The method includes: receiving trajectory data of a driving trajectory of a road user arranged in the environment of an ego vehicle; ascertaining a latent-space representation of the driving trajectory of the road user in a latent space; ascertaining a distance of the latent-space representation of the driving trajectory of the road user to a latent-space representation of at least one normal trajectory in the latent space and classifying a driving behavior of the road user as a normal driving behavior if the distance in the latent space falls below a predetermined limit value; or classifying the driving behavior of the road user as an abnormal driving behavior if the distance in the latent space exceeds the predetermined limit value.

## Claims

### Claim 1

A method for classifying a driving behavior of a road user in an environment of an ego vehicle, comprising the following steps:
receiving trajectory data of a driving trajectory of a road user arranged in the environment of an ego vehicle, wherein the trajectory data include at least position information of the driving trajectory followed by the road user;
ascertaining a latent-space representation of the driving trajectory of the road user in a latent space, wherein the latent-space representation of the driving trajectory is given by a reduced-dimension representation of the trajectory data;
ascertaining a distance of the latent-space representation of the driving trajectory of the road user to a latent-space representation of at least one normal trajectory in the latent space, wherein the normal trajectory is classified as a driving trajectory based on a normal driving behavior;
i) classifying a driving behavior of the road user as a normal driving behavior when the distance of the latent-space representation of the driving trajectory of the road user to the latent-space representation of the at least one normal trajectory in the latent space falls below a predetermined limit value, or ii) classifying the driving behavior of the road user as an abnormal driving behavior when the distance of the latent-space representation of the driving trajectory of the road user to the latent-space representation of the at least one normal trajectory in the latent space exceeds the predetermined limit value; and
outputting classification information.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the distance is determined by a predetermined distance metric, and wherein the distance metric is given by a Euclidean distance function or by a logarithmic likelihood function of the latent-space representations within the latent space.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the classification is carried out by a classification module, wherein the classification module includes an appropriately trained encoder module of an autoencoder architecture for ascertaining the latent-space representation of the driving trajectory of the road user, and wherein the autoencoder architecture is trained to ascertain latent-space representations of driving trajectories in the latent space and to recognize and classify them as driving trajectories based on normal driving behavior.

### Claim 4 (depends on 3)

The method according to claim 3, further comprising:
receiving map data of a map representation mapping the environment of the ego vehicle; and
ascertaining a latent-space representation of the map data of the map representation in the latent space, wherein the latent-space representation of the driving trajectory is given by a reduced-dimension representation of the map data;
wherein the ascertaining of the distance and/or the classifying of the driving behavior of the road user takes place taking into account the latent-space representation of the map data of the map representation.

### Claim 5 (depends on 4)

The method according to claim 4, further comprising:
converting the trajectory data into image data of the trajectory data and/or converting the map data into image data of the map data;
wherein the ascertaining of the distance and/or the classifying the driving behavior of the road user takes place based on the image data of the trajectory data and/or based on the image data of the map data.

### Claim 6 (depends on 3)

The method according to claim 3, wherein: i) the classification module further includes a connection network, wherein the connection network is configured to combine a plurality of image data of the trajectory data and/or a plurality of image data of the map data into input data for the encoder module in vector representation, and/or ii) the classification module further includes a map network, wherein the map network is configured to ascertain latent-space representations of the map data based on the image data of the map data; and
wherein the connection network and/or the map network are appropriately trained neural networks.

### Claim 7 (depends on 3)

The method according to claim 3, wherein the autoencoder architecture is a variational autoencoder (VAE), and wherein the latent-space representations of the driving trajectories are statistical distributions in the latent space and are each characterized by an expected value μ and a variance σ2.

### Claim 8 (depends on 7)

The method according to claim 7, wherein the distributions are Gaussian normal distributions in the latent space.

### Claim 9 (depends on 3)

The method according to claim 3, wherein training of the autoencoder architecture is carried out on trajectory data of a plurality of trips of a vehicle or a plurality of vehicles along a roadway traveled by the ego vehicle, and wherein map information of a map representation mapping the respectively traveled roadway and/or information regarding a roadway condition and/or information regarding prevailing traffic regulations and/or information regarding prevailing traffic volumes and/or information regarding prevailing weather conditions and/or time-of-day and/or day-of-the-week information, are also taken into account in the training.

### Claim 10 (depends on 1)

The method according to claim 1, wherein:
a normal driving behavior of the road user includes:
driving in a lane, designated according to prevailing traffic regulations, of a traveled roadway, and/or
driving according to prevailing speed regulations, and/or
following a path of the traveled roadway, and/or
driving in a direction of travel designated by prevailing traffic regulations for the traveled roadway and/or lane; and
an abnormal driving behavior includes:
not following the path of the traveled roadway; and/or
driving in a meandering direction of travel deviating from a straight direction of travel; and/or
driving in a direction of travel opposite a direction of travel designated according to the prevailing traffic regulation; and/or
driving at an excessive speed; and/or
driving in a lane not designated according to the prevailing traffic regulations.

### Claim 11

A method for controlling an ego vehicle, comprising the following steps:
carrying out a classification of a driving behavior of a road user in an environment of the ego vehicle based on trajectory data of a driving trajectory of the road user, the carrying out of the classification including:
receiving the trajectory data of the driving trajectory of the road user arranged in the environment of an ego vehicle, wherein the trajectory data include at least position information of the driving trajectory followed by the road user,
ascertaining a latent-space representation of the driving trajectory of the road user in a latent space, wherein the latent-space representation of the driving trajectory is given by a reduced-dimension representation of the trajectory data,
ascertaining a distance of the latent-space representation of the driving trajectory of the road user to a latent-space representation of at least one normal trajectory in the latent space, wherein the normal trajectory is classified as a driving trajectory based on a normal driving behavior,
i) classifying a driving behavior of the road user as a normal driving behavior when the distance of the latent-space representation of the driving trajectory of the road user to the latent-space representation of the at least one normal trajectory in the latent space falls below a predetermined limit value, or ii) classifying the driving behavior of the road user as an abnormal driving behavior when the distance of the latent-space representation of the driving trajectory of the road user to the latent-space representation of the at least one normal trajectory in the latent space exceeds the predetermined limit value, and
outputting classification information; and
controlling the ego vehicle, taking into account the classification of the driving behavior of the road user.

### Claim 12 (depends on 11)

The method according to claim 11, wherein the controlling of the ego vehicle includes:
carrying out a safety maneuver when the driving behavior of the road user is classified as an abnormal driving behavior.

### Claim 13 (depends on 12)

The method according to claim 12, wherein the safety maneuver includes:
reducing a speed of the ego vehicle; and/or
increasing a distance of the ego vehicle to the road user; and/or
changing a lane, traveled by the ego vehicle, of the traveled roadway.

### Claim 14

A computing unit configured to classify a driving behavior of a road user in an environment of an ego vehicle, the computing unit configured to:
receive trajectory data of a driving trajectory of a road user arranged in the environment of an ego vehicle, wherein the trajectory data include at least position information of the driving trajectory followed by the road user;
ascertain a latent-space representation of the driving trajectory of the road user in a latent space, wherein the latent-space representation of the driving trajectory is given by a reduced-dimension representation of the trajectory data;
ascertain a distance of the latent-space representation of the driving trajectory of the road user to a latent-space representation of at least one normal trajectory in the latent space, wherein the normal trajectory is classified as a driving trajectory based on a normal driving behavior;
i) classify a driving behavior of the road user as a normal driving behavior when the distance of the latent-space representation of the driving trajectory of the road user to the latent-space representation of the at least one normal trajectory in the latent space falls below a predetermined limit value, or ii) classify the driving behavior of the road user as an abnormal driving behavior when the distance of the latent-space representation of the driving trajectory of the road user to the latent-space representation of the at least one normal trajectory in the latent space exceeds the predetermined limit value; and
output classification information.

### Claim 15 (depends on 14)

The computing unit according to claim 14, wherein the computing unit is further configured to:
control the ego vehicle, taking into account the classification of the driving behavior of the road user.

### Claim 16

A non-transitory computer-readable storage medium on which is stored a computer program including instructions for classifying a driving behavior of a road user in an environment of an ego vehicle, the instructions, when executed by a computer, causing the computer to perform the following steps:
receiving trajectory data of a driving trajectory of a road user arranged in the environment of an ego vehicle, wherein the trajectory data include at least position information of the driving trajectory followed by the road user;
ascertaining a latent-space representation of the driving trajectory of the road user in a latent space, wherein the latent-space representation of the driving trajectory is given by a reduced-dimension representation of the trajectory data;
ascertaining a distance of the latent-space representation of the driving trajectory of the road user to a latent-space representation of at least one normal trajectory in the latent space, wherein the normal trajectory is classified as a driving trajectory based on a normal driving behavior;
i) classifying a driving behavior of the road user as a normal driving behavior when the distance of the latent-space representation of the driving trajectory of the road user to the latent-space representation of the at least one normal trajectory in the latent space falls below a predetermined limit value, or ii) classifying the driving behavior of the road user as an abnormal driving behavior when the distance of the latent-space representation of the driving trajectory of the road user to the latent-space representation of the at least one normal trajectory in the latent space exceeds the predetermined limit value; and
outputting classification information.
