# US 2025/0085109 A1 · Systems and methods for classifying a vehicle maneuver using a spatiotemporal attention selector

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0085109 A1
- Document ID: US20250085109A1
- Status: Published application
- Kind: A1
- Application number: 18/464,003
- Filed: 2023-09-08
- Published: 2025-03-13
- Earliest claimed: 2023-09-08
- Assignees: Verizon Patent and Licensing Inc.
- Applicants: Verizon Patent and Licensing Inc.
- Inventors: Matteo SIMONCINI; Tommaso BIANCONCINI; Luca BRAVI; Leonardo SARTI; Leonardo TACCARI; Douglas COIMBRA DE ANDRADE; Francesco SAMBO
- Web page: https://patentgrep.com/patent/US20250085109A1

## CPC

- G01C 21/1656: {with passive imaging devices, e.g. cameras}
- B60W 40/09: Driving style or behaviour
- G01C 21/3697: {Output of additional, non-guidance related information, e.g. low fuel level (G01C21/3679 takes precedence)}
- G01S 19/47: the supplementary measurement being an inertial measurement, e.g. tightly coupled inertial
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES

## Abstract

A device may receive video data and corresponding GPS data and IMU data associated with a vehicle, and may process the video data, with an object detector model, to identify objects and to generate a first feature vector. The device may process the GPS data and the IMU data, with a first CNN model, to generate a second feature vector, and may process the objects and the video data, with a tracking model, to identify positions and classes of the objects and to generate a third feature vector. The device may utilize a second CNN model to generate a matrix of object features based on the first, second, and third feature vectors, and may utilize a spatiotemporal attention selector model or a max pooled model with the matrix of object features to identify a classification of a maneuver of the vehicle. The device may perform actions based on the classification.

## Claims

### Claim 1

A method, comprising:
receiving, by a device, video data that includes a plurality of video frames and corresponding global positioning system (GPS) data and inertial measurement unit (IMU) data associated with a vehicle;
processing, by the device, the video data, with an object detector model, to identify objects in the video data and to generate a first feature vector based on the objects;
processing, by the device, the GPS data and the IMU data, with a first convolutional neural network (CNN) model, to generate a second feature vector;
processing, by the device, the objects and the video data, with a tracking model, to identify positions and classes of the objects and to generate a third feature vector based on the positions and the classes;
utilizing, by the device, a second CNN model to generate a matrix of object features based on the first feature vector, the second feature vector, and the third feature vector;
utilizing, by the device, a spatiotemporal attention selector model or a max pooled model with the matrix of object features to identify a classification of a maneuver of the vehicle as safe or unsafe; and
performing, by the device, one or more actions based on the classification.

### Claim 2 (depends on 1)

The method of claim 1, wherein processing the GPS data and the IMU data, with the first CNN model, to generate the second feature vector comprises:
processing the GPS data and the IMU data, with a first CNN model, to align sampling rates of the GPS data and the IMU data and timestamps of the video data, to extract GPS and IMU features, and to generate the second feature vector based on aligning the sampling rates and the timestamps and based on the GPS and IMU features.

### Claim 3 (depends on 1)

The method of claim 1, wherein the object detector model includes a faster region-based convolutional neural network model and a residual neural network model.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first CNN model includes two-dimensional depth-wise separable convolutions.

### Claim 5 (depends on 1)

The method of claim 1, wherein the tracking model includes a greedy tracking model that utilizes the positions, the classes, and object detection confidence.

### Claim 6 (depends on 1)

The method of claim 1, wherein each of the object features of the matrix is a concatenation of the first feature vector, the second feature vector, and the third feature vector.

### Claim 7 (depends on 1)

The method of claim 1, wherein utilizing the spatiotemporal attention selector model or the max pooled model with the matrix of object features to identify the classification of the maneuver of the vehicle as safe or unsafe comprises:
utilizing the spatiotemporal attention selector model or the max pooled model with the matrix of object features to select particular object features; and
determining the classification of the maneuver of the vehicle as safe or unsafe based on the particular object features.

### Claim 8

A device, comprising:
one or more processors configured to:
receive video data that includes a plurality of video frames and corresponding global positioning system (GPS) data and inertial measurement unit (IMU) data associated with a vehicle;
process the video data, with an object detector model, to identify objects in the video data and to generate a first feature vector based on the objects,
wherein the object detector model includes a faster region-based convolutional neural network model and a residual neural network model;
process the GPS data and the IMU data, with a first convolutional neural network (CNN) model, to generate a second feature vector;
process the objects and the video data, with a tracking model, to identify positions and classes of the objects and to generate a third feature vector based on the positions and the classes;
utilize a second CNN model to generate a matrix of object features based on the first feature vector, the second feature vector, and the third feature vector;
utilize a spatiotemporal attention selector model or a max pooled model with the matrix of object features to identify a classification of a maneuver of the vehicle as safe or unsafe; and
perform one or more actions based on the classification.

### Claim 9 (depends on 8)

The device of claim 8, wherein the one or more processors, to utilize the spatiotemporal attention selector model or the max pooled model with the matrix of object features to identify the classification of the maneuver of the vehicle as safe or unsafe, are configured to:
utilize the spatiotemporal attention selector model or the max pooled model with the matrix of object features to select a set of object features;
perform fine-tuning of the set of object features to identify particular object features; and
determine the classification of the maneuver of the vehicle as safe or unsafe based on the particular object features.

### Claim 10 (depends on 8)

The device of claim 8, wherein the one or more processors, to perform the one or more actions, are configured to:
schedule a driver of the vehicle for training based on the classification of the maneuver of the vehicle being unsafe.

### Claim 11 (depends on 8)

The device of claim 8, wherein the one or more processors, to perform the one or more actions, are configured to:
determine the classification of the maneuver of the vehicle as unsafe due to one of an improper lane change, an improper turn, a collision, or a loss of vehicle control.

### Claim 12 (depends on 8)

The device of claim 8, wherein the one or more processors, to perform the one or more actions, are configured to:
generate an alert for a driver of the vehicle based on the classification of the maneuver of the vehicle being unsafe.

### Claim 13 (depends on 8)

The device of claim 8, wherein the one or more processors, to perform the one or more actions, are configured to:
generate an alert for a fleet manager of the vehicle based on the classification of the maneuver of the vehicle being unsafe.

### Claim 14 (depends on 8)

The device of claim 8, wherein the one or more processors, to perform the one or more actions, are configured to:
retrain one or more of the object detector model, the first CNN model, the tracking model, the second CNN model, the spatiotemporal attention selector model, or the max pooled model based on the classification of the maneuver of the vehicle.

### Claim 15

A non-transitory computer-readable medium storing a set of instructions, the set of instructions comprising:
one or more instructions that, when executed by one or more processors of a device, cause the device to:
receive video data that includes a plurality of video frames and corresponding global positioning system (GPS) data and inertial measurement unit (IMU) data associated with a vehicle;
process the video data, with an object detector model, to identify objects in the video data and to generate a first feature vector based on the objects;
process the GPS data and the IMU data, with a first convolutional neural network (CNN) model, to generate a second feature vector,
wherein the first CNN model includes two-dimensional depth-wise separable convolutions;
process the objects and the video data, with a tracking model, to identify positions and classes of the objects and to generate a third feature vector based on the positions and the classes;
utilize a second CNN model to generate a matrix of object features based on the first feature vector, the second feature vector, and the third feature vector;
utilize a spatiotemporal attention selector model or a max pooled model with the matrix of object features to identify a classification of a maneuver of the vehicle as safe or unsafe; and
perform one or more actions based on the classification.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the one or more instructions, that cause the device to process the GPS data and the IMU data, with the first CNN model, to generate the second feature vector, cause the device to:
process the GPS data and the IMU data, with a first CNN model, to align sampling rates of the GPS data and the IMU data and timestamps of the video data, to extract GPS and IMU features, and to generate the second feature vector based on aligning the sampling rates and the timestamps and based on the GPS and IMU features.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the one or more instructions, that cause the device to utilize the spatiotemporal attention selector model or the max pooled model with the matrix of object features to identify the classification of the maneuver of the vehicle as safe or unsafe, cause the device to:
utilize the spatiotemporal attention selector model or the max pooled model with the matrix of object features to select particular object features; and
determine the classification of the maneuver of the vehicle as safe or unsafe based on the particular object features.

### Claim 18 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the one or more instructions, that cause the device to utilize the spatiotemporal attention selector model or the max pooled model with the matrix of object features to identify the classification of the maneuver of the vehicle as safe or unsafe, cause the device to:
utilize the spatiotemporal attention selector model or the max pooled model with the matrix of object features to select a set of object features;
perform fine-tuning of the set of object features to identify particular object features; and
determine the classification of the maneuver of the vehicle as safe or unsafe based on the particular object features.

### Claim 19 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the one or more instructions, that cause the device to perform the one or more actions, cause the device to one or more of:
schedule a driver of the vehicle for training based on the classification of the maneuver of the vehicle being unsafe; or
determine the classification of the maneuver of the vehicle as unsafe due to one of an improper lane change, an improper turn, a collision, or a loss of vehicle control.

### Claim 20 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the one or more instructions, that cause the device to perform the one or more actions, cause the device to one or more of:
generate an alert for a driver of the vehicle based on the classification of the maneuver of the vehicle being unsafe;
generate an alert for a fleet manager of the vehicle based on the classification of the maneuver of the vehicle being unsafe; or
retrain one or more of the object detector model, the first CNN model, the tracking model, the second CNN model, the spatiotemporal attention selector model, or the max pooled model based on the classification of the maneuver of the vehicle.
