# US 12,223,701 B2 · Systems and methods for training event prediction models for camera-based warning systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,701 B2
- Document ID: US12223701B2
- Status: Granted
- Kind: B2
- Application number: 17/651,590
- Filed: 2022-02-18
- Published: 2023-08-24
- Granted: 2025-02-11
- Earliest claimed: 2022-02-18
- Assignees: Verizon Patent and Licensing Inc.
- Applicants: Verizon Patent and Licensing Inc.
- Inventors: Douglas Coimbra De Andrade; Aurel Pjetri
- Web page: https://patentgrep.com/patent/US12223701B2

## CPC

- G06V 10/82: using neural networks
- G06N 3/045: Combinations of networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06V 20/44: {Event detection}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

In some implementations, a device may receive image data and sensor data associated with the image data. The image data may depict a series of scenes that are captured over a time period and that are associated with an occurrence of an event. The device may determine a time of interest, within the time period, at which a value of the sensor data satisfies one or more criteria. The device may extract a portion of the image data based on the time of interest. The device may extract, from the portion of the image data, a slice of image data depicting a first set of scenes and a slice of image data depicting a second set of scenes. The device may train, based on the slices of image data, a model to predict a probability of an occurrence of an event.

## Claims

### Claim 1

A method, comprising:
receiving, by a device, image data and sensor data associated with the image data, wherein the image data depicts a series of scenes that are captured over a time period and that are associated with an occurrence of an event;
determining, by the device, a time of interest, within the time period, at which a value of the sensor data satisfies one or more criteria;
extracting, by the device, a portion of the image data based on the time of interest;
extracting, by the device and from the portion of the image data, a first slice of image data depicting a first set of scenes, of the series of scenes, and a second slice of image data depicting a second set of scenes, of the series of scenes, wherein a start of the first slice of image data is after a beginning of the portion of interest, wherein an end of the second slice of image data is before an end of the portion of interest, wherein the time of interest is at least partially between an end of the first slice of image data and a beginning of the second slice of image data; and
training, by the device and based on the first slice of image data and the second slice of image data, a model to predict a probability of an occurrence of an event and to determine whether the probability satisfies a probability threshold.

### Claim 2 (depends on 1)

The method of claim 1, wherein the image data includes a plurality of frames, and wherein extracting the portion of the image data comprises:
determining a frame, of the plurality of frames, associated with the time of interest; and
extracting a series of frames, of the plurality of frames, based on the frame, wherein the series of frames corresponds to the portion of the image data, and wherein the frame is a last frame of the series of frames.

### Claim 3 (depends on 1)

The method of claim 1, wherein the model includes a neural network, and wherein training the model comprises:
providing the first slice of image data to a first neural network, of the neural network; and
providing the second slice of image data to a second neural network, of the neural network.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
determining a sum of squares based on a portion of the sensor data associated with the time of interest;
determining that the value of the sensor data satisfies the one or more criteria based on the sum of squares; and
determining the time of interest based on the value of the sensor data satisfying the one or more criteria.

### Claim 5 (depends on 1)

The method of claim 1, wherein the first set of scenes occur during the time period prior to the second set of scenes, and wherein training the model comprises:
determining a first score associated with the first slice of image data, wherein the first score indicates a first probability of the occurrence of the event;
determining a second score associated with the second slice of image data, wherein the second score indicates a second probability of the occurrence of the event; and
determining a loss associated with the model based on whether the first score is greater than the second score.

### Claim 6 (depends on 5)

The method of claim 5, wherein determining the loss comprises one or more of:
determining a monotonic loss;
determining a start loss associated with a minimum score being assigned to a beginning of the first slice of image data and a beginning of the second slice of image data;
determining an end loss associated with a maximum score being assigned to an end of the first slice of image data and an end of the second slice of image data; or
determining a near loss associated with a similarity between the first score and the second score based on a distance between the end of the first slice of image data and the beginning of the second slice of image data.

### Claim 7 (depends on 5)

The method of claim 5, wherein determining the loss comprises:
determining the loss based on a sum of a monotonic loss, a start loss associated with a minimum score being assigned to a beginning of the first slice of image data and a beginning of the second slice of image data, an end loss associated with a maximum score being assigned to an end of the first slice of image data and an end of the second slice of image data, and a near loss associated with a similarity between the first score and the second score based on a distance between the end of the first slice of image data and the beginning of the second slice of image data.

### Claim 8

A device, comprising:
one or more processors configured to:
receive image data and sensor data associated with the image data, wherein the image data depicts a series of scenes that are captured over a time period and that are associated with an occurrence of an event;
determine a time of interest, within the time period, at which a value of the sensor data satisfies one or more criteria;
extract a portion of the image data based on the time of interest;
extract, from the portion of the image data, a first slice of image data depicting a first set of scenes, of the series of scenes, and a second slice of image data depicting a second set of scenes, of the series of scenes, wherein a start of the first slice of image data is after a beginning of the portion of interest, wherein an end of the second slice of image data is before an end of the portion of interest, wherein the time of interest is at least partially between an end of the first slice of image data and a beginning of the second slice of image data; and
train, based on the first slice of image data and the second slice of image data, a model to predict a probability of an occurrence of an event and to determine whether the probability satisfies a probability threshold.

### Claim 9 (depends on 8)

The device of claim 8, wherein the image data includes a plurality of frames, and wherein the one or more processors, to extract the portion of the image data, are configured to:
determine a frame, of the plurality of frames, associated with the time of interest; and
extract a series of frames, of the plurality of frames, based on the frame, wherein the series of frames corresponds to the portion of the image data, and wherein the frame is a last frame of the series of frames.

### Claim 10 (depends on 8)

The device of claim 8, wherein the model includes a neural network, and wherein the one or more processors, to train the model, are configured to:
provide the first slice of image data to a first neural network, of the neural network; and
provide the second slice of image data to a second neural network, of the neural network.

### Claim 11 (depends on 8)

The device of claim 8, wherein the one or more processors are further configured to:
determine a sum of squares based on a portion of the sensor data associated with the time of interest;
determine that the value of the sensor data satisfies the one or more criteria based on the sum of squares; and
determine the time of interest based on the value of the sensor data satisfying the one or more criteria.

### Claim 12 (depends on 8)

The device of claim 8, wherein the first set of scenes occur during the time period prior to the second set of scenes, and wherein the one or more processors, to train the model, are configured to:
determine a first score associated with the first slice of image data, wherein the first score indicates a first probability of the occurrence of the event;
determine a second score associated with the second slice of image data, wherein the second score indicates a second probability of the occurrence of the event; and
determine a loss associated with the model based on whether the first score is greater than the second score.

### Claim 13 (depends on 12)

The device of claim 12, wherein the one or more processors, to determine the loss, are configured to one or more of:
determine a monotonic loss;
determine a start loss associated with a minimum score being assigned to a beginning of the first slice of image data and a beginning of the second slice of image data;
determine an end loss associated with a maximum score being assigned to an end of the first slice of image data and an end of the second slice of image data; or
determine a near loss associated with a similarity between the first score and the second score based on a distance between the end of the first slice of image data and the beginning of the second slice of image data.

### Claim 14 (depends on 12)

The device of claim 12, wherein the one or more processors, to determine the loss, are configured to:
determine the loss based on a sum of a monotonic loss, a start loss associated with a minimum score being assigned to a beginning of the first slice of image data and a beginning of the second slice of image data, an end loss associated with a maximum score being assigned to an end of the first slice of image data and an end of the second slice of image data, and a near loss associated with a similarity between the first score and the second score based on a distance between the end of the first slice of image data and the beginning of the second slice of image data.

### Claim 15

A non-transitory computer-readable medium storing a set of instructions, the set of instructions comprising:
one or more instructions that, when executed by one or more processors of a device, cause the device to:
receive image data and sensor data associated with the image data, wherein the image data depicts a series of scenes that are captured over a time period and that are associated with an occurrence of an event;
determine a time of interest, within the time period, at which a value of the sensor data satisfies one or more criteria;
extract a portion of the image data based on the time of interest;
extract, from the portion of the image data, a first slice of image data depicting a first set of scenes, of the series of scenes, and a second slice of image data depicting a second set of scenes, of the series of scenes, wherein a start of the first slice of image data is after a beginning of the portion of interest, wherein an end of the second slice of image data is before an end of the portion of interest, wherein the time of interest is at least partially between an end of the first slice of image data and a beginning of the second slice of image data; and
train, based on the first slice of image data and the second slice of image data, a model to predict a probability of an occurrence of an event and to determine whether the probability satisfies a probability threshold.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the image data includes a plurality of frames, and wherein the one or more instructions, that when executed by the one or more processors, cause the device to extract the portion of the image data, cause the device to:
determine a frame, of the plurality of frames, associated with the time of interest; and
extract a series of frames, of the plurality of frames, based on the frame, wherein the series of frames corresponds to the portion of the image data, and wherein the frame is a last frame of the series of frames.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the model includes a neural network, and wherein the one or more instructions, that when executed by the one or more processors, cause the device to train the model, cause the device to:
provide the first slice of image data to a first neural network, of the neural network; and
provide the second slice of image data to a second neural network, of the neural network.

### Claim 18 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the one or more instructions further cause the device to:
determine a sum of squares based on a portion of the sensor data associated with the time of interest;
determine that the value of the sensor data satisfies the one or more criteria based on the sum of squares; and
determine the time of interest based on the value of the sensor data satisfying the one or more criteria.

### Claim 19 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the first set of scenes occur during the time period prior to the second set of scenes, and wherein the one or more instructions, that when executed by the one or more processors, cause the device to train the model, cause the device to:
determine a first score associated with the first slice of image data, wherein the first score indicates a first probability of the occurrence of the event;
determine a second score associated with the second slice of image data, wherein the second score indicates a second probability of the occurrence of the event; and
determine a loss associated with the model based on whether the first score is greater than the second score.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the one or more instructions, that cause the device to determine the loss, cause the device to:
determine a monotonic loss;
determine a start loss associated with a minimum score being assigned to a beginning of the first slice of image data and a beginning of the second slice of image data;
determine an end loss associated with a maximum score being assigned to an end of the first slice of image data and an end of the second slice of image data;
determine a near loss associated with a similarity between the first score and the second score based on a distance between the end of the first slice of image data and the beginning of the second slice of image data; and
determine the loss based on a sum of the monotonic loss, the start loss, the end loss, and the near loss.