# US 2023/0401851 A1 · Temporal event detection with evidential neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0401851 A1
- Document ID: US20230401851A1
- Status: Published application
- Kind: A1
- Application number: 18/332,057
- Filed: 2023-06-09
- Published: 2023-12-14
- Earliest claimed: 2022-06-10
- Assignees: NEC Laboratories America, Inc.
- Applicants: NEC Laboratories America, Inc.
- Inventors: Xuchao Zhang; Xujiang Zhao; Yuncong Chen; Wenchao Yu; Haifeng Chen; Wei Cheng
- Web page: https://patentgrep.com/patent/US20230401851A1

## CPC

- G06V 20/44: {Event detection}
- G06V 10/82: using neural networks
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06T 2207/20081: Training; Learning
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Methods and systems for event detection include training a joint neural network model with respective neural networks for audio data and video data relating to a same scene. The joint neural network model is configured to output a belief value, a disbelief value, and an uncertainty value. It is determined that an event has occurred based on the belief value, the disbelief value, and the uncertainty value.

## Claims

### Claim 1

A computer-implemented method for event detection, comprising:
training a joint neural network model with respective neural networks for audio data and video data relating to a same scene, wherein the joint neural network model is configured to output a belief value, a disbelief value, and an uncertainty value; and
determining that an event has occurred based on the belief value, the disbelief value, and the uncertainty value.

### Claim 2 (depends on 1)

The method of claim 1, wherein the joint neural network model includes a first network architecture that includes a transformer for video data and a second network architecture that includes a convolutional neural network (CNN) and a recurrent neural network (RNN) for audio data.

### Claim 3 (depends on 1)

The method of claim 1, further comprising collecting operational data of a system that includes audio and video data streams from the same scene.

### Claim 4 (depends on 3)

The method of claim 3, wherein detecting that the event has occurred includes inputting the audio and video data streams to the respective neural networks of the joint neural network model.

### Claim 5 (depends on 4)

The method of claim 4, wherein the audio and video data streams are collected from a same location at a same time.

### Claim 6 (depends on 1)

The method of claim 1, wherein training the joint neural network model includes minimizing a loss function that includes respective cross-entropy losses for the different data types.

### Claim 7 (depends on 1)

The method of claim 1, wherein determining that the event includes determining that the belief value is greater than the disbelief value and that the uncertainty value is below a threshold value.

### Claim 8 (depends on 1)

The method of claim 1, wherein the belief value, the disbelief value, and the uncertainty value sum to 1.

### Claim 9 (depends on 1)

The method of claim 1, further comprising collecting new audio and video data from a new scene, wherein determining that the event has occurred includes inputting the new audio and video data to the trained joint neural network model.

### Claim 10 (depends on 1)

The method of claim 1, further comprising performing an action responsive to the event, selected from the group consisting of changing a security setting for an application or hardware component, changing an operational parameter of an application or hardware component, halting an application, restarting an application, halting a hardware component, rebooting a hardware component, changing an environmental condition, and changing a network interface's status or settings.

### Claim 11

A system for event detection, comprising:
a hardware processor; and
a memory that stores a computer program which, when executed by the hardware processor, causes the hardware processor to:
train a joint neural network model with respective neural networks for audio data and video data relating to a same scene, wherein the joint neural network model is configured to output a belief value, a disbelief value, and an uncertainty value; and
determine that an event has occurred based on the belief value, the disbelief value, and the uncertainty value.

### Claim 12 (depends on 11)

The system of claim 11, wherein the joint neural network model includes a first network architecture that includes a transformer for video data and a second network architecture that includes a convolutional neural network (CNN) and a recurrent neural network (RNN) for audio data.

### Claim 13 (depends on 11)

The system of claim 11, wherein the computer program further causes the hardware processor to collect operational data of a system that includes audio and video data streams from a same scene.

### Claim 14 (depends on 13)

The system of claim 13, wherein the computer program further causes the hardware processor to input the audio and video data streams to the respective neural networks of the joint neural network model.

### Claim 15 (depends on 14)

The system of claim 14, wherein the audio and video data streams are collected from the same location at a same time.

### Claim 16 (depends on 11)

The system of claim 11, wherein the computer program further causes the hardware processor to minimize a loss function that includes respective cross-entropy losses for the different data types.

### Claim 17 (depends on 11)

The system of claim 11, wherein the computer program further causes the hardware processor to determine that the belief value is greater than the disbelief value and that the uncertainty value is below a threshold value.

### Claim 18 (depends on 11)

The system of claim 11, wherein the belief value, the disbelief value, and the uncertainty value sum to 1.

### Claim 19 (depends on 11)

The system of claim 11, wherein the computer program further causes the hardware processor to collect new audio and video data from a new scene, wherein the determination that the event has occurred includes processing the new audio and video data with the trained joint neural network model.

### Claim 20 (depends on 11)

The system of claim 11, wherein the computer program further causes the hardware processor to perform an action responsive to the event, selected from the group consisting of changing a security setting for an application or hardware component, changing an operational parameter of an application or hardware component, halting an application, restarting an application, halting a hardware component, rebooting a hardware component, changing an environmental condition, and changing a network interface's status or settings.
