# US 2021/0334645 A1 · Notifications determined using one or more neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334645 A1
- Document ID: US20210334645A1
- Status: Published application
- Kind: A1
- Application number: 16/860,175
- Filed: 2020-04-28
- Published: 2021-10-28
- Earliest claimed: 2020-04-28
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: Siddhant Pardeshi; Pranit P. Kothari; Vinayak Vilas Gaikwad
- Web page: https://patentgrep.com/patent/US20210334645A1

## CPC

- G06V 20/20: in augmented reality scenes
- B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
- B60K 35/213: {Virtual instruments}
- B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G10L 15/16: using artificial neural networks
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 25/51: for comparison or discrimination
- B60K 35/22: Display screens
- B60K 35/80: Arrangements for controlling instruments
- B60K 2360/148: Instrument input by voice
- B60K 2360/175: Autonomous driving
- B60K 2360/176: Camera images
- B60K 2360/178: Warnings
- B60K 2360/179: Distances to obstacles or vehicles
- B60K 2360/21: using cameras
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06V 20/44: {Event detection}
- G08B 1/08: using electric transmission {; transformation of alarm signals to electrical signals from a different medium, e.g. transmission of an electric alarm signal upon detection of an audible alarm signal}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS

## Abstract

Apparatuses, systems, and techniques are presented to determine actions to be taken for data anomalies. In at least one embodiment, audio and video data captured for an environment of a user can be analyzed to detect one or more data anomalies and determine whether to notify this user depending on whether the anomalies are applicable to this user.

## Claims

### Claim 1

A processor, comprising:
one or more circuits to use one or more neural networks to detect one or more data anomalies and cause one or more users to be notified of the one or more anomalies, wherein the one or more users to be notified depends on whether the one or more data anomalies are applicable to the one or more users.

### Claim 2 (depends on 1)

The processor of claim 1, wherein the one or more neural networks include an audio anomaly detector and a video anomaly detector for providing instance and confidence data for the one or more data anomalies, the audio anomaly detector taking as input audio data captured for an environment of the one or more users and the video anomaly detector taking as input video data captured for the environment of the one or more users.

### Claim 3 (depends on 2)

The processor of claim 2, wherein the one or more neural networks include an event detector for determining a classification for each of the one or more data anomalies.

### Claim 4 (depends on 3)

The processor of claim 3, wherein the one or more neural networks include a decision maker network for determining whether to cause the one or more users to be notified, the decision maker network using the instance and confidence data, along with the classification for each of the one or more data anomalies, to determine whether the one or more data anomalies are applicable to the one or more users.

### Claim 5 (depends on 4)

The processor of claim 4, wherein the one or more circuits are further to use a speech recognition module to provide the decision maker network with text for detected speech related to the one or more data anomalies.

### Claim 6 (depends on 4)

The processor of claim 4, wherein the decision maker network is further to determine whether to take an action to reduce an immersiveness of an experience for the one or more users.

### Claim 7

A system comprising:
one or more processors to use one or more neural networks to detect one or more data anomalies in the video data and audio data, and to cause one or more users to be notified of the one or more anomalies, wherein the one or more users to be notified depends on whether the one or more data anomalies are applicable to the one or more users.

### Claim 8 (depends on 7)

The system of claim 7, wherein the one or more neural networks include an audio anomaly detector and a video anomaly detector for providing instance and confidence data for the one or more data anomalies, the audio anomaly detector taking as input audio data captured for an environment of the one or more users and the video anomaly detector taking as input video data captured for the environment of the one or more users.

### Claim 9 (depends on 8)

The system of claim 8, wherein the one or more neural networks include an event detector for determining a classification for each of the one or more data anomalies.

### Claim 10 (depends on 9)

The system of claim 9, wherein the one or more neural networks include a decision maker network for determining whether to cause the one or more users to be notified, the decision maker network using the instance and confidence data, along with the classification for each of the one or more data anomalies, to determine whether the one or more data anomalies are applicable to the one or more users.

### Claim 11 (depends on 10)

The system of claim 10, wherein the one or more processors are further to use a speech recognition module to provide the decision maker network with text for detected speech related to the one or more data anomalies.

### Claim 12 (depends on 10)

The system of claim 10, wherein the decision maker network is further to determine whether to take an action to reduce an immersiveness of an experience for the one or more users.

### Claim 13

A method comprising:
using one or more neural networks to detect one or more data anomalies in the video data and audio data, and to cause one or more users to be notified of the one or more anomalies, wherein the one or more users to be notified depends on whether the one or more data anomalies are applicable to the one or more users.

### Claim 14 (depends on 13)

The method of claim 13, wherein the one or more neural networks include an audio anomaly detector and a video anomaly detector for providing instance and confidence data for the one or more data anomalies, the audio anomaly detector taking as input audio data captured for an environment of the one or more users and the video anomaly detector taking as input video data captured for the environment of the one or more users.

### Claim 15 (depends on 14)

The method of claim 14, wherein the one or more neural networks include an event detector for determining a classification for each of the one or more data anomalies.

### Claim 16 (depends on 15)

The method of claim 15, wherein the one or more neural networks include a decision maker network for determining whether to cause the one or more users to be notified, the decision maker network using the instance and confidence data, along with the classification for each of the one or more data anomalies, to determine whether the one or more data anomalies are applicable to the one or more users.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
using a speech recognition module to provide the decision maker network with text for detected speech related to the one or more data anomalies.

### Claim 18 (depends on 16)

The method of claim 16, wherein the decision maker network is further to determine whether to take an action to reduce an immersiveness of an experience for the one or more users.

### Claim 19

A machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:
use one or more neural networks to detect one or more data anomalies in the video data and audio data, and to cause one or more users to be notified of the one or more anomalies, wherein the one or more users to be notified depends on whether the one or more data anomalies are applicable to the one or more users.

### Claim 20 (depends on 19)

The machine-readable medium of claim 19, wherein the one or more neural networks include an audio anomaly detector and a video anomaly detector for providing instance and confidence data for the one or more data anomalies, the audio anomaly detector taking as input audio data captured for an environment of the one or more users and the video anomaly detector taking as input video data captured for the environment of the one or more users.

### Claim 21 (depends on 20)

The machine-readable medium of claim 20, wherein the one or more neural networks include an event detector for determining a classification for each of the one or more data anomalies.

### Claim 22 (depends on 21)

The machine-readable medium of claim 21, wherein the one or more neural networks include a decision maker network for determining whether to cause the one or more users to be notified, the decision maker network using the instance and confidence data, along with the classification for each of the one or more data anomalies, to determine whether the one or more data anomalies are applicable to the one or more users.

### Claim 23 (depends on 22)

The machine-readable medium of claim 22, wherein the one or more processors are further to use a speech recognition module to provide the decision maker network with text for detected speech related to the one or more data anomalies.

### Claim 24 (depends on 22)

The machine-readable medium of claim 22, wherein the decision maker network is further to determine whether to take an action to reduce an immersiveness of an experience for the one or more users.

### Claim 25

A user notification system, comprising:
a camera to capture video data;
a microphone to capture audio data;
one or more processors to use one or more neural networks to detect one or more data anomalies in the video data and audio data, and to cause one or more users to be notified of the one or more anomalies, wherein the one or more users to be notified depends on whether the one or more data anomalies are applicable to the one or more users; and
memory for storing network parameters for the one or more neural networks.

### Claim 26 (depends on 25)

The distance determination system of claim 25, wherein the one or more neural networks include an audio anomaly detector and a video anomaly detector for providing instance and confidence data for the one or more data anomalies.

### Claim 27 (depends on 26)

The distance determination system of claim 26, wherein the one or more neural networks include an event detector for determining a classification for each of the one or more data anomalies.

### Claim 28 (depends on 27)

The distance determination system of claim 27, wherein the one or more neural networks include a decision maker network for determining whether to cause the one or more users to be notified, the decision maker network using the instance and confidence data, along with the classification for each of the one or more data anomalies, to determine whether the one or more data anomalies are applicable to the one or more users.

### Claim 29 (depends on 28)

The distance determination system of claim 28, wherein the one or more processors are further to use a speech recognition module to provide the decision maker network with text for detected speech related to the one or more data anomalies.

### Claim 30 (depends on 28)

The distance determination system of claim 28, wherein the decision maker network is further to determine whether to take an action to reduce an immersiveness of an experience for the one or more users.
