# US 2020/0394499 A1 · Identifying complex events from hierarchical representation of data set features

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0394499 A1
- Document ID: US20200394499A1
- Status: Published application
- Kind: A1
- Application number: 16/439,508
- Filed: 2019-06-12
- Published: 2020-12-17
- Earliest claimed: 2019-06-12
- Assignees: SRI International
- Applicants: SRI International
- Inventors: Yi Yao; Ajay Divakaran; Pallabi Ghosh
- Web page: https://patentgrep.com/patent/US20200394499A1

## CPC

- G06N 3/045: Combinations of networks
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Techniques are disclosed for identifying multimodal subevents within an event having spatially-related and temporally-related features. In one example, a system receives a Spatio-Temporal Graph (STG) comprising (1) a plurality of nodes, each node having a feature descriptor that describes a feature present in the event, (2) a plurality of spatial edges, each spatial edge describing a spatial relationship between two of the plurality of nodes, and (3) a plurality of temporal edges, each temporal edge describing a temporal relationship between two of the plurality of nodes. Furthermore, the STG comprises at least one of: (1) variable-length descriptors for the feature descriptors or (2) temporal edges that span multiple time steps for the event. A machine learning system processes the STG to identify the multimodal subevents for the event. In some examples, the machine learning system comprises stacked Spatio-Temporal Graph Convolutional Networks (STGCNs), each comprising a plurality of STGCN layers.

## Claims

### Claim 1

A system for identifying one or more multimodal subevents within an event having spatially-related and temporally-related features, the system comprising:
an input unit configured to receive a Spatio-Temporal Graph (STG) comprising (1) a plurality of nodes, each node of the plurality of nodes having a feature descriptor that describes a feature present in the event, (2) a plurality of spatial edges, each spatial edge of the plurality of spatial edges describing a spatial relationship between two of the plurality of nodes, and (3) a plurality of temporal edges, each temporal edge of the plurality of temporal edges describing a temporal relationship between two of the plurality of nodes,
wherein the STG comprises at least one of: (1) variable-length descriptors for the feature descriptors or (2) temporal edges that span multiple time steps for the event;
a computation engine comprising processing circuitry for executing a machine learning system comprising one or more stacked Spatio-Temporal Graph Convolutional Networks (STGCNs) each comprising a plurality of STGCN layers, the one or more stacked STGCNs configured to process the STG to identify the one or more multimodal subevents for the event; and
an output device configured to output an indication of the one or more multimodal subevents.

### Claim 2 (depends on 1)

The system of claim 1,
wherein the STG comprises the variable-length feature descriptors, and
wherein the one or more stacked STGCNs are further configured to process the STG to convert the variable-length feature descriptors to fixed-length feature descriptors.

### Claim 3 (depends on 2)

The system of claim 2,
wherein the plurality of nodes of the STG comprise a plurality of clusters of nodes, each cluster of the plurality of clusters having feature descriptors that have a length that is different from a length of feature descriptors of each other cluster of the one or more clusters,
wherein, to process the STG to convert the variable-length feature descriptors to fixed-length feature descriptors, the one or more stacked STGCNs further comprise, for each cluster of the plurality of clusters, a spatial graph convolutional network (GCN) configured to process the cluster to convert the variable-length feature descriptors of the nodes of the cluster to the fixed-length feature descriptors.

### Claim 4 (depends on 2)

The system of claim 2,
wherein, to process the STG to convert the variable-length feature descriptors to fixed-length feature descriptors, the one or more stacked STGCNs comprise one or more convolutional layers configured to convert the variable-length feature descriptors of the plurality of nodes to the fixed-length feature descriptors.

### Claim 5 (depends on 1)

The system of claim 1,
wherein the STG comprises an adjacency matrix that represents irregular connections between the plurality of nodes, and
wherein to process the STG to identify the one or more multimodal subevents for the event, the one or more STGCNs are configured to perform a plurality of down-sampling operations and a plurality of up-sampling operations on the adjacency matrix to identify the one or more multimodal subevents for the event.

### Claim 6 (depends on 5)

The system of claim 5, wherein each stacked STGCN of the plurality of stacked STGCNs is a graph convolutional neural network comprising a first plurality of graph convolutional layers and a second plurality of graph convolutional layers,
wherein the first plurality of layers comprises one or more STGCN layers and one or more convolutional layers, the first plurality of layers configured to perform the plurality of down-sampling operations, and
wherein the second plurality of layers comprises one or more deconvolutional layers, the second plurality of layers configured to perform the plurality of up-sampling operations.

### Claim 7 (depends on 6)

The system of claim 6, wherein the one or more STGCN layers comprise one or more spatial graph convolution layers, one or more temporal graph convolution layers, and one or more non-linear convolution layers.

### Claim 8 (depends on 1)

The system of claim 1, wherein the plurality of stacked STGCNs are trained with a plurality of STGs, each STG of the plurality of STGs labeled with one or more multimodal subevents present within the plurality of spatial edges and the plurality of temporal edges of the STG.

### Claim 9 (depends on 1)

The system of claim 1, each node of the plurality of nodes of the STG is connected to one or more of the plurality of spatial edges and the plurality of temporal edges to enable one or more spatial relationships and temporal relationships between the plurality of nodes.

### Claim 10 (depends on 1)

The system of claim 1,
wherein the one or more multimodal subevents comprise one or more recognized human or animal activities,
wherein the feature descriptor of each node of the plurality of nodes describes a human event feature, the human event feature including one of at least an actor, an object, a scene, or a human action,
wherein each spatial edge of the plurality of spatial edges describes a spatial relationship between two of the human event features, and
wherein each temporal edge of the plurality of spatial edges describes a temporal relationship between two of the human event features.

### Claim 11 (depends on 1)

The system of claim 1,
wherein the one or more multimodal subevents comprise one or more meteorological predictions,
wherein the feature descriptor of each node of the plurality of nodes describes a meteorological feature,
wherein each spatial edge of the plurality of spatial edges describes a spatial relationship between two of the meteorological features, and
wherein each temporal edge of the plurality of spatial edges describes a temporal relationship between two of the meteorological features.

### Claim 12 (depends on 1)

The system of claim 1,
wherein the one or more multimodal subevents comprise one or more financial predictions,
wherein the feature descriptor of each node of the plurality of nodes describes a financial feature,
wherein each spatial edge of the plurality of spatial edges describes a spatial relationship between two of the financial features, and
wherein each temporal edge of the plurality of spatial edges describes a temporal relationship between two of the financial features.

### Claim 13

A method for identifying one or more multimodal subevents within an event having spatially-related and temporally-related features, the method comprising:
receiving, by a computing system comprising a machine learning system, a Spatio-Temporal Graph (STG) comprising (1) a plurality of nodes, each node of the plurality of nodes having a feature descriptor that describes a feature present in the event, (2) a plurality of spatial edges, each spatial edge of the plurality of spatial edges describing a spatial relationship between two of the plurality of nodes, and (3) a plurality of temporal edges, each temporal edge of the plurality of temporal edges describing a temporal relationship between two of the plurality of nodes,
wherein the STG comprises at least one of: (1) variable-length descriptors for the feature descriptors; or (2) temporal edges that span multiple time steps for the event;
executing, by the computing system, the machine learning system to process the STG to identify the one or more multimodal subevents for the event, wherein the machine learning system comprises one or more stacked Spatio-Temporal Graph Convolutional Networks (STGCNs) each comprising a plurality of STGCN layers; and
outputting, by an output device, an indication of the one or more multimodal subevents.

### Claim 14 (depends on 13)

The method of claim 13,
wherein the STG comprises the variable-length feature descriptors, and
wherein executing the machine learning system to process the STG further comprises processing the STG to convert the variable-length feature descriptors to fixed-length feature descriptors.

### Claim 15 (depends on 14)

The method of claim 14,
wherein the plurality of nodes of the STG comprise a plurality of clusters of nodes, each cluster of the plurality of clusters having feature descriptors that have a length that is different from a length of feature descriptors of each other cluster of the one or more clusters,
wherein, to process the STG to convert the variable-length feature descriptors to fixed-length feature descriptors, the one or more stacked STGCNs further comprise, for each cluster of the plurality of clusters, a spatial graph convolutional network (GCN) configured to process the cluster to convert the variable-length feature descriptors of the nodes of the cluster to the fixed-length feature descriptors.

### Claim 16 (depends on 14)

The method of claim 14,
wherein, processing the STG to convert the variable-length feature descriptors to fixed-length feature descriptors comprises converting, by one or more convolutional layers of the one or more stacked STGCNs, the variable-length feature descriptors of the plurality of nodes of the fixed-length feature descriptors.

### Claim 17 (depends on 13)

The method of claim 13,
wherein the STG comprises an adjacency matrix that represents irregular connections between the plurality of nodes, and
wherein processing the STG to identify the one or more multimodal subevents for the event comprises performing, by the one or more STGCNs, a plurality of down-sampling operations and a plurality of up-sampling operations on the adjacency matrix to identify the one or more multimodal subevents for the event.

### Claim 18 (depends on 13)

The method of claim 13, wherein each node of the plurality of nodes of the STG is connected to one or more of the plurality of spatial edges and the plurality of temporal edges to enable one or more spatial relationships and temporal relationships between the plurality of nodes.

### Claim 19 (depends on 1)

The method of claim 1,
wherein the one or more multimodal subevents comprise one or more recognized human or animal activities,
wherein the feature descriptor of each node of the plurality of nodes describes a human event feature, the human event feature including one of at least an actor, an object, a scene, or a human action,
wherein each spatial edge of the plurality of spatial edges describes a spatial relationship between two of the human event features, and
wherein each temporal edge of the plurality of spatial edges describes a temporal relationship between two of the human event features.

### Claim 20

A non-transitory computer-readable medium comprising instructions for identifying one or more multimodal subevents within an event having spatially-related and temporally-related features, the instructions configured to cause processing circuitry of a computation engine to:
receive a Spatio-Temporal Graph (STG) comprising (1) a plurality of nodes, each node of the plurality of nodes having a feature descriptor that describes a feature present in the event, (2) a plurality of spatial edges, each spatial edge of the plurality of spatial edges describing a spatial relationship between two of the plurality of nodes, and (3) a plurality of temporal edges, each temporal edge of the plurality of temporal edges describing a temporal relationship between two of the plurality of nodes,
wherein the STG comprises at least one of: (1) variable-length descriptors for the feature descriptors; or (2) temporal edges that span multiple time steps for the event;
execute a machine learning system to process the STG to identify the one or more multimodal subevents for the event, wherein the machine learning system comprises one or more stacked Spatio-Temporal Graph Convolutional Networks (STGCNs) each comprising a plurality of STGCN layers; and
output an indication of the one or more multimodal subevents.
