# US 2021/0081673 A1 · Action recognition with high-order interaction through spatial-temporal object tracking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0081673 A1
- Document ID: US20210081673A1
- Status: Published application
- Kind: A1
- Application number: 17/016,260
- Filed: 2020-09-09
- Published: 2021-03-18
- Earliest claimed: 2019-09-12
- Assignees: NEC Laboratories America, Inc.
- Applicants: NEC Laboratories America, Inc.
- Inventors: Farley LAI; Asim KADAV; Jie CHEN
- Web page: https://patentgrep.com/patent/US20210081673A1

## CPC

- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Aspects of the present disclosure describe systems, methods, and structures that provide action recognition with high-order interaction with spatio-temporal object tracking. Image and object features are organized into into tracks, which advantageously facilitates many possible learnable embeddings and intra/inter-track interaction(s). Operationally, our systems, method, and structures according to the present disclosure employ an efficient high-order interaction model to learn embeddings and intra/inter object track interaction across the space and time for AR. Each frame is detected by an object detector to locate visual objects. Those objects are linked through time to form object tracks. The object tracks are then organized and combined with the embeddings as the input to our model. The model is trained to generate representative embeddings and discriminative video features through high-order interaction which is formulated as an efficient matrix operation without iterative processing delay.

## Claims

### Claim 1

A method for determining action recognition in frames of a video through spatio-temporal object tracking, the method comprising:
detecting visual objects in frames of the video;
linking visual objects that are the same through time to form object tracks;
organizing and combining the object tracks with embeddings;
applying the organized and combined object tracks to a neural network model, said model trained to generate representative embeddings and discriminative video features through high-order interaction formulated as a matrix operation without iterative processing delay.

### Claim 2 (depends on 1)

The method of claim 1 wherein the neural network model is a transformer.
