# US 2023/0410487 A1 · Online learning method and system for action recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410487 A1
- Document ID: US20230410487A1
- Status: Published application
- Kind: A1
- Application number: 18/250,498
- Filed: 2020-11-30
- Published: 2023-12-21
- Earliest claimed: 2020-11-30
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Lidan Zhang; Qi She; Ping Guo; Yimin Zhang
- Web page: https://patentgrep.com/patent/US20230410487A1

## CPC

- G06V 10/778: Active pattern-learning, e.g. online learning of image or video features
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/44: {Event detection}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Performing online learning for a model to detect unseen actions in an action recognition system is disclosed. The method includes extracting semantic features in a semantic domain from semantic action labels, transforming the semantic features from the semantic domain into mixed features in a mixed domain, and storing the mixed features in a feature database. The method further includes extracting visual features in a visual domain from a video stream and determining if the visual features indicate an unseen action in the video stream. If no unseen action is determined, applying an offline classification model to the visual features to identify seen actions, assigning identifiers to the identified seen actions, transforming the visual features from the visual domain into mixed features in the mixed domain, and storing the mixed features and seen action identifiers in the feature database. If an unseen action is determined, transforming the visual features from the visual domain into mixed features in the mixed domain, applying a continual learner model to mixed features from the feature database to identify unseen actions in the video stream, assigning identifiers to the identified unseen actions, and storing the unseen action identifiers in the feature database.

## Claims

### Claim 1

At least one computer-readable medium having stored thereon instructions which, when executed, cause a computing device to perform operations comprising:
extract semantic features in a semantic domain from semantic action labels, transform the semantic features from the semantic domain into mixed features in a mixed domain, and store the mixed features in a feature database;
extract visual features in a visual domain from a video stream;
determine if the visual features indicate an unseen action in the video stream;
if no unseen action is determined, apply an offline classification model to the visual features to identify seen actions, assign identifiers to the identified seen actions, transform the visual features from the visual domain into mixed features in the mixed domain, and store the mixed features and seen action identifiers in the feature database; and
if an unseen action is determined, transform the visual features from the visual domain into mixed features in the mixed domain, apply a continual learner model to mixed features from the feature database to identify unseen actions in the video stream, assign identifiers to the identified unseen actions, and store the unseen action identifiers in the feature database.

### Claim 2 (depends on 1)

The computer-readable medium of claim 1, wherein determining if the visual features indicate an unseen action in the video stream comprises applying a machine learning (ML) classifier with a binary output value.

### Claim 3 (depends on 2)

The computer-readable medium of claim 2, wherein the operations comprise training the ML classifier using a generative adversarial network to generate unseen visualization features from semantic features.

### Claim 4 (depends on 1)

The computer-readable medium of claim 1, wherein the continual learner model applies a K nearest neighbors process to the mixed features to identify unseen actions.

### Claim 5 (depends on 1)

The computer-readable medium of claim 1, wherein the offline classification model recognizes human actions using a video action transformer network.

### Claim 6 (depends on 1)

The computer-readable medium of claim 1, wherein semantic features are extracted, the semantic features are transformed into mixed features, and the mixed features are stored in the feature database, in a training phase, wherein extracting visual features comprises applying an offline I3D classification model to the video stream.

### Claim 7 (depends on 1)

The computer-readable medium of claim 1, wherein action identifiers are associated with action categories.

### Claim 8

An apparatus comprising:
a processing device; and
a memory device coupled to the processing device, the memory device having instructions stored thereon that, in response to execution by the processing device, cause the processing device to:
extract semantic features in a semantic domain from semantic action labels, transform the semantic features from the semantic domain into mixed features in a mixed domain, and store the mixed features in a feature database;
extract visual features in a visual domain from a video stream;
determine if the visual features indicate an unseen action in the video stream;
if no unseen action is determined, apply an offline classification model to the visual features to identify seen actions, assign identifiers to the identified seen actions, transform the visual features from the visual domain into mixed features in the mixed domain, and store the mixed features and seen action identifiers in the feature database; and
if an unseen action is determined, transform the visual features from the visual domain into mixed features in the mixed domain, apply a continual learner model to mixed features from the feature database to identify unseen actions in the video stream, assign identifiers to the identified unseen actions, and store the unseen action identifiers in the feature database.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein determining if the visual features indicate an unseen action in the video stream comprises applying a machine learning (ML) classifier with a binary output value, wherein the ML classifier is trained using a generative adversarial network to generate unseen visualization features from semantic features.

### Claim 10 (depends on 8)

The apparatus of claim 8, wherein the continual learner model applies a K nearest neighbors process to the mixed features to identify unseen actions.

### Claim 11 (depends on 8)

The apparatus of claim 8, wherein the offline classification model recognizes human actions using a video action transformer network.

### Claim 12 (depends on 8)

The apparatus of claim 8, wherein semantic features are extracted, the semantic features are transformed into mixed features, and the mixed features are stored in the feature database, in a training phase, wherein extracting visual features comprises applying an offline I3D classification model to the video stream.

### Claim 13 (depends on 8)

The apparatus of claim 8, wherein action identifiers are associated with action categories.

### Claim 14

A method comprising:
extracting semantic features in a semantic domain from semantic action labels, transforming the semantic features from the semantic domain into mixed features in a mixed domain, and storing the mixed features in a feature database;
extracting visual features in a visual domain from a video stream;
determining if the visual features indicate an unseen action in the video stream;
if no unseen action is determined, applying an offline classification model to the visual features to identify seen actions, assigning identifiers to the identified seen actions, transforming the visual features from the visual domain into mixed features in the mixed domain, and storing the mixed features and seen action identifiers in the feature database; and
if an unseen action is determined, transforming the visual features from the visual domain into mixed features in the mixed domain, applying a continual learner model to mixed features from the feature database to identify unseen actions in the video stream, assigning identifiers to the identified unseen actions, and storing the unseen action identifiers in the feature database.

### Claim 15 (depends on 14)

The method of claim 14, wherein determining if the visual features indicate an unseen action in the video stream comprises applying a machine learning (ML) classifier with a binary output value.

### Claim 16 (depends on 15)

The method of claim 15, further comprising training the ML classifier using a generative adversarial network to generate unseen visualization features from semantic features.

### Claim 17 (depends on 14)

The method of claim 14, wherein the continual learner model applies a K nearest neighbors process to the mixed features to identify unseen actions.

### Claim 18 (depends on 14)

The method of claim 14, wherein the offline classification model recognizes human actions using a video action transformer network.

### Claim 19 (depends on 14)

The method of claim 14, wherein semantic features are extracted, the semantic features are transformed into mixed features, and the mixed features are stored in the feature database, in a training phase, wherein extracting visual features comprises applying an offline I3D classification model to the video stream.

### Claim 20 (depends on 14)

The method of claim 14, wherein action identifiers are associated with action categories.
