# US 2021/0232825 A1 · Video classification method, model training method, device, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0232825 A1
- Document ID: US20210232825A1
- Status: Published application
- Kind: A1
- Application number: 17/230,778
- Filed: 2021-04-14
- Published: 2021-07-29
- Earliest claimed: 2019-03-06
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Yongyi TANG; Lin Ma; Wei Liu
- Web page: https://patentgrep.com/patent/US20210232825A1

## CPC

- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06V 10/56: relating to colour
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 10/82: using neural networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

This application discloses a video classification method performed by a computer device. The method includes: obtaining an image frame sequence corresponding to a to-be-classified video file; obtaining an appearance information feature sequence corresponding to the image frame sequence by using an image classification network model, the appearance information feature sequence including T appearance information features; obtaining a motion information feature sequence corresponding to the appearance information feature sequence by using a motion prediction network model, the motion information feature sequence including T motion information features, and the motion prediction network model being configured to predict the motion information features corresponding to the appearance information features; and determining a video classification result of the to-be-classified video file according to the appearance information feature sequence and the motion information feature sequence. This application further provides a model training method, thereby saving computing time and resources while improving data processing efficiency.

## Claims

### Claim 1

A video classification method, applicable to a computer device, the method comprising:
obtaining an image frame sequence corresponding to a to-be-classified video file, the image frame sequence comprising T image frames, T being an integer greater than 1;
obtaining an appearance information feature sequence corresponding to the image frame sequence, the appearance information feature sequence comprising T appearance information features, each appearance information feature having a correspondence with one of the T image frames;
obtaining a motion information feature sequence corresponding to the appearance information feature sequence, the motion information feature sequence comprising T motion information features, each motion information feature having a correspondence with one of the T appearance information features; and
determining a video classification result of the to-be-classified video file according to the appearance information feature sequence and the motion information feature sequence.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the obtaining an appearance information feature sequence corresponding to the image frame sequence comprises:
obtaining the appearance information feature sequence corresponding to the image frame sequence by using an image classification network model; and
the obtaining a motion information feature sequence corresponding to the appearance information feature sequence comprises:
obtaining the motion information feature sequence corresponding to the appearance information feature sequence by using a motion prediction network model, the motion prediction network model being configured to predict the motion information features corresponding to the appearance information features.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the obtaining an image frame sequence corresponding to a to-be-classified video file comprises:
obtaining the to-be-classified video file; and
decoding the to-be-classified video file, to obtain the image frame sequence, a sampling frequency for each image frame being the same.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the obtaining an appearance information feature sequence corresponding to the image frame sequence comprises:
obtaining an appearance information feature corresponding to a tth image frame, t being an integer greater than or equal to 1 and less than or equal to T; and
generating the appearance information feature sequence according to the T appearance information features when an appearance information feature corresponding to each of the T image frames has been obtained.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the obtaining a motion information feature sequence corresponding to the appearance information feature sequence comprises:
obtaining a tth appearance information feature corresponding to an ith step, i being an integer greater than or equal to 1, and t being an integer greater than or equal to 1 and less than or equal to T;
obtaining a (t+1)th motion information feature and a (t−1)th motion information feature corresponding to an (i−1)th step;
determining a tth motion information feature corresponding to the ith step according to the tth appearance information feature, the (t+1)th motion information feature, and the (t−1)th motion information feature; and
generating the motion information feature sequence according to the T motion information features when a motion information feature corresponding to each of the T appearance information features has been obtained.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the determining a tth motion information feature corresponding to the ith step according to the tth appearance information feature, the (t+1)th motion information feature, and the (t−1)th motion information feature comprises:
calculating a (t+1)th update gate vector and a (t−1)th update gate vector corresponding to the ith step according to the tth appearance information feature, the (t+1)th motion information feature, and the (t−1)th motion information feature by using an update gate of the motion prediction network model;
calculating a (t+1)th output gate vector and a (t−1)th output gate vector corresponding to the ith step according to the tth appearance information feature, the (t+1)th motion information feature, and the (t−1)th motion information feature by using an output gate of the motion prediction network model;
calculating a tth hidden state feature vector corresponding to the ith step according to the tth appearance information feature, the (t+1)th motion information feature, the (t−1)th motion information feature, the (t+1)th update gate vector, and the (t−1)th update gate vector by using a hidden state of the motion prediction network model; and
calculating the tth motion information feature corresponding to the ith step according to the tth hidden state feature vector and the (t+1)th output gate vector and the (t−1)th output gate vector corresponding to the ith step by using a final output layer of the motion prediction network model.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the determining a video classification result of the to-be-classified video file according to the appearance information feature sequence and the motion information feature sequence comprises:
obtaining a video class vector corresponding to the appearance information feature sequence and the motion information feature sequence, the video class vector comprising C video class elements, C being an integer greater than 1;
determining at least one target video class element from the video class vector; and
generating the video classification result of the to-be-classified video file according to the at least one target video class element.

### Claim 8

A computer device, comprising: a memory, a transceiver, a processor, and a bus system, the bus system being configured to connect the memory and the processor, to enable the memory and the processor to perform communication,
the memory being configured to store a plurality of programs;
the processor being configured to execute the plurality of programs in the memory to perform a plurality of operations including:
obtaining an image frame sequence corresponding to a to-be-classified video file, the image frame sequence comprising T image frames, T being an integer greater than 1;
obtaining an appearance information feature sequence corresponding to the image frame sequence, the appearance information feature sequence comprising T appearance information features, each appearance information feature having a correspondence with one of the T image frames;
obtaining a motion information feature sequence corresponding to the appearance information feature sequence, the motion information feature sequence comprising T motion information features, each motion information feature having a correspondence with one of the T appearance information feature; and
determining a video classification result of the to-be-classified video file according to the appearance information feature sequence and the motion information feature sequence.

### Claim 9 (depends on 8)

The computer device according to claim 8, wherein the processor is configured to:
obtain the appearance information feature sequence corresponding to the image frame sequence by using an image classification network model; and
the obtaining a motion information feature sequence corresponding to the appearance information feature sequence comprises:
obtaining the motion information feature sequence corresponding to the appearance information feature sequence by using a motion prediction network model, the motion prediction network model being configured to predict the motion information features corresponding to the appearance information features.

### Claim 10 (depends on 8)

The computer device according to claim 8, wherein the processor is configured to:
obtain the to-be-classified video file; and
decode the to-be-classified video file, to obtain the image frame sequence, a sampling frequency for each image frame being the same.

### Claim 11 (depends on 8)

The computer device according to claim 8, wherein the processor is configured to:
obtain an appearance information feature corresponding to a tth image frame, t being an integer greater than or equal to 1 and less than or equal to T; and
generate the appearance information feature sequence according to the T appearance information features when an appearance information feature corresponding to each of the T image frames has been obtained.

### Claim 12 (depends on 8)

The computer device according to claim 8, wherein the processor is configured to:
obtain a tth appearance information feature corresponding to an ith step, i being an integer greater than or equal to 1, and t being an integer greater than or equal to 1 and less than or equal to T;
obtain a (t+1)th motion information feature and a (t−1)th motion information feature corresponding to an (i−1)th step;
determine a tth motion information feature corresponding to the ith step according to the tth appearance information feature, the (t+1)th motion information feature, and the (t−1)th motion information feature; and
generate the motion information feature sequence according to the T motion information features when a motion information feature corresponding to each of the T appearance information features has been obtained.

### Claim 13 (depends on 12)

The computer device according to claim 12, wherein the processor is configured to:
calculate a (t+1)th update gate vector and a (t−1)th update gate vector corresponding to the ith step according to the tth appearance information feature, the (t+1)th motion information feature, and the (t−1)th motion information feature by using an update gate of the motion prediction network model;
calculate a (t+1)th output gate vector and a (t−1)th output gate vector corresponding to the ith step according to the tth appearance information feature, the (t+1)th motion information feature, and the (t−1)th motion information feature by using an output gate of the motion prediction network model;
calculate a tth hidden state feature vector corresponding to the ith step according to the tth appearance information feature, the (t+1)th motion information feature, the (t−1)th motion information feature, the (t+1)th update gate vector, and the (t−1)th update gate vector by using a hidden state of the motion prediction network model; and
calculate the tth motion information feature corresponding to the ith step according to the tth hidden state feature vector and the (t+1)th output gate vector and the (t−1)th output gate vector corresponding to the ith step by using a final output layer of the motion prediction network model.

### Claim 14 (depends on 8)

The computer device according to claim 8, wherein the processor is configured to:
obtain a video class vector corresponding to the appearance information feature sequence and the motion information feature sequence, the video class vector comprising C video class elements, C being an integer greater than 1;
determine at least one target video class element from the video class vector; and
generate the video classification result of the to-be-classified video file according to the at least one target video class element.

### Claim 15

A non-transitory computer-readable storage medium, comprising instructions, the instructions, when run on a computer device, causing the computer device to perform a plurality of operations including:
obtaining an image frame sequence corresponding to a to-be-classified video file, the image frame sequence comprising T image frames, T being an integer greater than 1;
obtaining an appearance information feature sequence corresponding to the image frame sequence, the appearance information feature sequence comprising T appearance information features, each appearance information feature having a correspondence with one of the T image frames;
obtaining a motion information feature sequence corresponding to the appearance information feature sequence, the motion information feature sequence comprising T motion information features, each motion information feature having a correspondence with one of the T appearance information features; and
determining a video classification result of the to-be-classified video file according to the appearance information feature sequence and the motion information feature sequence.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium according to claim 15, wherein the obtaining an appearance information feature sequence corresponding to the image frame sequence comprises:
obtaining the appearance information feature sequence corresponding to the image frame sequence by using an image classification network model; and
the obtaining a motion information feature sequence corresponding to the appearance information feature sequence comprises:
obtaining the motion information feature sequence corresponding to the appearance information feature sequence by using a motion prediction network model, the motion prediction network model being configured to predict the motion information features corresponding to the appearance information features.

### Claim 17 (depends on 15)

The non-transitory computer-readable storage medium according to claim 15, wherein the obtaining an image frame sequence corresponding to a to-be-classified video file comprises:
obtaining the to-be-classified video file; and
decoding the to-be-classified video file, to obtain the image frame sequence, a sampling frequency for each image frame being the same.

### Claim 18 (depends on 15)

The non-transitory computer-readable storage medium according to claim 15, wherein the obtaining an appearance information feature sequence corresponding to the image frame sequence comprises:
obtaining an appearance information feature corresponding to a tth image frame, t being an integer greater than or equal to 1 and less than or equal to T; and
generating the appearance information feature sequence according to the T appearance information features when an appearance information feature corresponding to each of the T image frames has been obtained.

### Claim 19 (depends on 15)

The non-transitory computer-readable storage medium according to claim 15, wherein the obtaining a motion information feature sequence corresponding to the appearance information feature sequence comprises:
obtaining a tth appearance information feature corresponding to an ith step, i being an integer greater than or equal to 1, and t being an integer greater than or equal to 1 and less than or equal to T;
obtaining a (t+1)th motion information feature and a (t−1)th motion information feature corresponding to an (i−1)th step;
determining a tth motion information feature corresponding to the ith step according to the tth appearance information feature, the (t+1)th motion information feature, and the (t−1)th motion information feature; and
generating the motion information feature sequence according to the T motion information features when a motion information feature corresponding to each of the T appearance information features has been obtained.

### Claim 20 (depends on 15)

The non-transitory computer-readable storage medium according to claim 15, wherein the determining a video classification result of the to-be-classified video file according to the appearance information feature sequence and the motion information feature sequence comprises:
obtaining a video class vector corresponding to the appearance information feature sequence and the motion information feature sequence, the video class vector comprising C video class elements, C being an integer greater than 1;
determining at least one target video class element from the video class vector; and
generating the video classification result of the to-be-classified video file according to the at least one target video class element.
