# US 2021/0174152 A1 · Video classification method and server

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174152 A1
- Document ID: US20210174152A1
- Status: Published application
- Kind: A1
- Application number: 17/180,510
- Filed: 2021-02-19
- Published: 2021-06-10
- Earliest claimed: 2018-11-29
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Lin MA
- Web page: https://patentgrep.com/patent/US20210174152A1

## CPC

- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06F 18/24: Classification techniques
- G06F 18/29: {Graphical models, e.g. Bayesian networks}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G10L 25/30: using neural networks
- G10L 25/51: for comparison or discrimination
- G10L 25/57: for processing of video signals
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Embodiments of this application disclose a video classification method, including: obtaining a to-be-processed video; obtaining a visual signal feature sequence corresponding to the to-be-processed video by using a video classification prediction model; obtaining an audio signal feature sequence corresponding to the visual signal feature sequence by using the video classification prediction model; generating a target signal feature sequence according to the visual signal feature sequence and the audio signal feature sequence; and obtaining a classification prediction result corresponding to the target signal feature sequence by using the video classification prediction model, the classification prediction result being used for predicting a video type of the to-be-processed video. The embodiments of this application further disclose a server. In the embodiments of this application, in the time dimension, not only feature changes of visual features but also feature changes of audio features are taken into account, so as improve the effect of video classification.

## Claims

### Claim 1

A video classification method, comprising:
obtaining a to-be-processed video, the to-be-processed video comprising at least one video frame, each video frame corresponding to a time point;
obtaining a visual signal feature sequence corresponding to the to-be-processed video, the visual signal feature sequence comprising at least one visual signal feature, each visual signal feature corresponding to a respective video frame at a time point;
obtaining an audio signal feature sequence corresponding to the visual signal feature sequence, the audio signal feature sequence comprising at least one audio signal feature, each audio signal feature corresponding to a respective visual signal feature at a time point;
generating a target signal feature sequence according to the visual signal feature sequence and the audio signal feature sequence; and
obtaining a classification prediction result corresponding to the target signal feature sequence, the classification prediction result being used for predicting a video type of the to-be-processed video.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the obtaining a visual signal feature sequence corresponding to the to-be-processed video comprises:
obtaining the visual signal feature of the video frame in the to-be-processed video by using a first submodel in a video classification prediction model, the first submodel comprising a convolutional neural network (CNN) model; and
generating the visual signal feature sequence corresponding to the to-be-processed video according to the visual signal feature of the video frame.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the obtaining an audio signal feature sequence corresponding to the visual signal feature sequence comprises:
obtaining the audio signal feature of the visual signal feature in the visual signal feature sequence by using a second submodel in the video classification prediction model, the second submodel comprising a gated-recurrent-unit (GRU)-based recurrent neural model; and
generating the audio signal feature sequence corresponding to the visual signal feature sequence according to the audio signal feature of the visual signal feature.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the obtaining the audio signal feature of the visual signal feature in the visual signal feature sequence by using a second submodel in the video classification prediction model comprises:
obtaining a visual signal feature of a first video frame at a first time point and a visual signal hidden state feature of a second video frame at a second time point, the second time point being immediately before the first time point;
obtaining a visual signal hidden state feature of the first video frame at the first time point by using the second submodel, the visual signal hidden state feature of the first video frame at the first time point corresponding to the visual signal feature of the first video frame at the first time point, the visual signal hidden state feature of the first video frame at the first time point corresponding to the visual signal hidden state feature of the second video frame at the second time point;
obtaining an audio signal hidden state feature of the second video frame at the second time point;
obtaining an audio signal hidden state feature of the first video frame at the first time point by using the second submodel, the audio signal hidden state feature of the first video frame at the first time point corresponding to the audio signal hidden state feature of the second video frame at the second time point, the audio signal hidden state feature of the first video frame at the first time point corresponding to the visual signal hidden state feature of the first video frame at the first time point; and
generating an audio signal feature of the first video frame at the first time point according to the audio signal hidden state feature of the first video frame at the first time point.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the obtaining a visual signal hidden state feature of the first video frame at the first time point by using the second submodel comprises:
calculating the visual signal hidden state feature of the first video frame at the first time point in the following manner:
ht=GRU(xt,ht-1) for tϵ[1,T],
wherein ht represents the visual signal hidden state feature of the first video frame at the first time point, GRU( ) represents a gated recurrent unit function, xt represents the visual signal feature of the first video frame at the first time point, ht-1 represents the audio signal hidden state feature of the second video frame at the second time point, t represents the first time point, and T represents a total quantity of time points of the to-be-processed video.

### Claim 6 (depends on 4)

The method according to claim 4, wherein the obtaining an audio signal hidden state feature of the first video frame at the first time point by using the second submodel comprises:
calculating the audio signal hidden state feature of the first video frame at the first time point in the following manner:
yt=GRU(ht,yt-1) for tϵ[1,T],
wherein yt represents the audio signal hidden state feature of the first video frame at the first time point, GRU( ) represents a gated recurrent unit function, ht represents the visual signal hidden state feature of the first video frame at the first time point, yt-1 represents the audio signal hidden state feature of the second video frame at the second time point, t represents the first time point, and T represents the total quantity of time points of the to-be-processed video.

### Claim 7 (depends on 4)

The method according to claim 4, wherein the generating an audio signal feature of the first video frame at the first time point according to the audio signal hidden state feature of the first video frame at the first time point comprises:
calculating the audio signal feature of the first video frame at the first time point in the following manner:
mt=FC(yt) for tϵ[1,T],
wherein mt represents the audio signal feature of the first video frame at the first time point, yt represents the audio signal hidden state feature of the first video frame at the first time point, FC( ) represents a fully-connected network function, t represents the first time point, and T represents the total quantity of time points of the to-be-processed video.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the obtaining a classification prediction result corresponding to the target signal feature sequence comprises:
obtaining an audio-visual signal feature of each target signal feature in the target signal feature sequence by using a third submodel in the video classification prediction model, the third submodel comprising a GRU-based recurrent neural model;
generating an audio-visual signal feature sequence corresponding to the target signal feature sequence according to the audio-visual signal feature of the target signal feature; and
determining the classification prediction result according to the audio-visual signal feature sequence.

### Claim 9 (depends on 8)

The method according to claim 8, wherein the obtaining an audio-visual signal feature of each target signal feature in the target signal feature sequence by using a third submodel in the video classification prediction model comprises:
obtaining a target signal feature of the first video frame at the first time point and an audio-visual signal feature of the second video frame at the second time point, the second time point being immediately before the first time point; and
obtaining an audio-visual signal feature of the first video frame at the first time point by using the third submodel, the audio-visual signal feature of the first video frame at the first time point and the target signal feature of the first video frame at the first time point corresponding to each other, the audio-visual signal feature of the first video frame at the first time point and the audio-visual signal feature of the second video frame at the second time point corresponding to each other.

### Claim 10 (depends on 9)

The method according to claim 9, wherein the obtaining an audio-visual signal feature of the first video frame at the first time point by using the third submodel comprises:
calculating the audio-visual signal feature of the first video frame at the first time point in the following manner:
Ht=GRU(vt,Ht-1) for tϵ[1,T], and
vt=[ht;mt],
wherein Ht represents the audio-visual signal feature of the first video frame at the first time point, GRU( ) represents a gated recurrent unit function, Ht-1 represents the audio-visual signal feature of the second video frame at the second time point, vt represents the target signal feature of the first video frame at the first time point, ht represents the visual signal hidden state feature of the first video frame at the first time point, and mt represents the audio signal feature of the first video frame at the first time point.

### Claim 11 (depends on 8)

The method according to claim 8, wherein the determining the classification prediction result according to the audio-visual signal feature sequence comprises:
calculating the classification prediction result in the following manner:
lable
=
∑
n
=
1
N


g
n

a
n
,

g
n
=
σ
g

(
W
g

H
T
+
b
g
)
,
n
∈
[
1
,
N
]
,
and
a
n
=
σ
a

(
W
a

H
T
+
b
a
)
,
n
∈
[
1
,
N
]
,
wherein lable represents the classification prediction result, gn represents a gate representation function, an represents an activation representation function, Wg represent a first matrix of linear transformation parameters, Wa represent a second matrix of linear transformation parameters, bg represents a first bias parameter vector, ba represents a second bias parameter vector, HT represents the audio-visual signal feature sequence, σg represents a softmax function, σa represents a sigmoid function, N represents a total quantity of paths obtained after nonlinear transformation, and n represents an nth path obtained after nonlinear transformation.

### Claim 12

A computing device, comprising a memory and a processor;
the memory storing a plurality of instructions that, when executed by the processor, cause the computing device to perform a plurality of operations including:
obtaining a to-be-processed video, the to-be-processed video comprising at least one video frame, each video frame corresponding to a time point;
obtaining a visual signal feature sequence corresponding to the to-be-processed video, the visual signal feature sequence comprising at least one visual signal feature, each visual signal feature corresponding to a respective video frame at a time point;
obtaining an audio signal feature sequence corresponding to the visual signal feature sequence, the audio signal feature sequence comprising at least one audio signal feature, each audio signal feature corresponding to a respective visual signal feature at a time point;
generating a target signal feature sequence according to the visual signal feature sequence and the audio signal feature sequence; and
obtaining a classification prediction result corresponding to the target signal feature sequence, the classification prediction result being used for predicting a video type of the to-be-processed video.

### Claim 13 (depends on 12)

The computing device according to claim 12, wherein the instruction for obtaining a visual signal feature sequence corresponding to the to-be-processed video comprises:
obtaining the visual signal feature of the video frame in the to-be-processed video by using a first submodel in a video classification prediction model, the first submodel comprising a convolutional neural network (CNN) model; and
generating the visual signal feature sequence corresponding to the to-be-processed video according to the visual signal feature of the video frame.

### Claim 14 (depends on 12)

The computing device according to claim 12, wherein the instruction for obtaining an audio signal feature sequence corresponding to the visual signal feature sequence comprises:
obtaining the audio signal feature of the visual signal feature in the visual signal feature sequence by using a second submodel in the video classification prediction model, the second submodel comprising a gated-recurrent-unit (GRU)-based recurrent neural model; and
generating the audio signal feature sequence corresponding to the visual signal feature sequence according to the audio signal feature of the visual signal feature.

### Claim 15 (depends on 14)

The computing device according to claim 14, wherein the obtaining the audio signal feature of the visual signal feature in the visual signal feature sequence by using a second submodel in the video classification prediction model comprises:
obtaining a visual signal feature of a first video frame at a first time point and a visual signal hidden state feature of a second video frame at a second time point, the second time point being immediately before the first time point;
obtaining a visual signal hidden state feature of the first video frame at the first time point by using the second submodel, the visual signal hidden state feature of the first video frame at the first time point corresponding to the visual signal feature of the first video frame at the first time point, the visual signal hidden state feature of the first video frame at the first time point corresponding to the visual signal hidden state feature of the second video frame at the second time point;
obtaining an audio signal hidden state feature of the second video frame at the second time point;
obtaining an audio signal hidden state feature of the first video frame at the first time point by using the second submodel, the audio signal hidden state feature of the first video frame at the first time point corresponding to the audio signal hidden state feature of the second video frame at the second time point, the audio signal hidden state feature of the first video frame at the first time point corresponding to the visual signal hidden state feature of the first video frame at the first time point; and
generating an audio signal feature of the first video frame at the first time point according to the audio signal hidden state feature of the first video frame at the first time point.

### Claim 16 (depends on 12)

The computing device according to claim 12, wherein the instruction for obtaining a classification prediction result corresponding to the target signal feature sequence comprises:
obtaining an audio-visual signal feature of each target signal feature in the target signal feature sequence by using a third submodel in the video classification prediction model, the third submodel comprising a GRU-based recurrent neural model;
generating an audio-visual signal feature sequence corresponding to the target signal feature sequence according to the audio-visual signal feature of the target signal feature; and
determining the classification prediction result according to the audio-visual signal feature sequence.

### Claim 17

A non-transitory computer-readable storage medium, storing a plurality of instructions that, when executed by a processor of a computing device, cause the computing device to perform a plurality of operations including:
obtaining a to-be-processed video, the to-be-processed video comprising at least one video frame, each video frame corresponding to a time point;
obtaining a visual signal feature sequence corresponding to the to-be-processed video, the visual signal feature sequence comprising at least one visual signal feature, each visual signal feature corresponding to a respective video frame at a time point;
obtaining an audio signal feature sequence corresponding to the visual signal feature sequence, the audio signal feature sequence comprising at least one audio signal feature, each audio signal feature corresponding to a respective visual signal feature at a time point;
generating a target signal feature sequence according to the visual signal feature sequence and the audio signal feature sequence; and
obtaining a classification prediction result corresponding to the target signal feature sequence, the classification prediction result being used for predicting a video type of the to-be-processed video.

### Claim 18 (depends on 17)

The non-transitory computer-readable storage medium according to claim 17, wherein the instruction for obtaining a visual signal feature sequence corresponding to the to-be-processed video comprises:
obtaining the visual signal feature of the video frame in the to-be-processed video by using a first submodel in a video classification prediction model, the first submodel comprising a convolutional neural network (CNN) model; and
generating the visual signal feature sequence corresponding to the to-be-processed video according to the visual signal feature of the video frame.

### Claim 19 (depends on 17)

The non-transitory computer-readable storage medium according to claim 17, wherein the instruction for obtaining an audio signal feature sequence corresponding to the visual signal feature sequence comprises:
obtaining the audio signal feature of the visual signal feature in the visual signal feature sequence by using a second submodel in the video classification prediction model, the second submodel comprising a gated-recurrent-unit (GRU)-based recurrent neural model; and
generating the audio signal feature sequence corresponding to the visual signal feature sequence according to the audio signal feature of the visual signal feature.

### Claim 20 (depends on 17)

The non-transitory computer-readable storage medium according to claim 17, wherein the instruction for obtaining a classification prediction result corresponding to the target signal feature sequence comprises:
obtaining an audio-visual signal feature of each target signal feature in the target signal feature sequence by using a third submodel in the video classification prediction model, the third submodel comprising a GRU-based recurrent neural model;
generating an audio-visual signal feature sequence corresponding to the target signal feature sequence according to the audio-visual signal feature of the target signal feature; and
determining the classification prediction result according to the audio-visual signal feature sequence.
