# US 2023/0069197 A1 · Method, apparatus, device and storage medium for training video recognition model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0069197 A1
- Document ID: US20230069197A1
- Status: Published application
- Kind: A1
- Application number: 17/983,208
- Filed: 2022-11-08
- Published: 2023-03-02
- Earliest claimed: 2021-05-28
- Assignees: Beijing Baidu Netcom Science Technology Co., Ltd.
- Applicants: Beijing Baidu Netcom Science Technology Co., Ltd.
- Inventors: Wenhao WU; Yuxiang Zhao
- Web page: https://patentgrep.com/patent/US20230069197A1

## CPC

- G06V 10/82: using neural networks
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/806: {of extracted features}
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method and an apparatus for training a video recognition model are provided. The method may include: dividing a sample video into a plurality of sample video segments; sampling a part of sample video frames from a sample video segment; inputting the part of sample video frames into a feature extraction network to obtain feature information of the sample video segment; performing convolution fusion on the feature information by using a dynamic segment fusion module to obtain fusion feature information, where a convolution kernel of the dynamic segment fusion module varies with different video inputs; inputting the fusion feature information to a fully connected layer to obtain an estimated category of the sample video; and performing a parameter adjustment based on a difference between the tag of a true category and the estimated category to obtain the video recognition model.

## Claims

### Claim 1

A method for training a video recognition model, comprising:
dividing a sample video into a plurality of sample video segments, wherein the sample video is labeled with a tag of a true category;
sampling a part of sample video frames from a sample video segment of the plurality of sample video segments;
inputting the part of sample video frames into a feature extraction network to obtain feature information of the sample video segment;
performing a convolution fusion on the feature information by using a dynamic segment fusion module to obtain fusion feature information, wherein a convolution kernel of the dynamic segment fusion module varies with different video inputs;
inputting the fusion feature information to a fully connected layer to obtain an estimated category of the sample video; and
performing a parameter adjustment based on a difference between the tag of the true category and the estimated category to obtain the video recognition model.

### Claim 2 (depends on 1)

The method according to claim 1, wherein performing the convolution fusion on the feature information by using the dynamic segment fusion module to obtain the fusion feature information comprises:
dividing the feature information into first feature information and second feature information in a channel dimension;
determining a convolution kernel corresponding to the sample video using a convolution kernel generation branch network;
performing convolution on the first feature information by using the convolution kernel corresponding to the sample video to obtain a convolution result; and
splicing the convolution result with the second feature information to obtain the fusion feature information.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the dividing the feature information into the first feature information and the second feature information in the channel dimension comprises:
dividing the feature information into the first feature information and the second feature information in the channel dimension according to a pre-set hyper-parameter β, wherein the channel dimension of the first feature information is βC, the channel dimension of the second feature information is (1−β)C, and C is the channel dimension of the feature information.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the determining the convolution kernel corresponding to the sample video using the convolution kernel generation branch network comprises:
calculating a product of the channel dimension βC of the first feature information, a number of the plurality of sample video segments, a number of the part of sample video frames of the sample video segment, and a height and a width of the sample video frames; and
inputting the product to the convolution kernel generation branch network to obtain the convolution kernel corresponding to the sample video.

### Claim 5 (depends on 2)

The method according to claim 2, wherein the convolution kernel generation branch network comprises a global average pooling layer and two fully connected layers.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the dynamic segment fusion module comprises at least one dynamic segment fusion module, the video recognition model comprises the at least one dynamic segment fusion module and a plurality of residual layers, and the at least one dynamic segment fusion module is arranged in the plurality of residual layers and at an interval of a residual layer.

### Claim 7 (depends on 1)

The method according to claim 1, wherein dividing the sample video into the plurality of sample video segments comprises:
evenly dividing the sample video according to a length of the sample video to obtain the plurality of sample video segments,
wherein sampling the part of sample video frames from the sample video segment comprises:
sampling video frames from the sample video segment at equal intervals to obtain the part of sample video frames.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the performing the parameter adjustment based on the difference between the tag of the true category and the estimated category to obtain the video recognition model comprises:
calculating a cross entropy loss based on the tag of the true category and the estimated category; and
optimizing the cross entropy loss by using a stochastic gradient descent and continuously updating parameters until the cross entropy loss converges to obtain the video recognition model.

### Claim 9 (depends on 1)

The method according to claim 1, comprising:
acquiring a to-be-recognized video;
dividing the to-be-recognized video into a plurality of to-be-recognized video segments;
sampling a part of to-be-recognized video frames from a to-be-recognized video segment; and
inputting the part of to-be-recognized video frames into the video recognition model to obtain a category of the to-be-recognized video.

### Claim 10 (depends on 9)

The method according to claim 9, wherein a dividing granularity of the to-be-recognized video is greater than a dividing granularity of the sample video used to train the video recognition model.

### Claim 11

An electronic device comprising:
one or more processors; and
a storage device in communication with one or more processor, wherein the storage device stores instructions executable by the one or more processor, to cause the one or more processor to perform operations comprising:
dividing a sample video into a plurality of sample video segments, wherein the sample video is labeled with a tag of a true category;
sampling a part of sample video frames from a sample video segment of the plurality of sample video segments;
inputting the part of sample video frames into a feature extraction network to obtain feature information of the sample video segment;
performing a convolution fusion on the feature information by using a dynamic segment fusion module to obtain fusion feature information, wherein a convolution kernel of the dynamic segment fusion module varies with different video inputs;
inputting the fusion feature information to a fully connected layer to obtain an estimated category of the sample video; and
performing a parameter adjustment based on a difference between the tag of the true category and the estimated category to obtain a video recognition model.

### Claim 12 (depends on 11)

The electronic device according to claim 11, wherein performing the convolution fusion on the feature information by using the dynamic segment fusion module to obtain the fusion feature information comprises:
dividing the feature information into first feature information and second feature information in a channel dimension;
determining a convolution kernel corresponding to the sample video using a convolution kernel generation branch network;
performing convolution on the first feature information by using the convolution kernel corresponding to the sample video to obtain a convolution result; and
splicing the convolution result with the second feature information to obtain the fusion feature information.

### Claim 13 (depends on 12)

The electronic device according to claim 12, wherein the dividing the feature information into the first feature information and the second feature information in the channel dimension comprises:
dividing the feature information into the first feature information and the second feature information in the channel dimension according to a pre-set hyper-parameter β, wherein the channel dimension of the first feature information is βC, the channel dimension of the second feature information is (1−β)C, and C is the channel dimension of the feature information.

### Claim 14 (depends on 13)

The electronic device according to claim 13, wherein the determining the convolution kernel corresponding to the sample video using the convolution kernel generation branch network comprises:
calculating a product of the channel dimension βC of the first feature information, a number of the plurality of sample video segments, a number of the part of sample video frames of the sample video segment, and a height and a width of the sample video frames; and
inputting the product to the convolution kernel generation branch network to obtain the convolution kernel corresponding to the sample video.

### Claim 15 (depends on 12)

The electronic device according to claim 12, wherein the convolution kernel generation branch network comprises a global average pooling layer and two fully connected layers.

### Claim 16 (depends on 11)

The electronic device according to claim 11, wherein the dynamic segment fusion module comprises at least one dynamic segment fusion module, the video recognition model comprises the at least one dynamic segment fusion module and a plurality of residual layers, and the at least one dynamic segment fusion module is arranged in the plurality of residual layers and at an interval of a residual layer.

### Claim 17 (depends on 11)

The electronic device according to claim 11, wherein the dividing the sample video into the plurality of sample video segments comprises:
evenly dividing the sample video according to a length of the sample video to obtain the plurality of sample video segments,
wherein sampling the part of sample video frames from the sample video segment comprises:
sampling video frames from the sample video segment at equal intervals to obtain the part of sample video frames.

### Claim 18 (depends on 11)

The electronic device according to claim 11, wherein the performing the parameter adjustment based on the difference between the tag of the true category and the estimated category to obtain the video recognition model comprises:
calculating a cross entropy loss based on the tag of the true category and the estimated category; and
optimizing the cross entropy loss by using a stochastic gradient descent and continuously updating parameters until the cross entropy loss converges to obtain the video recognition model.

### Claim 19 (depends on 11)

The electronic device according to claim 11, comprising:
acquiring a to-be-recognized video;
dividing the to-be-recognized video into a plurality of to-be-recognized video segments;
sampling a part of to-be-recognized video frames from a to-be-recognized video segment; and
inputting the part of to-be-recognized video frames into the video recognition model to obtain a category of the to-be-recognized video.

### Claim 20

A non-transitory computer readable storage medium, storing a computer instruction, wherein the computer instruction when executed by a computer causes the computer to perform operations comprising:
dividing a sample video into a plurality of sample video segments, wherein the sample video is labeled with a tag of a true category;
sampling a part of sample video frames from a sample video segment of the plurality of sample video segments;
inputting the part of sample video frames into a feature extraction network to obtain feature information of the sample video segment;
performing convolution fusion on the feature information by using a dynamic segment fusion module to obtain fusion feature information, wherein a convolution kernel of the dynamic segment fusion module varies with different video inputs;
inputting the fusion feature information to a fully connected layer to obtain an estimated category of the sample video; and
performing a parameter adjustment based on a difference between the tag of the true category and the estimated category to obtain a video recognition model.
