# US 11,113,536 B2 · Video identification method, video identification device, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,113,536 B2
- Document ID: US11113536B2
- Status: Granted
- Kind: B2
- Application number: 16/523,124
- Filed: 2019-07-26
- Published: 2020-09-17
- Granted: 2021-09-07
- Earliest claimed: 2019-03-15
- Assignees: Boe Technology Group Co., Ltd.
- Applicants: Boe Technology Group Co., Ltd.
- Inventors: Honghong Jia; Yanzhen Cui
- Web page: https://patentgrep.com/patent/US11113536B2

## CPC

- G06F 16/75: Clustering; Classification
- G06F 18/25: Fusion techniques
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/20: Ensemble learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The present disclosure provides a video identification method, a video identification device and a storage medium. The video identification device extracts an image and an optical flow from a video, classifies the image by using a first machine learning model to obtain a first classification result, classifies the optical flow by using a second machine learning model to obtain a second classification result, wherein a depth of the first machine learning model is larger than a depth of the second machine learning model, and fuses the first classification result and the second classification result to obtain the identification result of the video.

## Claims

### Claim 1

A video identification method, comprising:
extracting an image and an optical flow from a video;
classifying the image by using a first machine learning model to obtain a first classification result, wherein the first machine learning model comprises a first machine learning submodel, a second machine learning submodel, a first fully connected layer, and a first classifier, and the classifying the image comprises:
inputting the image into the first machine learning submodel to obtain first feature information;
inputting the first feature information into the second machine learning submodel to obtain second feature information;
inputting the second feature information into the first fully connected layer to obtain third feature information; and
inputting the third feature information into the first classifier to obtain the first classification result;
classifying the optical flow by using a second machine learning model to obtain a second classification result, wherein a depth of the first machine learning model is larger than a depth of the second machine learning model; and
fusing the first classification result and the second classification result to obtain an identification result of the video.

### Claim 2 (depends on 1)

The identification method of claim 1, wherein the second machine learning submodel comprises a plurality of convolutional layers.

### Claim 3 (depends on 2)

The identification method of claim 2, wherein the number of the convolutional layers is three.

### Claim 4 (depends on 1)

The identification method of claim 1, wherein the first machine learning model comprises a preset number of first machine learning submodels, a second machine learning submodel, a first fully connected layer, and a first classifier, and the classifying the image comprises:
dividing the image into a preset number of information segments that are in one-to-one correspondence with the preset number of first machine learning submodels;
inputting the information segment into the corresponding first machine learning submodel to obtain first feature information;
inputting the first feature information into the second machine learning submodel to obtain second feature information;
inputting the second feature information into the first fully connected layer to obtain third feature information; and
inputting the third feature information into the first classifier to obtain the first classification result.

### Claim 5 (depends on 4)

The identification method of claim 4, wherein the inputting an information segment comprises:
extracting a plurality of information frames from the information segment at preset intervals; and
inputting the plurality of information frames into the corresponding first machine learning submodel.

### Claim 6 (depends on 1)

The identification method of claim 1, wherein the second machine learning model comprises a first machine learning submodel, a second fully connected layer, and a second classifier, and the classifying the optical flow comprises:
inputting the optical flow into the first machine learning submodel to obtain fourth feature information;
inputting the fourth feature information into the second fully connected layer to obtain fifth feature information; and
inputting the fifth feature information into the second classifier to obtain the second classification result.

### Claim 7 (depends on 1)

The identification method of claim 1, wherein the second machine learning model comprises a preset number of first machine learning submodels, a second fully connected layer, and a second classifier, and the classifying the image comprises:
dividing the optical flow into a preset number of information segments which are in one-to-one correspondence to the preset number of first machine learning submodels;
inputting the information segment into a corresponding first machine learning submodel to obtain fourth feature information;
inputting the fourth feature information into the second fully connected layer to obtain fifth feature information; and
inputting the fifth feature information into the second classifier to obtain the second classification result.

### Claim 8 (depends on 7)

The identification method of claim 7, wherein the inputting the information segment comprises:
extracting a plurality of information frames from the information segment at preset intervals; and
inputting the plurality of information frames into the corresponding first machine learning submodel.

### Claim 9 (depends on 1)

A non-transitory computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method of claim 1.

### Claim 10

A video identification device, comprising:
a memory configured to store instructions;
a processor coupled to the memory, wherein based on the instructions stored in the memory, the processor is configured to:
extract an image and an optical flow from a video;
classify the image by using a first machine learning model to obtain a first classification result, wherein the first machine learning model comprises a first machine learning submodel, a second machine learning submodel, a first fully connected layer, and a first classifier; and the processor is configured to:
input the image into the first machine learning submodel to obtain first feature information;
input the first feature information into the second machine learning submodel to obtain second feature information;
input the second feature information into the first fully connected layer to obtain third feature information; and
input the third feature information into the first classifier to obtain the first classification result;
classify the optical flow by using a second machine learning model to obtain a second classification result, wherein a depth of the first machine learning model is larger than a depth of the second machine learning model; and
fuse the first classification result and the second classification result to obtain an identification result of the video.

### Claim 11 (depends on 10)

The identification device of claim 10, wherein the second machine learning submodel comprises a plurality of convolutional layers.

### Claim 12 (depends on 11)

The identification device of claim 11, wherein the number of the convolutional layers is three.

### Claim 13 (depends on 10)

The identification device of claim 10, wherein the first machine learning model comprises a preset number of first machine learning submodels, a second machine learning submodel, a first fully connected layer, and a first classifier, and the processor is configured to:
divide the image into a preset number of information segments that are in one-to-one correspondence with the preset number of first machine learning submodels;
input the information segment into a corresponding first machine learning submodel to obtain first feature information;
input the first feature information into the second machine learning submodel to obtain second feature information;
input the second feature information into the first fully connected layer to obtain third feature information; and
input the third feature information into the first classifier to obtain the first classification result.

### Claim 14 (depends on 13)

The identification device of claim 13, wherein the processor is configured to:
extract a plurality of information frames from the information segment at preset intervals; and
input the plurality of information frames into the corresponding first machine learning submodel.

### Claim 15 (depends on 10)

The identification device of claim 10, wherein the second machine learning model includes a first machine learning submodel, a second fully connected layer, and a second classifier, and the processor is configured to:
input the optical flow into the first machine learning submodel to obtain fourth feature information;
input the fourth feature information into the second fully connected layer to obtain fifth feature information; and
input the fifth feature information into the second classifier to obtain the second classification result.

### Claim 16 (depends on 10)

The identification device of claim 10, wherein the second machine learning model comprises a preset number of first machine learning submodels, a second fully connected layer, and a second classifier, and the processor is configured to:
divide the optical flow into a preset number of information segments which are in one-to-one correspondence to the preset number of first machine learning submodels;
input the information segment into a corresponding first machine learning submodel to obtain fourth feature information;
inputting the fourth feature information into the second fully connected layer to obtain fifth feature information; and
inputting the fifth feature information into the second classifier to obtain the second classification result.

### Claim 17 (depends on 16)

The identification device of claim 16, wherein the processor is configured to:
extract a plurality of information frames from the information segment at preset intervals; and
input the plurality of information frames into the corresponding first machine learning submodel.