# US 12,008,810 B2 · Video sequence selection method, computer device, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,810 B2
- Document ID: US12008810B2
- Status: Granted
- Kind: B2
- Application number: 17/225,969
- Filed: 2021-04-08
- Published: 2021-07-22
- Granted: 2024-06-11
- Earliest claimed: 2019-03-05
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Zhenfang Chen; Lin Ma; Wenhan Luo; Wei Liu
- Web page: https://patentgrep.com/patent/US12008810B2

## CPC

- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 40/279: Recognition of textual entities
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/758: {Involving statistics of pixels or of feature values, e.g. histogram matching}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06F 40/103: Formatting, i.e. changing of presentation of documents (automatic justification G06F40/189; automatic line break hyphenation G06F40/191)
- Y02T 10/40: Engine management systems
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- Y02T: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION

## Abstract

This application discloses a video sequence selection method, applicable to a computer device, the method including: receiving a to-be-matched video and a to-be-matched text, the to-be-matched text having a to-be-matched text feature sequence; invoking a spatiotemporal candidate region generator to extract a spatiotemporal candidate region set from the to-be-matched video, the spatiotemporal candidate region set including N spatiotemporal candidate regions; performing feature extraction on each spatiotemporal candidate region by using a convolutional neural network, to obtain N to-be-matched video feature sequences; invoking an attention-based interactor to obtain a matching score corresponding to each spatiotemporal candidate region, the matching score being used for representing a matching relationship between the spatiotemporal candidate region and the to-be-matched text; and selecting a target spatiotemporal candidate region from the spatiotemporal candidate region set according to the matching score corresponding to each spatiotemporal candidate region, and outputting the target spatiotemporal candidate region. In this application, an association between the video and the text in time sequence is considered during matching, thereby increasing a degree of matching between a video sequence and the text.

## Claims

### Claim 1

A video sequence selection method, applicable to a computer device, the method comprising:
receiving, by the computer device, a to-be-matched video and a to-be-matched text, wherein the to-be-matched text is not part of the to-be-matched video, the to-be-matched video comprising a plurality of frames, the to-be-matched text comprising at least one word, and the to-be-matched text having a to-be-matched text feature sequence corresponding to a target object;
invoking, by the computer device, a spatiotemporal candidate region generator to extract a spatiotemporal candidate region set from the to-be-matched video, the spatiotemporal candidate region set comprising N spatiotemporal candidate regions, N being an integer greater than or equal to 1, and each spatiotemporal candidate region corresponding to images within a respective video sequence in the to-be-matched video that include a candidate object;
performing, by the computer device, feature extraction on each spatiotemporal candidate region in the spatiotemporal candidate region set by using a convolutional neural network, to obtain N to-be-matched video feature sequences, each to-be-matched video feature sequence corresponding to a respective spatiotemporal candidate region in the spatiotemporal candidate region set and representing a respective candidate object in the respective spatiotemporal candidate region;
invoking, by the computer device, an attention-based interactor to obtain a matching score corresponding to each spatiotemporal candidate region, the interactor being configured to process the to-be-matched video feature sequence and the to-be-matched text feature sequence, and the matching score being used for representing a matching relationship between a respective candidate object in the spatiotemporal candidate region and the target object corresponding to the to-be-matched text; and
selecting, by the computer device, from the spatiotemporal candidate region set, a target spatiotemporal candidate region having a highest matching score outputted by the interactor, and outputting the target spatiotemporal candidate region as representing the target object corresponding to the to-be-matched text.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the invoking, by the computer device, a spatiotemporal candidate region generator to extract a spatiotemporal candidate region set from the to-be-matched video comprises:
invoking, by the computer device, the spatiotemporal candidate region generator to obtain a candidate region and a confidence score of each frame in the to-be-matched video, each candidate region corresponding to a respective confidence score;
invoking, by the computer device, the spatiotemporal candidate region generator to obtain a degree of overlap of similar image content between every two adjacent frames in the to-be-matched video; and
invoking, by the computer device, the spatiotemporal candidate region generator to generate the spatiotemporal candidate region set according to the candidate region and the confidence score of each frame and the overlap degrees.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the invoking, by the computer device, an attention-based interactor to obtain a matching score corresponding to each spatiotemporal candidate region comprises:
invoking, by the computer device for each spatiotemporal candidate region, an encoder of the interactor to encode the to-be-matched video feature sequence corresponding to the spatiotemporal candidate region, to obtain a visual feature set, the visual feature set comprising at least one visual feature of a candidate object in the spatiotemporal candidate region;
invoking, by the computer device, the encoder of the interactor to encode the to-be-matched text feature sequence, to obtain a text feature set, the text feature set comprising at least one text feature of the target object;
invoking, by the computer device, the interactor to determine a visual text feature set according to the visual feature set and the text feature set, the visual text feature set comprising at least one visual text feature, the visual text feature representing a visual feature-based text feature; and
invoking, by the computer device, the interactor to determine the matching score corresponding to the candidate object in the spatiotemporal candidate region and the target object according to the visual text feature set and the visual feature set.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the invoking, by the computer device, an encoder of the interactor to encode the to-be-matched video feature sequence corresponding to the spatiotemporal candidate region, to obtain a visual feature set comprises:
calculating the visual feature set in the following manner:
Hp={htp}t=1tp, and
htp=LSTMp(ftp,ht-1p),
Hp representing the visual feature set, htp representing a tth visual feature in the visual feature set, tp representing a time step in the spatiotemporal candidate region, ht-1p representing a (t−1)th visual feature in the visual feature set, LSTMp( ) representing a first long short-term memory (LSTM) network encoder, and ftp representing a tth row of features in the to-be-matched video feature sequence; and
the invoking, by the computer device, the encoder of the interactor to encode the to-be-matched text feature sequence, to obtain a text feature set comprises:
calculating the text feature set in the following manner:
Hq={htq}t=1tq, and
htq=LSTMq(ftq,ht-1q),
Hq representing the text feature set, htq representing a tth text feature in the text feature set, tq representing a word quantity of the to-be-matched text, ht-1q representing a (t−1)th text feature in the text feature set, LSTMq( ) representing a second LSTM encoder, and ftq representing a tth row of features in the to-be-matched text feature sequence.

### Claim 5 (depends on 3)

The method according to claim 3, wherein the invoking, by the computer device, the interactor to determine a visual text feature set according to the visual feature set and the text feature set comprises:
invoking, by the computer device, the interactor to calculate an attention weight of the text feature corresponding to the visual feature according to the visual feature set and the text feature set;
invoking, by the computer device, the interactor to calculate a normalized attention weight of the text feature corresponding to the visual feature according to the attention weight; and
invoking, by the computer device, the interactor to calculate the visual text feature set according to the normalized attention weight and the text feature.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the invoking, by the computer device, the interactor to calculate an attention weight of the text feature corresponding to the visual feature according to the visual feature set and the text feature set comprises:
calculate the attention weight in the following manner:
ei,j=wT tanh(Wqhjq+Wphip+b1)+b2;
ei,j representing an attention weight of a jth text feature corresponding to an ith visual feature, hjq representing the jth text feature, hip representing the ith visual feature, WT representing a first model parameter, Wq representing a second model parameter, Wp representing a third model parameter, b1 representing a fourth model parameter, b2 representing a fifth model parameter, and tanh( ) representing a hyperbolic tangent function;
the invoking, by the computer device, the interactor to calculate a normalized attention weight of the text feature corresponding to the visual feature according to the attention weight comprises:
calculate the normalized attention weight in the following manner:
a
i
,
j
=
exp
⁡
(
e
i
,
j
)
∑
k
=
1
t
q
⁢
exp
⁡
(
e
i
,
k
)
,
ai,j representing a normalized attention weight of the jth text feature corresponding to the ith visual feature, tq representing a word quantity of the to-be-matched text, k representing a kth word in the to-be-matched text, k being an integer greater than or equal to 1 and less than or equal to tq, and exp( ) representing an exponential function; and
the invoking, by the computer device, the interactor to calculate the visual text feature set according to the normalized attention weight and the text feature comprises:
calculating the visual text feature set in the following manner:
Hqp={hqp}t=1tp; and
h
q
⁢
p
=
∑
j
=
1
t
q
a
i
,
j
⁢
h
j
q
,
Hqp representing the visual text feature set, tp representing a time step of the spatiotemporal candidate region, and hqp representing the visual text feature.

### Claim 7 (depends on 3)

The method according to claim 3, wherein the invoking, by the computer device, the interactor to determine the matching score corresponding to the spatiotemporal candidate region according to the visual text feature set and the visual feature set comprises:
calculating the matching score in the following manner:
s
⁡
(
q
,
p
)
=
1
t
p
⁢
∑
i
=
1
t
p
s
i
(
h
i
p
,
h
i
q
⁢
p
)
;
and
si(hip,hiqp)=ϕ(hip,hiqp),
s(q, p) representing the matching score corresponding to the spatiotemporal candidate region, si(hip, hiqp) representing a matching subscore between a visual feature and a visual text feature that correspond to an ith time step, hiqp representing the visual text feature corresponding to the ith time step, hip representing the visual feature corresponding to the ith time step, and ϕ( ) representing a similarity calculation function.

### Claim 8 (depends on 1)

The method according to claim 1, further comprising:
obtaining, by the computer device, a first to-be-trained video, a second to-be-trained video, a first to-be-trained text, and a second to-be-trained text, the first to-be-trained video having a matching relationship with the first to-be-trained text, the first to-be-trained video having no matching relationship with the second to-be-trained text, the second to-be-trained video having a matching relationship with the second to-be-trained text, and the second to-be-trained video having no matching relationship with the first to-be-trained text;
determining, by the computer device, a permutation loss function according to the first to-be-trained video, the second to-be-trained video, the first to-be-trained text, and the second to-be-trained text, the permutation loss function being configured to process the first to-be-trained video and the second to-be-trained text and process the second to-be-trained video and the first to-be-trained text;
determining, by the computer device, a diversity loss function according to the first to-be-trained video, the second to-be-trained video, the first to-be-trained text, and the second to-be-trained text, the diversity loss function being configured to process the first to-be-trained video and the first to-be-trained text and process the second to-be-trained video and the second to-be-trained text;
determining, by the computer device, a target loss function according to the permutation loss function and the diversity loss function; and
training, by the computer device, a to-be-trained interactor by using the target loss function, to obtain the interactor, the interactor being configured to output a matching score between the to-be-matched video and the to-be-matched text.

### Claim 9 (depends on 8)

The method according to claim 8, wherein the determining, by the computer device, a permutation loss function according to the first to-be-trained video, the second to-be-trained video, the first to-be-trained text, and the second to-be-trained text comprises:
obtaining, by the computer device, a first spatiotemporal candidate region set in the first to-be-trained video, and obtaining a second spatiotemporal candidate region set in the second to-be-trained video, the first spatiotemporal candidate region set comprising at least one first spatiotemporal candidate region, the first spatiotemporal candidate region being a video sequence, the second spatiotemporal candidate region set comprising at least one second spatiotemporal candidate region, and the second spatiotemporal candidate region being a video sequence;
calculating, by the computer device, a first matching score according to the first to-be-trained text and the second spatiotemporal candidate region set;
calculating, by the computer device, a second matching score according to the second to-be-trained text and the first spatiotemporal candidate region set;
calculating, by the computer device, a third matching score according to the first to-be-trained text and the first spatiotemporal candidate region set; and
determining, by the computer device, the permutation loss function according to the first matching score, the second matching score, and the third matching score.

### Claim 10 (depends on 8)

The method according to claim 8, wherein the determining, by the computer device, a diversity loss function according to the first to-be-trained video, the second to-be-trained video, the first to-be-trained text, and the second to-be-trained text comprises:
determining, by the computer device, a matching behavior distribution according to a first spatiotemporal candidate region set and the first to-be-trained text, the first spatiotemporal candidate region set being generated according to the first to-be-trained video, and the matching behavior distribution representing a matching relationship between each first spatiotemporal candidate region in the first spatiotemporal candidate region set and the first to-be-trained text;
performing, by the computer device, normalization processing on the matching behavior distribution, to obtain a target matching behavior distribution; and
determining, by the computer device, the diversity loss function according to the target matching behavior distribution.

### Claim 11 (depends on 8)

The method according to claim 8, wherein the determining, by the computer device, a target loss function according to the permutation loss function and the diversity loss function comprises:
obtaining, by the computer device, a control coefficient, and determining the target loss function according to the control coefficient, the permutation loss function, and the diversity loss function.

### Claim 12

A computer device, comprising: a memory, a processor, and a bus system connected to the memory and the processor, and a plurality of instructions stored in the memory that, when executed by the processor, cause the computer device to perform a plurality of operations including:
receiving, by the computer device, a to-be-matched video and a to-be-matched text, wherein the to-be-matched text is not part of the to-be-matched video, the to-be-matched video comprising a plurality of frames, the to-be-matched text comprising at least one word, and the to-be-matched text having a to-be-matched text feature sequence corresponding to a target object;
invoking, by the computer device, a spatiotemporal candidate region generator to extract a spatiotemporal candidate region set from the to-be-matched video, the spatiotemporal candidate region set comprising N spatiotemporal candidate regions, N being an integer greater than or equal to 1, and each spatiotemporal candidate region corresponding to images within a respective video sequence in the to-be-matched video that include a candidate object;
performing, by the computer device, feature extraction on each spatiotemporal candidate region in the spatiotemporal candidate region set by using a convolutional neural network, to obtain N to-be-matched video feature sequences, each to-be-matched video feature sequence corresponding to a respective spatiotemporal candidate region in the spatiotemporal candidate region set and representing a respective candidate object in the respective spatiotemporal candidate region;
invoking, by the computer device, an attention-based interactor to obtain a matching score corresponding to each spatiotemporal candidate region, the interactor being configured to process the to-be-matched video feature sequence and the to-be-matched text feature sequence, and the matching score being used for representing a matching relationship between a respective candidate object in the spatiotemporal candidate region and the target object corresponding to the to-be-matched text; and
selecting, by the computer device, from the spatiotemporal candidate region set, a target spatiotemporal candidate region having a highest matching score outputted by the interactor, and outputting the target spatiotemporal candidate region as representing the target object corresponding to the to-be-matched text.

### Claim 13 (depends on 12)

The computer device according to claim 12, wherein the invoking a spatiotemporal candidate region generator to extract a spatiotemporal candidate region set from the to-be-matched video comprises:
invoking the spatiotemporal candidate region generator to obtain a candidate region and a confidence score of each frame in the to-be-matched video, each candidate region corresponding to a respective confidence score;
invoking the spatiotemporal candidate region generator to obtain a degree of overlap of similar image content between every two adjacent frames in the to-be-matched video; and
invoking the spatiotemporal candidate region generator to generate the spatiotemporal candidate region set according to the candidate region and the confidence score of each frame and the overlap degrees.

### Claim 14 (depends on 12)

The computer device according to claim 12, wherein the invoking an attention-based interactor to obtain a matching score corresponding to each spatiotemporal candidate region comprises:
invoking, by the computer device for each spatiotemporal candidate region, an encoder of the interactor to encode the to-be-matched video feature sequence corresponding to the spatiotemporal candidate region, to obtain a visual feature set, the visual feature set comprising at least one visual feature of a candidate object in the spatiotemporal candidate region;
invoking, by the computer device, the encoder of the interactor to encode the to-be-matched text feature sequence, to obtain a text feature set, the text feature set comprising at least one text feature of the target object;
invoking, by the computer device, the interactor to determine a visual text feature set according to the visual feature set and the text feature set, the visual text feature set comprising at least one visual text feature, the visual text feature representing a visual feature-based text feature; and
invoking, by the computer device, the interactor to determine the matching score corresponding to the candidate object in the spatiotemporal candidate region and the target object according to the visual text feature set and the visual feature set.

### Claim 15 (depends on 14)

The computer device according to claim 14, wherein the invoking an encoder of the interactor to encode the to-be-matched video feature sequence corresponding to the spatiotemporal candidate region, to obtain a visual feature set comprises:
calculating the visual feature set in the following manner:
Hp={htp}t=1tp, and
htp=LSTMp(ftp,ht-1p),
Hp representing the visual feature set, htp representing a tth visual feature in the visual feature set, tp representing a time step in the spatiotemporal candidate region, ht-1p representing a (t−1)th visual feature in the visual feature set, LSTMp( ) representing a first long short-term memory (LSTM) network encoder, and ftp representing a tth row of features in the to-be-matched video feature sequence; and
the invoking the encoder of the interactor to encode the to-be-matched text feature sequence, to obtain a text feature set comprises:
calculating the text feature set in the following manner:
Hq={htq}t=1tq, and
htq=LSTMq(ftq,ht-1q),
Hq representing the text feature set, htq representing a tth text feature in the text feature set, tq representing a word quantity of the to-be-matched text, ht-1q representing a (t−1)th text feature in the text feature set, LSTMq ( ) representing a second LSTM encoder, and ftq representing a tth row of features in the to-be-matched text feature sequence.

### Claim 16 (depends on 14)

The computer device according to claim 14, wherein the invoking the interactor to determine a visual text feature set according to the visual feature set and the text feature set comprises:
invoking the interactor to calculate an attention weight of the text feature corresponding to the visual feature according to the visual feature set and the text feature set;
invoking the interactor to calculate a normalized attention weight of the text feature corresponding to the visual feature according to the attention weight; and
invoking the interactor to calculate the visual text feature set according to the normalized attention weight and the text feature.

### Claim 17 (depends on 12)

The computer device according to claim 12, wherein the plurality of operations further comprise:
obtaining a first to-be-trained video, a second to-be-trained video, a first to-be-trained text, and a second to-be-trained text, the first to-be-trained video having a matching relationship with the first to-be-trained text, the first to-be-trained video having no matching relationship with the second to-be-trained text, the second to-be-trained video having a matching relationship with the second to-be-trained text, and the second to-be-trained video having no matching relationship with the first to-be-trained text;
determining a permutation loss function according to the first to-be-trained video, the second to-be-trained video, the first to-be-trained text, and the second to-be-trained text, the permutation loss function being configured to process the first to-be-trained video and the second to-be-trained text and process the second to-be-trained video and the first to-be-trained text;
determining a diversity loss function according to the first to-be-trained video, the second to-be-trained video, the first to-be-trained text, and the second to-be-trained text, the diversity loss function being configured to process the first to-be-trained video and the first to-be-trained text and process the second to-be-trained video and the second to-be-trained text;
determining a target loss function according to the permutation loss function and the diversity loss function; and
training a to-be-trained interactor by using the target loss function, to obtain the interactor, the interactor being configured to output a matching score between the to-be-matched video and the to-be-matched text.

### Claim 18

A non-transitory computer-readable storage medium, storing instructions, the instructions, when executed by a processor of a computer device, causing the computer device to perform a plurality of operations including:
receiving, by the computer device, a to-be-matched video and a to-be-matched text, wherein the to-be-matched text is not part of the to-be-matched video, the to-be-matched video comprising a plurality of frames, the to-be-matched text comprising at least one word, and the to-be-matched text having a to-be-matched text feature sequence corresponding to a target object;
invoking, by the computer device, a spatiotemporal candidate region generator to extract a spatiotemporal candidate region set from the to-be-matched video, the spatiotemporal candidate region set comprising N spatiotemporal candidate regions, N being an integer greater than or equal to 1, and each spatiotemporal candidate region corresponding to images within a respective video sequence in the to-be-matched video that include a candidate object;
performing, by the computer device, feature extraction on each spatiotemporal candidate region in the spatiotemporal candidate region set by using a convolutional neural network, to obtain N to-be-matched video feature sequences, each to-be-matched video feature sequence corresponding to a respective spatiotemporal candidate region in the spatiotemporal candidate region set and representing a respective candidate object in the respective spatiotemporal candidate region;
invoking, by the computer device, an attention-based interactor to obtain a matching score corresponding to each spatiotemporal candidate region, the interactor being configured to process the to-be-matched video feature sequence and the to-be-matched text feature sequence, and the matching score being used for representing a matching relationship between a respective candidate object in the spatiotemporal candidate region and the target object corresponding to the to-be-matched text; and
selecting, by the computer device, from the spatiotemporal candidate region set, a target spatiotemporal candidate region having a highest matching score outputted by the interactor, and outputting the target spatiotemporal candidate region as representing the target object corresponding to the to-be-matched text.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium according to claim 18, wherein the invoking a spatiotemporal candidate region generator to extract a spatiotemporal candidate region set from the to-be-matched video comprises:
invoking the spatiotemporal candidate region generator to obtain a candidate region and a confidence score of each frame in the to-be-matched video, each candidate region corresponding to a respective confidence score;
invoking, by the computer device, the spatiotemporal candidate region generator to obtain a degree of overlap of similar image content between every two adjacent frames in the to-be-matched video; and
invoking the spatiotemporal candidate region generator to generate the spatiotemporal candidate region set according to the candidate region and the confidence score of each frame and the overlap degrees.

### Claim 20 (depends on 18)

The non-transitory computer-readable storage medium according to claim 18, wherein the invoking an attention-based interactor to obtain a matching score corresponding to each spatiotemporal candidate region comprises:
invoking, by the computer device for each spatiotemporal candidate region, an encoder of the interactor to encode the to-be-matched video feature sequence corresponding to the spatiotemporal candidate region, to obtain a visual feature set, the visual feature set comprising at least one visual feature of a candidate object in the spatiotemporal candidate region;
invoking, by the computer device, the encoder of the interactor to encode the to-be-matched text feature sequence, to obtain a text feature set, the text feature set comprising at least one text feature of the target object;
invoking, by the computer device, the interactor to determine a visual text feature set according to the visual feature set and the text feature set, the visual text feature set comprising at least one visual text feature, the visual text feature representing a visual feature-based text feature; and
invoking, by the computer device, the interactor to determine the matching score corresponding to the candidate object in the spatiotemporal candidate region and the target object according to the visual text feature set and the visual feature set.