# US 12,111,866 B2 · Term weight generation method, apparatus, device and medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,111,866 B2 - Document ID: US12111866B2 - Status: Granted - Kind: B2 - Application number: 17/975,519 - Filed: 2022-10-27 - Published: 2023-02-23 - Granted: 2024-10-08 - Earliest claimed: 2021-03-09 - Assignees: Tencent Technology (Shenzhen) Company Limited - Applicants: Tencent Technology (Shenzhen) Company Limited - Inventors: Jianhui Huang - Web page: https://patentgrep.com/patent/US12111866B2 ## CPC - G06F 16/7844: {using original textual content or text extracted from visual content or transcript of audio data} - G06F 16/71: Indexing; Data structures therefor; Storage structures - G06V 10/761: {Proximity, similarity or dissimilarity measures} - G06V 10/764: using classification, e.g. of video objects - G06V 10/806: {of extracted features} - G06V 10/82: using neural networks - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 20/635: {Overlay text, e.g. embedded captions in a TV programme} - G06V 30/153: {using recognition of characters or words} - G06V 30/18: Extraction of features or characteristics of the image - G06V 30/1918: {Fusion techniques, i.e. combining data from various sources, e.g. sensor fusion} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A term weight determination method includes: obtaining a video and video-associated text, the video-associated text including at least one term; generating a halfway vector of the term by performing multimodal feature fusion on the features of the video, the video-associated text and the at least one term; and generating the weight of the at least one term based on the halfway vector of the at least one term. ## Claims ### Claim 1 A term weight generation method performed by a computer device, comprising: obtaining a video and video-associated text, the video-associated text comprising at least one term, and the video-associated text being text information associated with the content of the video; generating a halfway vector of the at least one term by performing multimodal feature fusion on features of the video, the video-associated text and the at least one term by: extracting a video feature vector of the video; extracting a text feature vector of the video-associated text; segmenting the video-associated text to obtain a plurality of terms, the plurality of terms including the at least one term; extracting, using a deep neural network, a term feature vector of the at least one term; and fusing the video feature vector, the text feature vector and the term feature vector into the halfway vector of the at least one term; generating a weight of the at least one term based on dimension transformation of the halfway vector of the at least one term into a one-dimension parameter to enhance representation of the at least one term for content of the video and the video-associated text; calculating a similarity score between the at least one term and a search term in a video search request submitted by a terminal according to the weight of the at least one term; and returning the video to the requesting terminal when the similarity score is greater than a predefined threshold. ### Claim 2 (depends on 1) The method according to claim 1, wherein fusing the video feature vector, the text feature vector and the term feature vector to obtain the halfway vector of the at least one term comprises: performing a first fusion operation on the video feature vector, the text feature vector and the term feature vector to obtain a first fusion vector; and performing a second fusion operation on the first fusion vector and the term feature vector to obtain the halfway vector of the at least one term. ### Claim 3 (depends on 2) The method according to claim 2, wherein performing the first fusion operation on the video feature vector, the text feature vector and the term feature vector to obtain a first fusion vector comprises: sequentially splicing the video feature vector, the text feature vector and the term feature vector to obtain a first splicing vector; and performing fully connected feature mapping on the first splicing vector to obtain the first fusion vector. ### Claim 4 (depends on 2) The method according to claim 2, wherein performing the second fusion operation on the first fusion vector and the term feature vector to obtain the halfway vector of the at least one term comprises: sequentially splicing the first fusion vector and the term feature vector to obtain a second splicing vector; and performing fully connected feature mapping on the second splicing vector to obtain the halfway vector of the at least one term. ### Claim 5 (depends on 1) The method according to claim 1, wherein the weight of the at least one term is obtained by normalizing the one-dimension parameter into a value between zero and 1 using a threshold function. ### Claim 6 (depends on 1) The method according to claim 1, wherein extracting the video feature vector of the video comprises: performing a framing operation on the video to obtain at least two video frames; extracting video frame vectors of the at least two video frames; and computing an average vector of the video frame vectors of the at least two video frames, or computing a weighted vector of the video frame vectors of the at least two video frames; and setting the average vector or the weighted vector as the video feature vector. ### Claim 7 (depends on 6) The method according to claim 6, wherein computing the weighted vector of the video frame vectors of the at least two video frames, and setting the weighted vector as the video feature vector comprises: determining, using a target detection model, a target object included in one or more video frames; classifying, using a classification model, the target object to obtain the target object classification corresponding to the one or more video frames; computing a similarity score between the target object classification and a respective term of each video frame; determining a weight of the video frame vector of each video frame according to the similarity score corresponding to each video frame, wherein the weight is positively correlated to the similarity score; and computing the weighted vector of the video frame vectors of the at least two video frames based on the video frame vectors of the at least two video frames and the respective weights of the video frame vectors of the at least two video frames, and setting the weighted vector as the video feature vector. ### Claim 8 (depends on 6) The method according to claim 6, wherein extracting the video frame vectors of the at least two video frames comprises: using a residual network to extract the video frame vectors of the at least two video frames in the video. ### Claim 9 (depends on 6) The method according to claim 6, wherein extracting the video feature vector of the video comprises: extracting a video frame vector based on the video frame in the video; extracting an audio frame vector based on an audio frame in the video; extracting a text screen vector based on text in the video frame; and fusing at least two of the video frame vector, the audio frame vector, and the text screen vector to obtain the video feature vector. ### Claim 10 (depends on 1) The method according to claim 1, wherein extracting the text feature vector of the video-associated text comprises: using a bidirectional encoder representation from transformers (BERT) network or a long short-term memory (LSTM) network to extract the text feature vector of the video-associated text. ### Claim 11 (depends on 1) The method according to claim 1, wherein segmenting the video-associated text to obtain the plurality of terms comprises: segmenting, using a Chinese term segmentation tool, the video-associated text to obtain the plurality of terms. ### Claim 12 An electronic device, comprising: one or more processors; and memory storing one or more programs, the one or more programs comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: obtaining a video and video-associated text, the video-associated text comprising at least one term, and the video-associated text being text information associated with the content of the video; generating a halfway vector of the at least one term by performing multimodal feature fusion on features of the video, the video-associated text and the at least one term by: extracting a video feature vector of the video; extracting a text feature vector of the video-associated text; segmenting the video-associated text to obtain a plurality of terms, the plurality of terms including the at least one term; extracting, using a deep neural network, a term feature vector of the at least one term; and fusing the video feature vector, the text feature vector and the term feature vector into the halfway vector of the at least one term; generating a weight of the at least one term based on dimension transformation of the halfway vector of the at least one term into a one-dimension parameter to enhance representation of the at least one term for content of the video and the video-associated text; calculating a similarity score between the at least one term and a search term in a video search request submitted by a terminal according to the weight of the at least one term; and returning the video to the requesting terminal when the similarity score is greater than a predefined threshold. ### Claim 13 (depends on 12) The electronic device according to claim 12, wherein fusing the video feature vector, the text feature vector and the term feature vector to obtain the halfway vector of the at least one term comprises: performing a first fusion operation on the video feature vector, the text feature vector and the term feature vector to obtain a first fusion vector; and performing a second fusion operation on the first fusion vector and the term feature vector to obtain the halfway vector of the at least one term. ### Claim 14 (depends on 13) The electronic device according to claim 13, wherein performing the first fusion operation on the video feature vector, the text feature vector and the term feature vector to obtain a first fusion vector comprises: sequentially splicing the video feature vector, the text feature vector and the term feature vector to obtain a first splicing vector; and performing fully connected feature mapping on the first splicing vector to obtain the first fusion vector. ### Claim 15 (depends on 12) The electronic device according to claim 12, wherein fusing the video feature vector, the text feature vector and the term feature vector to obtain the halfway vector of the at least one term comprises: performing a first fusion operation on the video feature vector, the text feature vector and the term feature vector to obtain a first fusion vector; and performing a second fusion operation on the first fusion vector and the term feature vector to obtain the halfway vector of the at least one term. ### Claim 16 (depends on 12) The electronic device according to claim 12, wherein the weight of the at least one term is obtained by normalizing the one-dimension parameter into a value between zero and 1 using a threshold function. ### Claim 17 A non-transitory computer-readable storage medium, storing a computer program, the computer program, when executed by one or more processors of an electronic device, cause the one or more processors to perform operations comprising: obtaining a video and video-associated text, the video-associated text comprising at least one term, and the video-associated text being text information associated with the content of the video; generating a halfway vector of the at least one term by performing multimodal feature fusion on features of the video, the video-associated text and the at least one term by: extracting a video feature vector of the video; extracting a text feature vector of the video-associated text; segmenting the video-associated text to obtain a plurality of terms, the plurality of terms including the at least one term; extracting, using a deep neural network, a term feature vector of the at least one term; and fusing the video feature vector, the text feature vector and the term feature vector into the halfway vector of the at least one term; generating a weight of the at least one term based on dimension transformation of the halfway vector of the at least one term into a one-dimension parameter to enhance representation of the at least one term for content of the video and the video-associated text; calculating a similarity score between the at least one term and a search term in a video search request submitted by a terminal according to the weight of the at least one term; and returning the video to the requesting terminal when the similarity score is greater than a predefined threshold. ### Claim 18 (depends on 17) The non-transitory computer-readable storage medium according to claim 17, wherein fusing the video feature vector, the text feature vector and the term feature vector to obtain the halfway vector of the at least one term comprises: performing a first fusion operation on the video feature vector, the text feature vector and the term feature vector to obtain a first fusion vector; and performing a second fusion operation on the first fusion vector and the term feature vector to obtain the halfway vector of the at least one term. ### Claim 19 (depends on 17) The non-transitory computer-readable storage medium according to claim 17, wherein the weight of the at least one term is obtained by normalizing the one-dimension parameter into a value between zero and 1 using a threshold function.