# US 12,183,076 B2 · Method and apparatus for extracting video segment, device and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,076 B2 - Document ID: US12183076B2 - Status: Granted - Kind: B2 - Application number: 17/625,696 - Filed: 2019-12-30 - Published: 2022-09-01 - Granted: 2024-12-31 - Earliest claimed: 2019-07-30 - Assignees: Tencent Music Entertainment Technology (Shenzhen) Co., Ltd. - Applicants: Tencent Music Entertainment Technology (Shenzhen) Co., Ltd. - Inventors: Yue Yang; Zhi Dong; Shenyuan Li - Web page: https://patentgrep.com/patent/US12183076B2 ## CPC - G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes} - G06V 10/56: relating to colour - G06V 10/762: using clustering, e.g. of similar faces in social networks - G06V 10/82: using neural networks - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 20/635: {Overlay text, e.g. embedded captions in a TV programme} - G06V 40/161: {Detection; Localisation; Normalisation} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Provided is a method for extracting a video segment, including: acquiring a boundary value of content of a video, wherein the boundary value includes an upper boundary, a lower boundary, a left boundary, and a right boundary; acquiring a plurality of first segments by performing key frame segmentation on the video; detecting an upper boundary of subtitles in each of the plurality of first segments; detecting a face position in each of the plurality of first segments; selecting, from the plurality of first segments, a second segment in which the face position satisfies a preset condition; and acquiring a third segment without subtitles by cropping the second segment based on an upper boundary of subtitles in the second segment. ## Claims ### Claim 1 A method for extracting a video segment, comprising: acquiring a boundary value of content of a video, wherein the boundary value comprises an upper boundary, a lower boundary, a left boundary, and a right boundary; acquiring a plurality of first segments by performing key frame segmentation on the video; detecting an upper boundary of subtitles in each of the plurality of first segments; detecting a face position in each of the plurality of first segments; selecting, from the plurality of first segments, a second segment in which the face position satisfies a preset condition; and acquiring a third segment without subtitles by cropping the second segment based on an upper boundary of subtitles in the second segment. ### Claim 2 (depends on 1) The method according to claim 1, wherein detecting the upper boundary of the subtitles in each of the plurality of first segments comprises: extracting a plurality of video frames from each of the plurality of first segments; acquiring subtitle positions in the plurality of video frames by inputting the plurality of video frames into a subtitle detection model, wherein the subtitle detection model is configured to detect a subtitle position in a video frame; and acquiring the upper boundary of the subtitles in the first segment based on the subtitle positions in the plurality of video frames. ### Claim 3 (depends on 2) The method according to claim 2, wherein extracting the plurality of video frames from each of the plurality of first segments comprises: acquiring a time point of each line of lyrics in the video from a lyric file of the video; and extracting a video frame corresponding to the time point from the first segment. ### Claim 4 (depends on 2) The method according to claim 2, wherein upon acquiring the subtitle positions in the plurality of video frames by inputting the plurality of video frames into the subtitle detection model, the method further comprises: acquiring a plurality of clusters by clustering the subtitle positions in the plurality of video frames; and deleting, from the subtitle positions in the plurality of video frames, subtitle positions that do not belong to any one of the clusters. ### Claim 5 (depends on 2) The method according to claim 2, wherein prior to acquiring the subtitle positions in the plurality of video frames by inputting the plurality of video frames into the subtitle detection model, the method further comprises: acquiring a first sample image and characters; generating a second sample image based on the first sample image and the characters, wherein the second sample image contains subtitles; and acquiring the subtitle detection model by training a model by using the second sample image. ### Claim 6 (depends on 1) The method according to claim 1, wherein selecting, from the plurality of first segments, the second segment in which the face position satisfies the preset condition comprises at least one of: acquiring a face position score based on the face position in the first segment and selecting the second segment based on the face position score, wherein the face position score represents a centering degree of a face position in a video frame; or acquiring a face size score based on the face position in the first segment and selecting the second segment based on the face size score, wherein the face size score represents an appropriateness degree of a face size relative to a size of the video frame. ### Claim 7 (depends on 6) The method according to claim 6, wherein selecting, from the plurality of first segments, the second segment in which the face position satisfies the preset condition comprises any one of: acquiring a score of the first segment based on the face position score and the face size score and selecting a first segment of which the score is greater than a first threshold as the second segment; selecting a first segment of which the face position score is greater than a second threshold as the second segment; and selecting a first segment of which the face size score is greater than a third threshold as the second segment. ### Claim 8 (depends on 1) The method according to claim 1, wherein acquiring the plurality of first segments by performing the key frame segmentation on the video comprises: detecting key frames in the video, wherein the key frame is a video frame in which a shot change occurs; and acquiring the plurality of first segments by segmenting the video using the key frames as segmentation points. ### Claim 9 (depends on 8) The method according to claim 8, wherein detecting the key frames of the video comprises: acquiring a color histogram of each of a plurality of video frames of the video; classifying the plurality of video frames into a plurality of types based on the color histogram of each of the plurality of video frames, wherein a difference between color histograms of any two video frames of a same type is less than a preset threshold; and selecting a first frame of each type in the plurality of types as a key frame corresponding to the type. ### Claim 10 (depends on 9) The method according to claim 9, wherein acquiring the color histogram of each of the plurality of video frames of the video comprises: acquiring a first histogram of a first region in the video frame, wherein the first region is a region in a center of a view field of the video frame; acquiring a second histogram of a second region in the video frame, wherein the second region is a region at an edge of the view field of the video frame; and acquiring the color histogram by performing weighted average on the first histogram and the second histogram based on a first weight of the first region and a second weight of the second region, wherein the first weight is greater than the second weight. ### Claim 11 (depends on 1) The method according to claim 1, wherein prior to performing the key frame segmentation on the video, the method further comprises: deleting a border of the video based on the boundary value. ### Claim 12 (depends on 1) The method according to claim 1, wherein acquiring the boundary value of the content of the video comprises: traversing any video frame of the video from a center of the video frame to a periphery of the video frame; and in response to value differences between different pixels in any currently traversed row falling within a preset range, determining the row as an upper or lower boundary of the content of the video frame; and/or in response to value differences between different pixels in any currently traversed column falling within the preset range, determining the column as a left or right boundary of the content of the video frame. ### Claim 13 (depends on 1) The method according to claim 1, wherein acquiring the boundary value of the content of the video comprises: acquiring boundary values of content of a plurality of video frames of the video; acquiring a plurality of clusters by clustering the boundary values of the content of the plurality of video frames; and deleting, from the boundary values of the content of the plurality of video frames, boundary values that do not belong to any cluster. ### Claim 14 A computer device, comprising at least one processor and at least one memory storing at least one instruction, wherein the at least one processor, when loading and executing the at least one instruction, is caused to perform: acquiring a boundary value of content of a video, wherein the boundary value comprises an upper boundary, a lower boundary, a left boundary, and a right boundary; acquiring a plurality of first segments by performing key frame segmentation on the video; detecting an upper boundary of subtitles in each of the plurality of first segments; detecting a face position in each of the plurality of first segments; selecting, from the plurality of first segments, a second segment in which the face position satisfies a preset condition; and acquiring a third segment without subtitles by cropping the second segment based on an upper boundary of subtitles in the second segment. ### Claim 15 (depends on 14) The computer device according to claim 14, wherein the at least one processor, when loading and executing the at least one instruction, is caused to perform: extracting a plurality of video frames from each of the plurality of first segments; acquiring subtitle positions in the plurality of video frames by inputting the plurality of video frames into a subtitle detection model, wherein the subtitle detection model is configured to detect a subtitle position in a video frame; and acquiring the upper boundary of the subtitles in the first segment based on the subtitle positions in the plurality of video frames. ### Claim 16 (depends on 14) The computer device according to claim 14, wherein the at least one processor, when loading and executing the at least one instruction, is caused to perform at least one of: acquiring a face position score based on the face position in the first segment and selecting the second segment based on the face position score, wherein the face position score represents a centering degree of a face position in a video frame; or acquiring a face size score based on the face position in the first segment and selecting the second segment based on the face size score, wherein the face size score represents an appropriateness degree of a face size relative to a size of the video frame. ### Claim 17 (depends on 14) The computer device according to claim 14, wherein the at least one processor, when loading and executing the at least one instruction, is caused to perform: detecting key frames in the video, wherein the key frame is a video frame in which a shot change occurs; and acquiring the plurality of first segments by segmenting the video using the key frames as segmentation points. ### Claim 18 (depends on 14) The computer device according to claim 14, wherein the at least one processor, when loading and executing the at least one instruction, is caused to perform: deleting a border of the video based on the boundary value. ### Claim 19 (depends on 14) The computer device according to claim 14, wherein the at least one processor, when loading and executing the at least one instruction, is caused to perform: traversing any video frame of the video from a center of the video frame to a periphery of the video frame; and in response to value differences between different pixels in any currently traversed row falling within a preset range, determining the row as an upper or lower boundary of the content of the video frame; and/or in response to value differences between different pixels in any currently traversed column falling within the preset range, determining the column as a left or right boundary of the content of the video frame. ### Claim 20 A non-transitory computer-readable storage medium storing at least one instruction, wherein the at least one instruction, when loaded and executed by a processor, causes the processor to perform: acquiring a boundary value of content of a video, wherein the boundary value comprises an upper boundary, a lower boundary, a left boundary, and a right boundary; acquiring a plurality of first segments by performing key frame segmentation on the video; detecting an upper boundary of subtitles in each of the plurality of first segments; detecting a face position in each of the plurality of first segments; selecting, from the plurality of first segments, a second segment in which the face position satisfies a preset condition; and acquiring a third segment without subtitles by cropping the second segment based on an upper boundary of subtitles in the second segment.