# US 2020/0401808 A1 · Method and device for identifying key time point of video, computer apparatus and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401808 A1 - Document ID: US20200401808A1 - Status: Published application - Kind: A1 - Application number: 17/013,953 - Filed: 2020-09-08 - Published: 2020-12-24 - Earliest claimed: 2018-07-18 - Assignees: Tencent Technology (Shenzhen) Company Limited - Applicants: Tencent Technology (Shenzhen) Company Ltd - Inventors: Tao Wu; Xu Yuan Xu; Guo Ping Gong - Web page: https://patentgrep.com/patent/US20200401808A1 ## CPC - G06N 3/08: Learning methods - G06F 18/24: Classification techniques - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06T 5/70: Denoising; Smoothing - G06T 7/11: Region-based segmentation - G06T 7/75: {involving models} - G06V 10/764: using classification, e.g. of video objects - G06V 10/7784: {based on feedback from supervisors} - G06V 10/82: using neural networks - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes} - G06T 2207/10016: Video; Image sequence - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method for recognizing a key time point in a video includes: obtaining at least one video segment by processing each image frame in the video by an image classification model; determining a target video segment in the at least one video segment based on a shot type; obtaining respective locations of a first object and a second object in an image frame of the target video segment by an image detection model; and based on a distance between the location of the first object and the location of the second object in the image frame satisfying a preset condition, determining a time point of the image frame as the key time point of the video. ## Claims ### Claim 1 A method for recognizing a key time point in a video, performed by a computer device, the method comprising: obtaining at least one video segment by processing each image frame in the video by an image classification model, the image classification model being a machine learning model obtained by training according to a first sample image frame marked with a shot type, wherein each of the at least one video segment comprises at least two consecutive image frames in the video and each of the at least one video segment corresponds to one shot type among a plurality of shot types; determining a target video segment in the at least one video segment based on the shot type of the at least one video segment; obtaining a first location of a first object and a second location of a second object in an image frame of the target video segment by an image detection model, the image detection model being a machine learning model obtained by training according to a second sample image frame marked with the first location of the first object and the second location of the second object; and based on a distance between the first location of the first object and the second location of the second object in the image frame satisfying a preset condition, determining a time point of the image frame as the key time point of the video. ### Claim 2 (depends on 1) The method according to claim 1, wherein the obtaining the at least one video segment through the image classification model comprises: inputting each image frame in the video into the image classification model to obtain a model classification result that indicates the shot type for the each image frame in the video; performing a smoothing correction according to the shot type of the each image frame in the video; and dividing the video into the at least one video segment according to the shot type of the each image frame in the video. ### Claim 3 (depends on 2) The method according to claim 2, wherein the performing the smoothing correction according to the shot type of the each image frame in the video comprises: obtaining a first category mode of a first image group and a second category mode of a second image group, the first category mode and the second category mode indicating the shot types of the first image group and second image group with a maximum quantity of corresponding image frames, wherein the first image group includes r image frames prior to a second image frame, the second image group including r image frames subsequent to the second image frame, the second image frame being any one image frame among a plurality of image frames in the video other than the first r frames and the last r frames, where r is an integer greater than or equal to 1; and based on the first category mode and the second category mode being the same, setting the shot type of the second image frame to the shot type of the first image group. ### Claim 4 (depends on 2) The method according to claim 2, wherein the dividing the video into the at least one video segment according to the shot type of the each image frame in the video comprises: dividing the video into at least one temporary video segment according to the shot type of the each image frame in the video such that each of the at least one temporary video segment includes image frames belong to the same shot type, wherein the shot types of two consecutive temporary video segments are different; based on a quantity of image frames in a target temporary video segment being less than a preset quantity threshold, modifying a shot type of each image frame in the target temporary video segment into a shot type of each image frame in a temporary video segment previous to the target temporary video segment, and combining the target temporary video segment with the temporary video segment previous to the target temporary video segment, wherein the target temporary video segment is any one temporary video segment in the at least one temporary video segment other than a first temporary video segment; and using each remaining temporary video segment in the at least one temporary video segment as the at least one video segment. ### Claim 5 (depends on 1) The method according to claim 1, wherein the obtaining the first location of the first object and the second location of the second object in the image frame of the target video segment comprises: inputting each image frame in the target video segment into the image detection model to obtain a model detection result that indicates respective temporary locations of the first object and the second object in the image frame of the target video segment; and performing smoothing correction on the respective temporary locations of the first object and the second object in the image frame of the target video segment to obtain the respective locations of the first object and the second object in the image frame of the target video segment. ### Claim 6 (depends on 5) The method according to claim 5, wherein the performing the smoothing correction on the respective temporary locations of the first object and the second object in the image frame of the target video segment comprises: obtaining temporary locations of a target object in image frames of a third image group and a fourth image group, the target object being any one of the first object and the second object, the third image group including w image frames previous to a third image frame, the fourth image group including w image frames subsequent to the third image frame, the third image frame being any one of a plurality of image frames in the target video segment other than the first w frames and the last w frames, where w is an integer greater than or equal to 1; obtaining an average location that is indicated as an average value of the temporary locations of the target object in the image frames of the third image group and the fourth image group; and correcting the temporary location of the target object in the third image frame according to the average location. ### Claim 7 (depends on 6) The method according to claim 6, wherein the correcting the temporary location of the target object in the third image frame according to the average location comprises: obtaining an offset of the temporary location of the target object in the third image frame relative to the average location; and correcting, in a case that the offset is greater than an offset threshold, the temporary location of the target object in the third image frame to be the average location. ### Claim 8 (depends on 5) The method according to claim 5, wherein the method further comprises: obtaining a template image corresponding to a fourth image frame which is an image frame including a target object and that is not detected by the image detection model, the target object being any one of the first object and the second object, the template image being an image corresponding to a temporary location of the target object in a fifth image frame which is an image frame including the target object and that is detected by the image detection model previous or subsequent to the fourth image frame among the image frames in the target video segment; and performing a template matching in the fourth image frame through the template image, to obtain the temporary location of the target object in the fourth image frame. ### Claim 9 (depends on 1) The method according to claim 1, wherein the determining the time point of the image frame as the key time point of the video comprises: based on the distance between the first location of the first object and the second location of the second object in the first image frame being less than a predetermined distance threshold, determining the time point of the first image frame in the video as the key time point of the video. ### Claim 10 (depends on 1) The method according to claim 1, wherein the determining the time point of the image frame as the key time point of the video comprises: based on the distance between the first location of the first object and the second location of the second object in the first image frame being greater than or equal to a predetermined distance threshold, determining that the time point of the first image frame in the video is not the key time point of the video. ### Claim 11 An apparatus for recognizing a key time point in a video, comprising: at least one memory storing computer program code; and at least one processor configured to access the at least one memory and operate as instructed by the computer program code, the computer program code comprising: first processing code configured to cause the at least one processor to obtain at least one video segment by processing each frame in the video by an image classification model, the image classification model being a machine learning model obtained by training according to a first sample image frame marked with a shot type, wherein each of the at least one video segment comprises at least two consecutive image frames in the video and each of the at least one video segment corresponds to one shot type among a plurality of shot types; second processing code configured to cause the at least one processor to determine a target video segment in the at least one video segment based on the shot type of the at least one video segment and obtain a first location of a first object and a second location of the second object in an image frame of the target video segment by an image detection model, the image detection model being a machine learning model obtained by training according to a second sample image frame marked with the first location of the first object and the second location of the second object; and determining code configured to cause the at least one processor to, based on a distance between the first location of the first object and the second location of the second object in the image frame satisfying a preset condition, determine a time point of the image frame as the key time point of the video. ### Claim 12 (depends on 11) The apparatus according to claim 11, wherein the first processing code is further configured to cause the at least one processor to: input each image frame in the video into the image classification model to obtain a model classification result that indicates the shot type for each image frame in the video; perform a smoothing correction according to the shot type of the each image frame in the video; and divide the video into the at least one video segment according to the shot type of the each image frame in the video. ### Claim 13 (depends on 12) The apparatus according to claim 12, wherein the first processing code is further configured to cause the at least one processor to: obtain a first category mode of a first image group and a second category mode of a second image group, the first category mode and the second category mode indicating the shot types in the first image group and the second image group with a maximum quantity of corresponding image frames, wherein the first image group includes r image frames prior to a second image frame, the second image group including r image frames subsequent to the second image frame, the second image frame being any one image frame among a plurality of image frames in the video other than the first r frames and the last r frames, where r is an integer greater than or equal to 1; and based on the first category mode and the second category mode being the same, set the shot type of the target image frame to the shot type of the first image group. ### Claim 14 (depends on 12) The apparatus according to claim 12, wherein the first processing code is further configured to cause the at least one processor to: divide the video into at least one temporary video segment according to the shot type of the each image frame in the video such that each of the at least one temporary video segment includes image frame belong to the same shot type, wherein the shot types of two consecutive temporary video segments are different; based on a quantity of image frames in a target temporary video segment being less than a preset quantity threshold, modify a shot type of each image frame in the target temporary video segment into a shot type of each image frame in a temporary video segment previous to the target temporary video segment, and combine the target temporary video segment with the temporary video segment previous to the target temporary video segment, wherein the target temporary video segment is any one temporary video segment in the at least one temporary video segment other than a first temporary video segment; and use each remaining temporary video segment in the at least one temporary video segment as the at least one video segment. ### Claim 15 (depends on 11) The apparatus according to claim 11, wherein the second processing code is further configured to cause the at least one processor to: input each image frame in the target video segment into the image detection model to obtain a model detection result that indicates respective temporary locations of the first object and the second object in the image frame of the target video segment; and perform smoothing correction on the respective temporary locations of the first object and the second object in the image frame of the target video segment to obtain the respective locations of the first object and the second object in the image frame of the target video segment. ### Claim 16 (depends on 15) The apparatus according to claim 15, wherein the second processing code is further configured to cause the at least one processor to: obtain temporary locations of a target object in image frames of a third image group and a fourth image group, the target object being any one of the first object and the second object, the third image group including w image frames previous to a third image frame, the fourth image group including w image frames subsequent to the third image frame, the third image frame being any one of a plurality of image frames in the target video segment other than the first w frames and the last w frames, where w is an integer greater than or equal to 1: obtain an average location that is indicated as an average value of the temporary locations of the target object in the image frames of the third image group and the fourth image group; and correct the temporary location of the target object in the third image frame according to the average location. ### Claim 17 (depends on 16) The apparatus according to claim 16, wherein the second processing code is further configured to cause the at least one processor to: obtain an offset of the temporary location of the target object in the third image frame relative to the average location; and correct, in a case that the offset is greater than an offset threshold, the temporary location of the target object in the third image frame to be the average location. ### Claim 18 (depends on 15) The apparatus according to claim 15, wherein the second processing code is further configured to cause the at least one processor to: obtain a template image corresponding to a fourth image frame which is an image frame including the target object and that is not detected by the image detection model, the target object being any one of the first object and the second object, the template image being an image corresponding to a temporary location of the target object in a fifth image frame which is an image frame including the target object and that is detected by the image detection model previous or subsequent to the fourth image frame among the image frames in the target video segment; and perform a template matching in the fourth image frame through the template image to obtain the temporary location of the target object in the fourth image frame. ### Claim 19 (depends on 11) The apparatus according to claim 11, wherein the determining code is further configured to cause the at least one processor to, based on the distance between the first location of the first object and the second location of the second object in the first image frame being less than a predetermined distance threshold, determining the time point of the first image frame in the video as they key time point of the video. ### Claim 20 A non-transitory computer-readable storage medium storing at least one computer program code configured to cause a computer processor to: obtain at least one video segment by processing each frame in a video by an image classification model, the image classification model being a machine learning model obtained by training according to a first sample image frame marked with a shot type, wherein each of the at least one video segment comprises at least two consecutive image frames in the video and each of the at least one video segment corresponds to one shot type among a plurality of shot types; determine a target video segment in the at least one video segment based on the shot type of the at least one video segment; obtain a first location of a first object and a second location of a second object in an image frame of the target video segment by an image detection model, the image detection model being a machine learning model obtained by training according to a second sample image frame marked with the first location of the first object and the second location of the second object; and based on a distance between the first location of the first object and the second location of the second object in the image frame satisfying a preset condition, determine a time point of the image frame as a key time point of the video.