# US 12,120,399 B2 · Method and system for matching clips with videos via media analysis > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,120,399 B2 - Document ID: US12120399B2 - Status: Granted - Kind: B2 - Application number: 17/599,434 - Filed: 2020-03-27 - Published: 2022-06-16 - Granted: 2024-10-15 - Earliest claimed: 2020-03-27 - Assignees: NEP Supershooters, L.P. - Applicants: Piksel, Inc. - Inventors: Gerald Chao - Web page: https://patentgrep.com/patent/US12120399B2 ## CPC - H04N 21/4884: {for displaying subtitles} - G06F 9/454: {Multi-language systems; Localisation; Internationalisation} - G06F 18/2413: based on distances to training or reference patterns - G06V 10/761: {Proximity, similarity or dissimilarity measures} - G06V 10/82: using neural networks - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 20/48: {Matching video sequences} - G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes} - G06V 20/635: {Overlay text, e.g. embedded captions in a TV programme} - H04N 19/42: characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation (H04N19/635 takes precedence) - H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method includes comparing each textless video clip to a plurality of portions of the video file corresponding to the full length video file; determining each textless video clip being similar to only one portion of the video file as being a matched pair; for each matched pair identifying if their text content is different, wherein identification of a different text content dictates that the textless video clip corresponds to a portion of the video file having overlaid text; training a classifier to predict whether an area of text detected in the full length video is overlaid text; determining the probability of each portion of the full-length video having overlaid text; determining each textless video clip being similar to more than one portion of the video file as being a potential matched pair; and resolving the potential matched pairs with the determined probability. ## Claims ### Claim 1 A method of processing a video file with overlaid text and a textless video file without overlaid text comprising a plurality of textless video clips, wherein the textless video clips are derived from the video file, the method comprising: comparing each textless video clip to a plurality of portions of the video file, the plurality of portions corresponding to a full length of the video file; determining each textless video clip that has a video portion image similar to only one video portion image of the video file as an unambiguous match; determining each textless video clip which has a video portion image similar to more than one video portion image of the video file as an ambiguous match; creating a matched pair for each textless video clip and portion of the video file which is an unambiguous match; creating a potential matched pair for each textless video clip and each portion of the video file which is an ambiguous match; comparing, for each matched pair, a text content of the textless video clip and the portion of the video file; identifying, for each matched pair, if the compared text content is different; determining, if the text content is different, that the textless video clip corresponds to a portion of the video file having overlaid text; training a classifier to predict whether an area of text detected in the full length video is overlaid text in dependence on this identification and determination; determining a probability of each portion of the full length video having overlaid text via the trained classifier; and resolving the potential matched pairs with the determined probability as being matched pairs. ### Claim 2 (depends on 1) The method of claim 1, further comprising a step of decompressing the video file and decompressing the plurality of textless video clips prior to the comparing step. ### Claim 3 (depends on 1) The method of claim 1 further comprising a step of decoding the video file and the plurality of textless videos into individual frames prior to the comparing step, and extracting individual textless clips by identifying start and end points of individual clips, such that the textless video is a collection of short video frames and the full length video is one long collection of video frames. ### Claim 4 (depends on 1) The method of claim 1 further comprising grouping the video file into shots and grouping the video clips into shots, each shot comprising a series of frames that are considered to contain the same amount of visual information, and converting each shot into a low-dimension representation, wherein converting each shot into a low-dimension representation comprises selecting one frame of the shot to represent all frames of the shot. ### Claim 5 (depends on 4) The method of claim 4 further comprising discarding information not needed for a matching process to provide a more compact representation, taking the low-dimensional representation per shot from each textless clip, and searching for matches within the low-dimensional representation per shot from the full length video, wherein the searching comprises allocating a distance metric to each comparison, with a lowest metric indicating the most likely match, the method further comprising applying a threshold to assess the distance information, wherein there is determined: (i) a set of matched textless clips, corresponding to textless clips which are determined to match only one portion of the video file, and (ii) a set of unmatched textless clips which are determined not to match only one portion of the video file. ### Claim 6 (depends on 1) The method of claim 1, wherein the step of determining the textless video clips having an image being similar to only one portion of the video file is unambiguous matching. ### Claim 7 (depends on 1) The method of claim 1, wherein the pair of matched textless video clip and similar only one portion are represented in their original frames for the identifying step. ### Claim 8 (depends on 1) The method of claim 1 further comprising performing text-in-image detection of an image frame per shot of both videos. ### Claim 9 (depends on 1) The method of claim 1 wherein there is provided a set of text boxes, and an identification of video frames in which they appear. ### Claim 10 (depends on 9) The method of claim 9 further comprising comparing each text box per shot from the full length video with ones from the textless clips, wherein any difference indicates that overlaid text is present in the full length video, and the absence of a difference indicates that the text is background text. ### Claim 11 (depends on 7) The method of claim 7, further comprising training a classifier based on this comparison, which receives as inputs pairs comprising overlaid text and background text, the classifier defining a model. ### Claim 12 (depends on 4) The method of claim 4 further computing a probability for each shot of the full length video, that each shot includes overlaid text, in dependence on a classifier model. ### Claim 13 (depends on 12) The method of claim 12 further comprising allocating a probability of overlaid text to each portion of the video having text. ### Claim 14 (depends on 13) The method of claim 13 further comprising identifying those portions of the video file having text and applying a probability to those identified portions. ### Claim 15 (depends on 12) The method of claim 12 further comprising determining each textless video clip having an image being similar to more than one image of more than one portion of the video file, and selecting one portion of the video file for that textless video clip independence on the one having the highest probability of overlaid text. ### Claim 16 (depends on 12) The method of claim 12 further comprising taking the low-dimensional representation per shot from each textless clip, and searching for matches within the low-dimensional representation per shot from the full length video. ### Claim 17 (depends on 13) The method of claim 13 further comprising increasing the likelihood of matching proportional to a probability of each shot including overlaid text. ### Claim 18 (depends on 16) The method of claim 16 further comprising taking a frame from each textless clip, and searching for per-frame matches within each frame from its unambiguously matched portion of the full length video, to find a frame offset that results in the overall minimum distance. ### Claim 19 (depends on 1) The method of claim 1 wherein the portions of the video which are unambiguously matched to a textless clip are output as final matches, and the method further comprising aligning the output final matches. ### Claim 20 A device for processing a video file with overland text and a textless video file without overlaid text comprising a plurality of textless video clips, wherein the textless video clips are derived from the video file, the device comprising: a matching module for comparing each textless video clip to a plurality of portions of the video file, the plurality of portions corresponding to a full length video file, the matching module configured to: determine each textless video clip having a video portion image being similar to only a video portion image of only one portion of the video file as being an unambiguous match; determine each textless video clip having a video portion image being similar to only a video portion image of only one portion of the video file as being an unambiguous match, create a matched pair for each textless video clip and portion of the video file which is an unambiguous match; and create a potential matched pair for each textless video clip and each portion of the video file which is an ambiguous match, a training module configured to: compare, for each matched pair, a text content of the textless video clip and the portion of the video file; identify, for each matched pair, if the compared text content is different; determine, if the text content is different, that the textless video clip corresponds to a portion of the video file having overlaid text; and training a classifier to predict whether an area of text detected in the full length video is overlaid text in dependence on this identification and determination; a processing module configured to: determine a probability of each portion of the full length video having overlaid text via the trained classifier; resolve the potential matched pairs with the determined probability as being matched pairs.