# US 12,112,771 B2 · Retiming digital videos utilizing deep learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,771 B2 - Document ID: US12112771B2 - Status: Granted - Kind: B2 - Application number: 18/185,137 - Filed: 2023-03-16 - Published: 2023-08-31 - Granted: 2024-10-08 - Earliest claimed: 2022-02-25 - Assignees: Adobe Inc. - Applicants: Adobe Inc. - Inventors: Simon Jenni; Markus Woodson; Fabian David Caba Heilbron - Web page: https://patentgrep.com/patent/US12112771B2 ## CPC - G11B 27/005: {Reproducing at a different information rate from the information rate of recording (for television signals H04N5/783)} - G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/234381: {by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping} - H04N 21/2402: {Monitoring of the downstream path of the transmission network, e.g. bandwidth available (traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876)} - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract This disclosure describes one or more implementations of systems, non-transitory computer-readable media, and methods that generate a temporally remapped video that satisfies a desired target duration while preserving natural video dynamics. In certain instances, the disclosed systems utilize a playback speed prediction machine-learning model that recognizes and localizes temporally varying changes in video playback speed to re-time a digital video with varying frame-change speeds. For instance, to re-time the digital video, the disclosed systems utilize the playback speed prediction machine-learning model to infer the slowness of individual video frames. Subsequently, in certain embodiments, the disclosed systems determine, from frames of a digital video, a temporal frame sub-sampling that is consistent with the slowness predictions and fit within a target video duration. In certain implementations, the disclosed systems utilize the temporal frame sub-sampling to generate a speed varying digital video that preserves natural video dynamics while fitting the target video duration. ## Claims ### Claim 1 A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a computing device to perform operations comprising: determining, utilizing a convolutional neural network, varying predicted playback speeds of frames of a digital video by generating between-frame speed up times utilizing custom network heads of the convolutional neural network; and generating a modified digital video with temporally varying playback speeds utilizing the varying predicted playback speeds of the frames of the digital video. ### Claim 2 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein determining, utilizing the convolutional neural network, the varying predicted playback speeds of the frames of the digital video comprises utilizing a three-dimensional convolutional neural network modified to predict per-frame playback speeds. ### Claim 3 (depends on 2) The non-transitory computer-readable medium of claim 2, wherein the three-dimensional convolutional neural network preserves a temporal dimension of the digital video by having all temporal strides set to one. ### Claim 4 (depends on 2) The non-transitory computer-readable medium of claim 2, wherein the three-dimensional convolutional neural network utilizes spatial-only global average pooling. ### Claim 5 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein determining, utilizing the convolutional neural network, the varying predicted playback speeds of the frames of the digital video comprises generating a varying playback speed prediction vector comprising varying speed classifications for frames of the digital video based on likelihoods of observing changes in playback speed between the frames of the digital video. ### Claim 6 (depends on 5) The non-transitory computer-readable medium of claim 5, wherein the operations further comprise determining frame slowness predictions from playback speed classifications that indicate a likelihood of frames receiving an increase in playback speed. ### Claim 7 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the operations further comprise determining a video frame sub-sampling from the frames of the digital video based on the varying predicted playback speeds and a target video duration. ### Claim 8 A method comprising: determining, utilizing a convolutional neural network, varying predicted playback speeds of frames of a digital video by generating between-frame speed up times utilizing custom network heads of the convolutional neural network; and generating a modified digital video with temporally varying playback speeds utilizing the varying predicted playback speeds of the frames of the digital video. ### Claim 9 (depends on 8) The method of claim 8, further comprising determining a video frame sub-sampling from the frames of the digital video based on the varying predicted playback speeds and a target video duration. ### Claim 10 (depends on 9) The method of claim 9, further comprising: determining the target video duration utilizing a source duration of digital audio content; and combining the modified digital video with the digital audio content. ### Claim 11 (depends on 9) The method of claim 9, wherein determining the video frame sub-sampling comprises skipping one or more of the frames of the digital video using a sequence of frame skippings determined from the varying predicted playback speeds and the target video duration. ### Claim 12 (depends on 11) The method of claim 11, further comprising generating, based on the sequence of frame skippings, a subsampled index of non-skipped frames as the video frame sub-sampling. ### Claim 13 (depends on 8) The method of claim 8, wherein generating the modified digital video comprises generating the modified digital video to comprise a first segment of frames for playback at a first playback speed and a second segment of frames for playback at a second playback speed. ### Claim 14 (depends on 8) The method of claim 8, wherein determining, utilizing the convolutional neural network, the varying predicted playback speeds of the frames of the digital video comprises utilizing a three-dimensional convolutional neural network modified to predict per-frame playback speeds. ### Claim 15 (depends on 14) The method of claim 14, wherein the three-dimensional convolutional neural network preserves a temporal dimension of the digital video by having all temporal strides set to one. ### Claim 16 (depends on 14) The method of claim 14, wherein the three-dimensional convolutional neural network utilizes spatial-only global average pooling.