# US 12,603,113 B1 · Synchronizing audio and video data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,603,113 B1 - Document ID: US12603113B1 - Status: Granted - Kind: B1 - Application number: 18/306,920 - Filed: 2023-04-25 - Granted: 2026-04-14 - Earliest claimed: 2022-09-26 - Assignees: Meta Platforms, Inc. - Applicants: Meta Platforms, Inc. - Inventors: Xueting Yan; Hao Li; Qin Huang; Jodi Jiaming Huang; Siyuan Yu; Jing Yang Zhang; Yu-Chih Tung; Hao Huang; Yuncheng Wu - Web page: https://patentgrep.com/patent/US12603113B1 ## CPC - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 11/00: Two-dimensional [2D] image generation - G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals - G11B 27/10: Indexing; Addressing; Timing or synchronising; Measuring tape travel - G11B 27/28: by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)} - G11B 27/34: Indicating arrangements {(indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D)} - G11B 27/36: Monitoring, i.e. supervising the progress of recording or reproducing {(for digital recording G11B20/00 and s.gr.; for monitoring, testing or measuring of TV recorders of the type covered by H04N5/76 and subgroups, see H04N17/06)} - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/8547: involving timestamps for synchronizing content - G06T 2207/10016: Video; Image sequence - G06T 2207/30196: Human being; Person - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Methods, systems, and storage media for synchronizing audio and video data are disclosed. Exemplary implementations may: receive at least one visual media file; receive audio data; determine a beat pattern associated with the audio data; generate a visual effect sequence comprising the at least one visual media file and at least one three-dimensional feature; and generate a resultant video file comprising the visual effect sequence and a transition sequence associated with the beat pattern. ## Claims ### Claim 1 A computer-implemented method for synchronizing audio and video data, the method comprising: receiving at least one visual media file, the at least one visual media file including an image file or a video file; receiving audio data, the audio data including at least one song; determining a beat pattern associated with the audio data, the beat pattern indicating a temporal position of a beat within the audio data; generating a first visual effect sequence and a second visual effect sequence comprising the at least one visual media file and at least one three-dimensional feature, wherein the at least one three-dimensional feature includes an object in a foreground of the at least one visual media file; and generating a resultant video file comprising the first visual effect sequence that transitions into the second visual effect sequence at the temporal position of the beat in the audio data, wherein the at least one visual media file is re-rendered with a zoom-and-pan effect such that the object appears to move relative to a background of the at least one visual media file as a field of view changes. ### Claim 2 (depends on 1) The method of claim 1, wherein the at least one three-dimensional feature is generated from a model configured to extract a depth projection from a two-dimensional image. ### Claim 3 (depends on 1) The method of claim 1, wherein receiving the audio data comprises determining a vibe associated with synchronizing the at least one visual media file and the audio data. ### Claim 4 (depends on 3) The method of claim 3, wherein the vibe is determined from metadata associated with one or both of the audio data and/or the at least one visual media file. ### Claim 5 (depends on 1) The method of claim 1, wherein the first visual effect sequence includes one or more successive visual effects. ### Claim 6 (depends on 5) The method of claim 5, wherein a given visual effect includes a process by which imagery is created and/or manipulated outside a context of underlying visual media. ### Claim 7 (depends on 6) The method of claim 6, wherein the given visual effect includes applying one or more of a gradient, a mathematical colormap, a change in color value, a translation, a flip, a mirror, a rotation, a scaling up or down, a shear, a distortion, a blur, and/or a filter to a given visual media file. ### Claim 8 (depends on 1) The method of claim 1, wherein the zoom-and-pan effect includes decreasing the field of view while simultaneously moving the field of view within bounds of the at least one visual media file. ### Claim 9 A system configured for synchronizing audio and video data, the system comprising: one or more hardware processors configured by machine-readable instructions to: receive at least one visual media file, the at least one visual media file including an image file or a video file; receive audio data, the audio data including at least one song, wherein receiving the audio data comprises determining a vibe associated with synchronizing the at least one visual media file and the audio data; determine a beat pattern associated with the audio data; generate a visual effect sequence comprising the at least one visual media file and at least one three-dimensional feature, wherein the at least one three-dimensional feature is generated from a model configured to extract a depth projection from a two-dimensional image, wherein the at least one three-dimensional feature includes an object in a foreground of the at least one visual media file, and wherein the at least one visual media file is re-rendered with a zoom-and-pan effect such that the object appears to move relative to a background of the at least one visual media file as a field of view changes; and generate a resultant video file comprising the visual effect sequence and a transition sequence associated with the beat pattern. ### Claim 10 (depends on 9) The system of claim 9, wherein the at least one song comprises a melody definable by a beat pattern. ### Claim 11 (depends on 10) The system of claim 10, wherein the beat pattern includes a series of beats in the at least one song. ### Claim 12 (depends on 10) The system of claim 10, wherein determining the beat pattern includes converting a sound waveform of the at least one song into a spectrogram. ### Claim 13 (depends on 12) The system of claim 12, wherein the spectrogram includes a two-dimensional time-frequency representation of the at least one song. ### Claim 14 (depends on 10) The system of claim 10, wherein determining the beat pattern includes weighting energy values within each critical band of a simulated response of a human brain to musical sounds. ### Claim 15 (depends on 14) The system of claim 14, wherein the simulated response is based on cochlea model. ### Claim 16 (depends on 10) The system of claim 10, wherein the beat pattern can be determined for multiple songs to be integrated into a resultant video file. ### Claim 17 (depends on 9) The system of claim 9, wherein generating the resultant video file includes synchronizing the at least one visual media to the beat pattern by trimming the at least one visual media file such that a beginning and an ending coincide with beats in the beat pattern. ### Claim 18 A non-transient computer-readable storage medium having instructions embodied thereon, the instructions being executable by one or more processors to perform a method for synchronizing audio and video data, the method comprising: receiving at least one visual media file, the at least one visual media file including an image file or a video file; receiving audio data, the audio data including at least one song, wherein receiving the audio data comprises determining a vibe associated with synchronizing the at least one visual media file and the audio data; determining a beat pattern associated with the audio data; generating a visual effect sequence comprising the at least one visual media file and at least one three-dimensional feature, wherein the at least one three-dimensional feature is generated from a model configured to extract a depth projection from a two-dimensional image, wherein the at least one three-dimensional feature includes an object in a foreground of the at least one visual media file, and wherein the at least one visual media file is re-rendered with a zoom-and-pan effect such that the object appears to move relative to a background of the at least one visual media file as a field of view changes; and generating a resultant video file comprising the visual effect sequence and a transition sequence associated with the beat pattern, wherein generating the resultant video file includes synchronizing the at least one visual media to the beat pattern by trimming the at least one visual media file such that a beginning and an ending coincide with beats in the beat pattern.