# US 10,313,722 B1 · Synchronizing audio content and video content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,313,722 B1 - Document ID: US10313722B1 - Status: Granted - Kind: B1 - Application number: 15/717,373 - Filed: 2017-09-27 - Granted: 2019-06-04 - Earliest claimed: 2017-09-27 - Assignees: Amazon Technologies, Inc. - Applicants: Amazon Technologies, Inc. - Inventors: Ron Searl; Yongjun Wu - Web page: https://patentgrep.com/patent/US10313722B1 ## CPC - H04N 21/242: Synchronisation processes, e.g. processing of PCR [Programme Clock References] {(arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18)} - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/23424: {involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement} - H04N 21/26258: {for generating a list of items to be played back in a given order, e.g. playlist, or scheduling item distribution according to such list (retrieval of multimedia data based on playlists G06F16/40)} - H04N 21/8106: {involving special audio data, e.g. different tracks for different languages} - H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)} - H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments} - H04N 21/8586: {by using a URL (processing chained hypermedia data for information retrieval G06F16/94; information retrieval from the Internet by using URLs G06F16/955; URL in broadcast information H04H20/93; Web-based protocols H04L67/02)} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Techniques are described for synchronizing audio content and video content when server-side fragment insertion techniques are used. ## Claims ### Claim 1 A system, comprising one or more processors and memory configured to: identify a first portion of primary media content; select a first set of primary media fragments associated with the first portion of the primary media content; identify a first portion of advertisement media content, the advertisement media content having first version audio fragments and second version audio fragments associated therewith, the second version audio fragments corresponding to the first version audio fragments and including fewer frames of audio content than the corresponding first version audio fragments; determine a first duration based on the first portion of the advertisement media content, the first duration representing a potential delay between audio and video during playback; determine that the first duration is below a threshold, the threshold representing a value at which lack of synchronization between audio and video is perceivable by a user; select a first set of advertisement media fragments associated with the first portion of the advertisement media content, the first set of advertisement media fragments including the first version audio fragments; identify a second portion of the primary media content; select a second set of primary media fragments associated with the second portion of the primary media content; identify a second portion of the advertisement media content; identify a second duration based on the first portion of the advertisement media content and the second portion of the advertisement media content; determine that the second duration exceeds the threshold; select a second set of advertisement media fragments associated with second portion of the advertisement media content, the second set of advertisement media fragments including the first version audio fragments and at least one of the second version audio fragments; and generate a manifest uniform resource locator including a first index associated with the first set of advertisement media fragments and a second index associated with the second set of advertisement media fragments. ### Claim 2 (depends on 1) The system of claim 1, wherein the threshold represents a synchronization difference in a range between 10 milliseconds and 130 milliseconds. ### Claim 3 (depends on 1) The system of claim 1, wherein the second version audio fragments include one less frame of audio content than the first version audio fragments. ### Claim 4 (depends on 1) The system of claim 1, wherein the first set of advertisement media fragments are selected using advertisement content manifest data, the advertisement content manifest data including references to the first version audio fragments and the second version audio fragments. ### Claim 5 A system, comprising one or more processors and memory configured to: identify a portion of first media content; select first media fragments associated with the portion of the first media content; identify a portion of second media content, the second media content having first version audio fragments and second version audio fragments associated therewith, the second version audio fragments corresponding to the first version audio fragments and including fewer media frames than the first version audio fragments; determine a duration based on the portion of the second media content, the duration representing a potential delay between audio and video during playback; determine that the duration exceeds a threshold; select second media fragments associated with the portion of the second media content, the second media fragments including a plurality of the first version audio fragments and at least one of the second version audio fragments; and generate client manifest data, the client manifest data including references to the first media fragments and an index associated with the second media fragments. ### Claim 6 (depends on 5) The system of claim 5, wherein the client manifest data is a manifest identifier, and wherein the index associated with the second media fragments includes at least one of: a secondary content identifier, a start time of the second media fragments, a length of the second media fragments, or an indication of an audio version. ### Claim 7 (depends on 5) The system of claim 5, wherein the one or more processors and memory are further configured to generate the manifest data by: identifying an additional portion of the first media content; selecting additional first media fragments associated with the additional portion of the first media content; identifying an additional portion of the second media content; determining an additional duration based on the portion of the second media content and the additional portion of the second media content; determining that the additional duration does not exceed the threshold; selecting additional second media fragments associated with the additional portion of the second media content, the second media fragments including only the first version audio fragments; and wherein the client manifest data further includes an additional index associated with the additional second media fragments. ### Claim 8 (depends on 7) The system of claim 7, wherein the one or more processors and memory are configured to select the second media fragments based on first source manifest data associated with the portion of second media content, and the additional second media fragments are selected using second source manifest data associated with the additional portion of the second media content, the first source manifest data being distinct from the second source manifest data. ### Claim 9 (depends on 5) The system of claim 5, wherein the one or more processors and memory are configured to select the second media fragments based on source manifest data associated with the second media content, the source manifest data including references to both the first version audio fragments and the second version audio fragments. ### Claim 10 (depends on 5) The system of claim 5, wherein the threshold represents a synchronization difference in a range between 10 milliseconds and 130 milliseconds. ### Claim 11 (depends on 5) The system of claim 5, wherein the threshold is an upper threshold or a lower threshold. ### Claim 12 (depends on 5) The system of claim 5, wherein the one or more processors and memory are configured to identify the portion of the second media content by also identifying one or more additional portions of the second media content to be included in the client manifest data prior to identifying the portion of the second media content. ### Claim 13 A method, comprising: identifying a portion of first media content; selecting first media fragments associated with the portion of the first media content; identifying a portion of second media content, the second media content having first version audio fragments and second version audio fragments associated therewith, the second version audio fragments corresponding to the first version audio fragments and including fewer media frames than the first version audio fragments; determining a duration based on the portion of the second media content, the duration representing a potential delay between audio and video during playback; determining that the duration exceeds a threshold; selecting second media fragments associated with the portion of the second media content, the second media fragments including a plurality of the first version audio fragments and at least one of the second version audio fragments; and generating client manifest data, the client manifest data including references to the first media fragments and an index associated with the second media fragments. ### Claim 14 (depends on 13) The method of claim 13, wherein the client manifest data is a manifest identifier, and wherein the index associated with the second media fragments includes at least one of: a secondary content identifier, a start time of the second media fragments, a length of the second media fragments, or an indication of an audio version. ### Claim 15 (depends on 13) The method of claim 13, wherein generating the manifest data includes: identifying an additional portion of the first media content; selecting additional first media fragments associated with the additional portion of the first media content; identifying an additional portion of the second media content; determining an additional duration based on the portion of the second media content and the additional portion of the second media content; determining that the additional duration does not exceed the threshold; selecting additional second media fragments associated with the additional portion of the second media content, the second media fragments including only the first version audio fragments; and wherein the client manifest data further includes an additional index associated with the additional second media fragments. ### Claim 16 (depends on 15) The method of claim 15, wherein selecting the second media fragments is based on first source manifest data associated with the portion of second media content, and the additional second media fragments are selected using second source manifest data associated with the additional portion of the second media content, the first source manifest data being distinct from the second source manifest data. ### Claim 17 (depends on 13) The method of claim 13, wherein selecting the second media fragments is based on source manifest data associated with the second media content, the source manifest data including references to both the first version audio fragments and the second version audio fragments. ### Claim 18 (depends on 13) The method of claim 13, wherein the threshold represents a synchronization difference in a range between 10 milliseconds and 130 milliseconds. ### Claim 19 (depends on 13) The method of claim 13, wherein the threshold is an upper threshold or a lower threshold. ### Claim 20 (depends on 13) The method of claim 13, wherein identifying the portion of the second media content includes identifying one or more additional portions of the second media content to be included in the client manifest data prior to identifying the portion of the second media content.