# US 12,225,272 B2 · Optimizing insertion points for content based on audio and video characteristics > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,225,272 B2 - Document ID: US12225272B2 - Status: Granted - Kind: B2 - Application number: 18/174,460 - Filed: 2023-02-24 - Published: 2024-06-13 - Granted: 2025-02-11 - Earliest claimed: 2023-02-24 - Assignees: Disney Enterprises, Inc.; Beijing YoJaJa Software Technology Development Co., Ltd. - Applicants: Disney Enterprises, Inc.; Beijing Hulu Software Technology Development Co., Ltd - Inventors: Yuanyi Xue; Michael John Bracco; Scott Christopher Labrozzi; Christopher Richard Schroers; Wenhao Zhang - Web page: https://patentgrep.com/patent/US12225272B2 ## CPC - H04N 21/8455: {involving pointers to the content, e.g. pointers to the I-frames of the video stream} - H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/44016: {involving splicing one content stream with another content stream, e.g. for substituting a video clip} - H04N 21/4662: {characterized by learning algorithms} - H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract One embodiment of the present invention sets forth a technique for inserting content into a media program. The technique includes determining a plurality of markers corresponding to a plurality of locations within a media program. The technique also includes for each marker included in the plurality of markers, automatically analyzing a first set of intervals within the media program that lead up to the marker and a second set of intervals within the media program that immediately follow the marker and determine a set of audio characteristics associated with the first set of intervals and the second set of intervals. The technique further includes generating a plurality of scores for the plurality of markers based on the set of audio characteristics for each marker and inserting additional content at one or more markers included in the plurality of markers based on the plurality of scores. ## Claims ### Claim 1 A computer-implemented method for inserting additional content into a media program, the method comprising: determining a plurality of markers corresponding to a plurality of locations within the media program; for each marker included in the plurality of markers, performing one or more operations to automatically analyze one or more portions of the media program proximate to the marker and determine a set of video characteristics associated with the one or more portions; for each marker included in the plurality of markers, generating a score based on a weighted combination of the set of video characteristics of the one or more portions proximate to the marker, thereby generating a plurality of scores each corresponding to the one or more portions proximate to each of the plurality of markers; and inserting the additional content into the media program at one or more markers included in the plurality of markers based on the plurality of scores. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: determining one or more marker treatments associated with the one or more markers based on the sets of video characteristics of the one or more portions proximate to each of the one or more markers, wherein the one or more marker treatments comprise at least one of a video fade-in, a video fade-out, or a black slate. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising: generating a ranking for the plurality of markers based on the plurality of scores of respective one or more portions proximate to each of the plurality of markers; and determining the one or more markers based on the ranking. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the set of video characteristics comprise a luminance level. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the set of video characteristics comprise one or more measurements of statistical dispersion associated with scene changes. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the set of video characteristics comprise at least one of a spatial activity or a temporal activity. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the set of video characteristics comprise a semantic characteristic. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, further comprising: for each marker included in the plurality of markers, performing one or more additional operations to automatically analyze the one or more portions of the media program proximate to the marker and determine a set of audio characteristics associated with the one or more portions proximate to the marker; and for each marker included in the plurality of markers, converting the sets of audio characteristics of the one or more portions proximate to the marker to a score to be included in the plurality of scores. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein the set of audio characteristics comprise at least one of a loudness level, a presence or absence of speech, an indication of semantic continuity, a mel-scale spectrogram, one or more mel-frequency ceptrum coefficients, or a tempogram. ### Claim 10 One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform: determining a plurality of markers corresponding to a plurality of locations within a media program; for each marker included in the plurality of markers, performing one or more operations to automatically analyze one or more portions of the media program proximate to the marker and determine a set of video characteristics associated with the one or more portions; for each marker included in the plurality of markers, generating a score based on a weighted combination of the set of video characteristics of the one or more portions proximate to the marker, thereby generating a plurality of scores each corresponding to the one or more portions proximate to each of the plurality of markers; and inserting additional content into the media program at one or more markers included in the plurality of markers based on the plurality of scores. ### Claim 11 (depends on 10) The one or more non-transitory computer-readable media of claim 10, wherein the instructions further cause the one or more processors to perform: for each marker included in the plurality of markers, performing one or more additional operations to automatically analyze the one or more portions of the media program proximate to the marker and determine a set of audio characteristics associated with the one or more portions proximate to the marker; and for each marker included in the plurality of markers, converting the sets of audio characteristics of the one or more portions proximate to the marker to a score to be included in the plurality of scores. ### Claim 12 (depends on 10) The one or more non-transitory computer-readable media of claim 10, wherein the set of video characteristics comprise at least one of an average distance between scene changes that fall within an interval of video around one of the plurality of markers and a variance of distances between the scene changes within the interval of video. ### Claim 13 (depends on 12) The one or more non-transitory computer-readable media of claim 12, wherein the plurality of scores comprise a first score that is generated based on a minimum threshold for the average distance between the scene changes and a maximum threshold for the variance of distances between the scene changes. ### Claim 14 (depends on 10) The one or more non-transitory computer-readable media of claim 10, wherein the set of video characteristics comprise a spatial perceptual information metric and a temporal perceptual information metric for an interval of video around one of the plurality of markers. ### Claim 15 (depends on 14) The one or more non-transitory computer-readable media of claim 14, wherein the plurality of scores comprise a first score that is generated based on a first threshold for the spatial perceptual information metric and a second threshold for the temporal perceptual information metric. ### Claim 16 (depends on 10) The one or more non-transitory computer-readable media of claim 10, wherein the set of video characteristics comprise at least one of a creator, a genre, an action, an activity, a character, an object, a facial expression, a sentiment, an emotion, or a scenario associated with an interval of video around the first one of the plurality of markers. ### Claim 17 (depends on 10) The one or more non-transitory computer-readable media of claim 10, wherein the plurality of scores comprise a first score that is generated based on a presence or an absence of a semantic characteristic within an interval of video around a first one of the plurality of markers. ### Claim 18 (depends on 10) The one or more non-transitory computer-readable media of claim 10, wherein at least one marker included in the plurality of markers comprises at least one of a scene change, a chapter marker, or a black slug. ### Claim 19 A system comprising: one or more memories that store instructions, and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform: determining a plurality of markers corresponding to a plurality of locations within a media program; for each marker included in the plurality of markers, performing one or more operations to automatically analyze one or more portions of the media program proximate to the marker and determine a set of video characteristics associated with the one or more portions; for each marker included in the plurality of markers, generating a score based on a weighted combination of the set of video characteristics of the one or more portions proximate to the marker, thereby generating a plurality of scores each corresponding to the one or more portions proximate to each of the plurality of markers; and inserting additional content into the media program at one or more markers included in the plurality of markers based on the plurality of scores. ### Claim 20 (depends on 19) The system of claim 19, wherein the one or more processors, when executing the instructions, are further configured to perform: for each marker included in the plurality of markers, performing one or more additional operations to automatically analyze the one or more portions of the media program proximate to the marker and determine a set of audio characteristics associated with the one or more portions proximate to the marker; and for each marker included in the plurality of markers, converting the sets of audio characteristics of the one or more portions proximate to the marker to a score to be included in the plurality of scores.