# US 2021/0241739 A1 · Methods, Systems, Devices and Computer Program Products for Adapting External Content to a Video Stream > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241739 A1 - Document ID: US20210241739A1 - Status: Published application - Kind: A1 - Application number: 17/146,263 - Filed: 2021-01-11 - Published: 2021-08-05 - Earliest claimed: 2017-06-29 - Assignees: Dolby International AB - Applicants: Dolby International AB - Inventors: Antonio MATEOS SOLE - Web page: https://patentgrep.com/patent/US20210241739A1 ## CPC - G10H 1/368: {displaying animated or moving pictures synchronized with the music or audio part} - G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals - G11B 27/10: Indexing; Addressing; Timing or synchronising; Measuring tape travel - G11B 27/28: by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)} - G11B 27/34: Indicating arrangements {(indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D)} - G10H 2240/085: Mood, i.e. generation, detection or selection of a particular emotional content or atmosphere in a musical piece - G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) ## Abstract This disclosure falls into the field of adapting external content to a video stream, and more specifically it is related to analyzing the video stream to define a suitable narrative model, and adapting the external content based on this narrative model. ## Claims ### Claim 1 A method for adapting external content to a video stream having a time span, the video stream comprising a plurality of frames each comprising pixels, the method comprising the steps of: computing a first plurality of difference indicators, each difference indicator of the first plurality of difference indicators indicating a difference between values of the pixels between a frame of the plurality of frames and an nth subsequent frame of the plurality of frames, wherein n is an integer≥1; computing a second plurality of difference indicators by convolving the first plurality of difference indicators with a strictly decreasing function, each difference indicator of the second plurality of difference indicators corresponding to a specific position in time of the video stream, fitting a predetermined parametrizable function to the second plurality of difference indicators, thereby yielding a function descriptive of a narrative model, FNM, for the video stream during the time span of the video stream, using the FNM to adapt external content to the video stream. ### Claim 2 (depends on 1) The method according to claim 1, wherein the computing of each specific difference indicator of the first plurality of difference indicators comprises: comparing corresponding pixel values between the frame of the plurality of frames and the nth subsequent frame of the plurality of frames; and calculating the specific difference indicator based on the comparisons. ### Claim 3 (depends on 1) The method according to claim 1, further comprising the steps of: after yielding the FNM for the video stream: comparing the second plurality of difference indicators to the FNM, determining that it exists a continuous subset of difference indicators among the second plurality of difference indicators wherein all difference indicators of the subset of difference indicators have a value being at least a threshold lower than a corresponding value in the FNM, wherein the continuous subset of difference indicators corresponding to a sub time span of the video stream, adapting the external content to the video stream also based on a starting point in time and an ending point in time for the sub time span of the video stream. ### Claim 4 (depends on 1) The method according to claim 1, wherein the step of computing the plurality of second difference indicators further comprises the step of comparing each of the first plurality of difference indicators to a predetermined threshold and setting any difference indicator being below said predetermined threshold to zero before performing the step of convolving the first plurality of difference indicators with the strictly decreasing function. ### Claim 5 (depends on 1) The method according to claim 1, wherein each specific difference indicator of the first plurality of difference indicators is computed by calculating differences between RGB values of corresponding pixels of the frame and the nth subsequent frame, and calculating the specific difference indicator based on the calculated differences. ### Claim 6 (depends on 1) The method according to claim 1, wherein the strictly decreasing function is an exponentially decaying function or a linear decreasing function. ### Claim 7 (depends on 6) The method according to claim 6, wherein the strictly decreasing function is an exponentially decaying function which falls below 1% of its initial value after 0.1-2.0 seconds. ### Claim 8 (depends on 7) The method according to claim 7, wherein the exponentially decaying function is an exponentially decaying function which falls below 1% of its initial value after 1 second. ### Claim 9 (depends on 1) The method according to claim 1 wherein the step of using the FNM to adapt external content to the video stream is automatic. ### Claim 10 (depends on 1) The method according to claim 1, wherein the external content consists of pre-recorded music. ### Claim 11 (depends on 10) The method according to claim 10, wherein the pre-recorded music is recorded in a multitrack format. ### Claim 12 (depends on 11) The method according to claim 11, wherein the step of using the FNM to adapt external content to the video stream comprises muting and unmuting of a track of a multitrack music recording based on the FNM. ### Claim 13 (depends on 11) The method according to claim 11, wherein the using the FNM to adapt external content to the video stream comprises volume adjustment of a track of a multitrack music recording based on the FNM. ### Claim 14 (depends on 3) The method according to claim 3, wherein the step of adapting the external content to the video stream also based on a starting point in time and an ending point in time for the sub time span of the video stream comprises muting and unmuting of a track of a multitrack music recording based on the starting point in time and the ending point in time. ### Claim 15 (depends on 3) The method according to claim 3, wherein the step of adapting the external content to the video stream also based on a starting point in time and an ending point in time for the sub time span of the video stream comprises volume adjustment of a track of a multitrack music recording based on the starting point in time and the ending point in time. ### Claim 16 (depends on 1) The method according to claim 1, wherein the step of fitting the predetermined parametrizable function to the second plurality of difference indicator, thereby yielding an FNM for the video stream during the time span of the video stream, is done by least squares fitting. ### Claim 17 A system for adapting external content to a video stream having a time span, the video stream comprising a plurality of frames each comprising pixels, the system comprising a device connected to a database, the database comprising external content, wherein the device comprises a processor adapted for: computing a first plurality of difference indicators, each difference indicator of the first plurality of difference indicators indicating a difference between values of the pixels between a frame of the plurality of frames and an nth of subsequent frame of the plurality of frames, wherein n is an integer≥1; computing a second plurality of difference indicators by convolving the first plurality of difference indicators with a strictly decreasing function, each difference indicator of the second plurality of difference indicators corresponding to a specific position in time of the video stream, fitting a predetermined parametrizable function to the second plurality of difference indicator, thereby yielding a function descriptive of a narrative model, FNM, for the video stream during the time span of the video stream, selecting external content from the database, and using the FNM to adapt the selected external content to the video stream. ### Claim 18 (depends on 17) The system of claim 17, wherein the computing of each specific difference indicator of the first plurality of difference indicators comprises: comparing corresponding pixel values between the frame of the plurality of frames and the nth subsequent frame of the plurality of frames; and calculating the specific difference indicator based on the comparisons, ### Claim 19 (depends on 1) A computer program product having instructions which, when executed by a computing device or system, cause said computing device or system to perform the method according to claim 1.