# US 11,910,060 B2 · System and method for automatic detection of periods of heightened audience interest in broadcast electronic media > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,910,060 B2 - Document ID: US11910060B2 - Status: Granted - Kind: B2 - Application number: 17/320,525 - Filed: 2021-05-14 - Published: 2022-06-09 - Granted: 2024-02-20 - Earliest claimed: 2020-12-07 - Assignees: Caspian Hill Group, LLC - Applicants: Caspian Hill Group, LLC - Inventors: Amy Bolivar; Steven Lubin; Audrey Faust - Web page: https://patentgrep.com/patent/US11910060B2 ## CPC - H04N 21/4667: {Processing of monitored end-user data, e.g. trend analysis based on the log file of viewer selections} - H04N 21/8549: Creating video summaries, e.g. movie trailer {(retrieval in video databases by using presentations in form of a video summary G06F16/739)} - H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/251: {Learning process for intelligent management, e.g. learning user preferences for recommending movies (details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08)} - H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract This relates to using a computer simulation to test another computer program in real time or simulated real time that is sped up. The disclosed method and system synchronizes information input into the simulation so that the program under test operates in an independent way. The method and system operates a protocol to connect one running computer process, a trading computer program, with another running process, a computer program that executes a market simulation in order to optimize the quality and speed of the simulation and testing of the external computer program. ## Claims ### Claim 1 A method executed by a computer system for determining periods of time in an audio-visual signal encoding an event comprising: receiving a data representing the audio-visual signal said signal comprised of data representing an audience attending said event; extracting from the received audio-visual data at least one parameter data corresponding to an at least one time-slice within the audio-visual signal, said at least one parameter data representing a magnitude of heightened emotional state of the audience attending the event that is the subject of the of the audio-visual signal within the corresponding an at least one time slice; storing in a data structure the at least one parameter data and an at least one time slice value corresponding to the at least one parameter; determining an at least one time values of heightened interest by calculating a correlation between the extracted at least one parameter data representing a magnitude of heightened emotional state of the audience attending the event that is the subject of the of the audio-visual signal within the corresponding an at least one time slice and an at least one pre-determined parameter data values associated with a heightened audience interest level; and storing the determined at least one time values in a data structure associated with the received audio-visual signal. ### Claim 2 (depends on 1) The method of claim 1 where the extracting step is comprised of: extracting an audio portion of the audio-visual data stream; and generating from the extracted audio an at least one parameters representing an at least one spectral vectors, each spectral vector corresponding to the at least one time slice. ### Claim 3 (depends on 2) The method of claim 2 where the determining step is comprised of: for each time slice, automatically determining if the spectral vector of that time slice is part of a spectral pattern that is correlated with heightened audience interest; and in dependence on the automatic determination whether the spectral vector is part of said spectral pattern, storing in a data structure at least one data values representing at least one time value corresponding to the determined spectral pattern. ### Claim 4 (depends on 1) The method of claim 1 where the extracting step is comprised of: extracting an audio portion of the audio-visual data stream; detecting in the extracted audio portion an at least one word text data item, said word text data item having a corresponding at least one time slice value; and determining if the at least one word text data item is an element in a pre-determined word set correlated with heightened audience emotion or audience interest. ### Claim 5 (depends on 1) The method of claim 1 where the extracting step is comprised of: receiving a data file comprised of text data representing at least one word spoken during the audio-visual data; and determining if the at least one word text data item is an element in a pre-determined word set correlated with heightened audience emotion or audience interest. ### Claim 6 (depends on 5) The method of claim 5 where the data file comprised of text data is further comprised of time data values that indicate an at least one time values during the audio visual data corresponding to where the at least one words were spoken. ### Claim 7 (depends on 1) The method of claim 1 further comprising: extracting an audio data portion of the audio-visual data stream; automatically detecting an at least one sound level in a corresponding at least one time slice of the audio data portion; and automatically detecting whether the at least one detected sound levels represent a pattern of sound levels correlate with a sound level pattern representing a higher emotive state of the audience. ### Claim 8 (depends on 1) The method of claim 1 further comprising: extracting a video data portion of the audio-visual data stream; automatically detecting at least one video data characteristics correlated with a heightened interest by the audience; determining an at least one time slice value corresponding to the at least one detected video characteristics; and storing in the data structure the at least one determined time slice values. ### Claim 9 (depends on 8) The method of claim 8 where the video data characteristics are comprised of one or both of variances in colour or luminosity. ### Claim 10 (depends on 8) The method of claim 8 further comprising: extracting from the video data portion data representing at least one facial expression; and determining whether the at least one extracted facial expression data is correlated with a pre-determined facial expression data correlated with heightened emotional interest by an audience member. ### Claim 11 (depends on 1) The method of claim 1 where the storing step is comprised of: automatically generating and storing a data structure comprised of data that lists an at least one start and a corresponding at least one a stop time of a corresponding at least one period of time during the audio-visual signal determined to be correlated with a heightened audience interest level. ### Claim 12 (depends on 1) The method of claim 1 further comprising: receiving at least one data representing a physiological response by an at least one viewer of the audio-visual signal; and storing an at least one data value representing the physiological response of the at least one viewer in a data structure that defines a correspondence between the at least one physiological data value and the at least one time slice. ### Claim 13 (depends on 1) The method of claim 1 where the extracting step is comprised of: determining an at least two time interval data comprised of a corresponding at least one start time slice value and a corresponding at least one stop time slice value where substantially the same video data during each of the at least two time intervals during the audio visual signal is repeated. ### Claim 14 (depends on 1) The method of claim 1 where the extracting step is comprised of extracting at least one parameter data representing an at least one audio sound level, and the storing step is comprised of storing at least one text data value corresponding to the at least one time slice and the calculating a correlation step is further comprised of using the text data in combination with the audio sound level data to determine the at least one time values of heightened audience interest. ### Claim 15 (depends on 13) The method of claim 13 where the storing step is comprised of storing at least one text data value corresponding to the at least one time slice and the calculating a correlation step is further comprised of using the text data in combination with the at least two video time interval data to determine the at least one time values of heightened audience interest. ### Claim 16 (depends on 1) The method of claim 1 where the at least one parameter data represents one of a colour or a luminosity variance. ### Claim 17 (depends on 1) The method of claim 1 where the at least one parameter data represents a facial expression characteristic. ### Claim 18 (depends on 1) The method of claim 1 where the determining step is comprised of determining an at least one time values of heightened audience interest by calculating a correlation among data variables representing at least one of: (i) an at least one audio spectral vector, (ii) an at least one audio sound level, (iii) an at least one colour, (iv) an at least one luminosity variance, (v) an at least one facial expression characteristic, (v) an at least one word data or (vi) at least one physiologic response. ### Claim 19 (depends on 18) The method of claim 18 where the determining step is comprised of calculating a matching score for one or more of the calculated correlations with the at least one predetermined audio spectral pattern, predetermined sound level pattern, predetermined colour or luminosity variance, predetermined facial characteristic and predetermined word set. ### Claim 20 (depends on 18) The method of claim 18 where the determining step is comprised of inputting the data variables into a machine learning engine and using an output of the machine learning engine for the calculating a correlation step. ### Claim 21 (depends on 20) The method of claim 20 where the machine learning engine is a neural network. ### Claim 22 (depends on 20) The method of claim 20 where the machine learning engine calculates a correlation matching score. ### Claim 23 (depends on 18) The method of claim 18 further comprising: using a video portion of the audio-visual data to detect an at least one time slices where the video portion has repeated an at least one earlier video portion of the audio-visual data by detecting an at least one self-similar region in the visual component of the audio-visual data. ### Claim 24 (depends on 11) The method of claim 11 further comprising: receiving the data structure comprised of at least one start and stop times; and generating an edited version of the audio-visual data by using the received data structure to select corresponding portions of the audio-visual data. ### Claim 25 (depends on 24) The method of claim 24 where the generating step is further comprised of: applying at least one mapping rule to an at least one time value in the data structure to specify an at least one edit point. ### Claim 26 (depends on 21) The method of claim 21 where the mapping rule is one of: setting the edit point to be at least one of: the time of a detected change in luminosity or colour that was detected just prior to the time value, or a predetermined pre-roll time before prior to the time value. ### Claim 27 (depends on 1) The method of claim 1 where the determining step is comprised of determining a logical condition where at substantially the same time slice values, either the audio spectral vectors or the audio sound levels are correlated with heightened audience interest and the word data are correlated with heightened audience interest. ### Claim 28 A method executed by a computer system for determining in an audio-visual signal encoding an event that is comprised of data representing an audience attending the event, periods of time that have heightened audience interest level, comprising: receiving a data representing the audio-visual signal; extracting from the received audio-visual data at least one parameter data corresponding to an at least one time-slice within the audio-visual signal, said at least one parameter data representing a magnitude of a sound level within a predetermined audio spectral range of the audio-visual signal within the corresponding an at least one time slice; receiving data representing an at least one word text data item representing at least one word spoken during the audio-visual data said at least one word text data item having a corresponding at least one time slice value; storing in a data structure the at least one parameter data and an at least one time slice value corresponding to the at least one parameter; storing in the data structure the at least one word text data item and corresponding at least one time slice value; determining an at least one time values of heightened audience interest by calculating a correlation between the extracted at least one parameter data representing a magnitude of a sound level within a predetermined audio spectral range of the audio-visual signal within the corresponding an at least one time slice, the stored at least one word text data items and an at least one pre-determined parameter data values and an at least one pre-determined text data items associated with a heightened audience interest level; and storing the determined at least one time values in a data structure associated with the audio-visual signal. ### Claim 29 (depends on 28) The method of claim 28 where the determining step is comprised of calculating a matching score for one or more of the calculated correlations with the at least one predetermined audio spectral pattern, predetermined sound level pattern, predetermined colour or luminosity variance, predetermined facial characteristic and predetermined word set. ### Claim 30 (depends on 28) The method of claim 28 where the determining step is comprised of inputting the data variables into a machine learning engine and using an output of the machine learning engine for the calculating a correlation step. ### Claim 31 (depends on 30) The method of claim 30 where the machine learning engine is a neural network. ### Claim 32 (depends on 30) The method of claim 30 where the machine learning engine calculates a correlation matching score. ### Claim 33 (depends on 28) The method of claim 28 further comprising: using a video portion of the audio-visual data to detect an at least one time slices where the video portion has repeated an at least one earlier video portion of the audio visual data by detecting an at least one self-similar region in the visual component of the audio-visual data. ### Claim 34 (depends on 28) The method of claim 28 further comprising: automatically generating and storing a data structure comprised of data that lists an at least one start and a corresponding at least one a stop time of a corresponding at least one period of time during the audio-visual signal determined to be correlated with a heightened audience interest level; and generating an edited version of the audio-visual data by using the stored data structure comprised of data that lists an at least one start and a corresponding at least one a stop time to select an at least one corresponding portions of the audio-visual data. ### Claim 35 (depends on 34) The method of claim 34 where the generating step is further comprised of: applying at least one mapping rule to an at least one time value in the data structure to specify an at least one edit point. ### Claim 36 (depends on 35) The method of claim 35 where the mapping rule is one of: setting the edit point to be the time of the scene transition that was detected just prior to the time value, setting the edit point to be a predetermined pre-roll time before prior to the time value.