# US 10,319,395 B2 · System and method for predictive generation of visual sequences > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,319,395 B2 - Document ID: US10319395B2 - Status: Granted - Kind: B2 - Application number: 15/457,758 - Filed: 2017-03-13 - Published: 2018-03-01 - Granted: 2019-06-11 - Earliest claimed: 2016-03-11 - Assignees: Limbic Media Corporation - Applicants: Limbic Media Corporation - Inventors: Justin Thomas Love; Manjinder Singh Benning; Adam Elliot; Jakob Leben; Gabrielle Odowichuk - Web page: https://patentgrep.com/patent/US10319395B2 ## CPC - G10L 21/10: Transforming into visible information - G06F 3/16: Sound input; Sound output (speech processing G10L) - G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - H05B 47/12: by detecting audible sound - H05B 47/155: Coordinated control of two or more light sources - G10L 25/18: the extracted parameters being spectral information of each sub-band - G10L 25/51: for comparison or discrimination - G10L 25/90: Pitch determination of speech signals - Y02B 20/40: Control techniques providing energy savings, e.g. smart controller or presence detection - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H05B: ELECTRIC HEATING; ELECTRIC LIGHT SOURCES NOT OTHERWISE PROVIDED FOR; CIRCUIT ARRANGEMENTS FOR ELECTRIC LIGHT SOURCES, IN GENERAL - Y02B: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO BUILDINGS, e.g. HOUSING, HOUSE APPLIANCES OR RELATED END-USER APPLICATIONS ## Abstract A system and method for predictively generating visual experiences based on streaming audio is disclosed. More specifically, the present invention is directed to systems and apparatus for analyzing streaming audio and predictively mapping the information in the stream to a sequence of visual patterns generated by a lighting system in a manner that induces a perceptual association between the streaming audio and visual patterns. ## Claims ### Claim 1 A method for controlling one or more light sources, comprising: receiving, in a buffer, a streaming audio signal; receiving, from the buffer, a buffer contents signal representing at least a portion of the streaming audio signal in the buffer; analyzing the buffer contents signal; generating a lighting control signal based on said analysis; outputting a real-time audio signal from the buffer; and outputting the lighting control signal in synchronization with the real-time audio signal; wherein: the real-time audio signal comprises an impulsive sound feature having a first duration; and said generating comprises generating a corresponding feature in the lighting control signal having a second duration that is longer than the first duration, thereby adapting a part of the lighting control signal corresponding to the impulsive sound feature to a human's visual response time. ### Claim 2 (depends on 1) The method of claim 1, wherein the second duration is three orders of magnitude longer than the first duration. ### Claim 3 (depends on 1) The method of claim 1, wherein said analysis determines whether the streaming audio signal includes an instance of a human voice. ### Claim 4 (depends on 1) The method of claim 1, wherein said analysis determines whether the streaming audio signal includes music. ### Claim 5 (depends on 1) The method of claim 1, further comprising storing pre-analyzed audio signals and their corresponding lighting control signals in a database, wherein the analyzing step comprises: identifying, in the database, a match for said portion of the streaming audio signal; retrieving, from the database, the corresponding lighting control signal for said match; and basing the output lighting control signal on the retrieved corresponding lighting control signal. ### Claim 6 (depends on 1) The method of claim 1, further comprising: storing previously analyzed audio signals and their corresponding lighting control signals in a database, in a format representing a light show; and retrieving the stored signals to perform the light show. ### Claim 7 (depends on 1) The method of claim 1, wherein said analysis comprises: detection and classification of music genre or music pattern based on one or more initial bars of a track in the streaming audio signal; and retrieval of a lighting control signal corresponding to the classification from a database. ### Claim 8 (depends on 1) The method of claim 1, further comprising: analyzing an upcoming portion of a track, a whole track, or a playlist in the buffer contents signal; planning a light show based on said upcoming portion of a track, a whole track, or a playlist; and maintaining continuity of the light show between successive tracks, based on said upcoming portion of a track, a whole track, or a playlist. ### Claim 9 (depends on 1) The method of claim 1, further comprising: analyzing a past portion of a track or whole track received in the buffer; and maintaining continuity of a light show between successive tracks, based on said past portion of a track or whole track received in the buffer. ### Claim 10 (depends on 1) The method of claim 1, further comprising: detecting a frequency range of the buffer contents signal; and dividing the buffer contents signal into sub-octave bins spanning the frequency range; wherein generating the lighting control signal is based on a content of each said bin. ### Claim 11 (depends on 1) The method of claim 1, wherein the lighting control signal includes a specification of color and a specification of brightness for at least one light source. ### Claim 12 (depends on 1) The method of claim 1, wherein the lighting control signal includes a specification of color and a specification of brightness for each of a plurality of light sources based on a physical location of each of the light sources. ### Claim 13 (depends on 1) The method of claim 1, wherein said lighting control signal is based on one or more of: frequency analysis; detection and classification of a percussive sound; detection of a melodic sound; strength of a melodic sound; a further received signal; a musical key; rhythmic dissonance; harmonic dissonance; tempo detection; beat detection; beat strength estimation; song transition; inter-song transition; a non-musical event; artificial intelligence; an interaction of multiple intelligent agents; an introduction of random colors; meta data of the streaming audio signal; an audience response; a sound envelope; maximizing a response of a lighting display system in relation to the real-time audio signal; varying a frequency range of a sub-octave bin used in said analysis; detection and classification of music genre or music pattern; detection and classification of speech style or pattern; a linear mapping of a sound feature in the streamed audio signal to the lighting control signal; and a non-linear mapping of a sound feature in the streamed audio signal to the lighting control signal. ### Claim 14 A system for controlling one or more light sources, comprising: a buffer configured to: receive a streaming audio signal; and output a real-time audio signal; and a predictive mapping unit comprising a processor and a computer readable memory storing computer readable instructions which, when executed by the processor, cause the predictive mapping unit to: receive, from the buffer, a buffer contents signal representing at least a portion of the streaming audio signal in the buffer; analyze the buffer contents signal; generate a lighting control signal based on said analysis; and output the lighting control signal in synchronization with the real-time audio signal; wherein: the real-time audio signal comprises an impulsive sound feature having a first duration; and said generating comprises generating a corresponding feature in the lighting control signal having a second duration that is longer than the first duration, thereby adapting a part of the lighting control signal corresponding to the impulsive sound feature to a human's visual response time. ### Claim 15 (depends on 14) The system of claim 14, wherein the predictive mapping unit further comprises: a frequency analyzer, defined by the computer readable instructions, that receives the buffer contents signal and outputs a frequency domain signal; and a binning module, defined by the computer readable instructions, that receives the frequency domain signal and outputs a plurality of sub-octave bin signals. ### Claim 16 (depends on 15) The system of claim 15, wherein the predictive mapping unit further comprises: a temporal filter that outputs an envelope analysis signal; a feature extractor, defined by the computer readable instructions, that is configured to detect one or more of genre; musical key; rhythmic dissonance; harmonic dissonance; tempo; beat; beat strength; song transition; inter-song transition; melody; melody strength; and percussive elements; and to output a corresponding feature signal; and a non-music detector, defined by the computer readable instructions, configured to detect one or more of speech; a whistle; a yell; clapping; singing; wind; and rain; and to output a non-musical event signal. ### Claim 17 (depends on 14) The system of claim 14, wherein the predictive mapping unit further comprises a color space mapper, defined by the computer readable instructions, configured to map the lighting control signal through a subset of color space that is delimited by chromaticities and maximum spectral radiant intensities of one or more light sources to which the lighting control signal is output.