Methods and systems for synchronizing visualizations with audio streams

One or more computer-readable storage media having computer-readable instructions thereon which, when executed by a computer, implement a system comprising:

• an audio sample pre-processor configured to receive and pre-process audio samples before the samples are rendered by a media player, the pre-processing providing frequency data associated with each sample, wherein the frequency data is derived from the audio samples;
• an audio rendering object called by the media player to render visualizations corresponding to the audio samples provided by the one or more audio sources, wherein the audio rendering object has an associated target frame rate defining how frequently the visualizations are drawn, redrawn, or painted;
• one or more effects associated with the audio rendering object and configured to receive the frequency data and use the frequency data to render a visualization that is synchronized with an audio sample that is being rendered by the media player;
• multiple data structures configured to hold the frequency data, each data structure being associated with an audio sample;
• wherein the audio sample pre-processor comprises a timestamp module that provides a timestamp for each audio sample, each timestamp being maintained by a data structure associated with the audio sample, and further wherein the audio sample pre-processor is configured to:
• query a media player audio sample renderer for a time associated with an audio sample that is being currently rendered,
• use the time to ascertain a timestamp of an associated audio sample, the audio sample pre-processor further being configured to provide the frequency data of the associated audio sample to the one or more effects so that the frequency data can be used to render the visualization, and
• associate the timestamp with the frequency data based on a current rendering time of the audio sample and a number of other audio samples in a pipeline scheduled for playing on the media player;
• and
• wherein the audio sample pre-processor pre-processes the audio samples by using a Fast Fourier Transform to provide the frequency data.

One or more computer-readable storage media having computer-readable instructions thereon which, when executed by a computer, implement a system comprising:

• an audio sample pre-processor configured to receive and preprocess audio samples before the samples are rendered by a renderer that comprises part of a media player, the audio sample preprocessor preprocessing the samples to provide waveform data derived from each sample and a timestamp associated with each audio sample, the timestamp being assigned in accordance with when the audio sample is calculated to be rendered by the renderer;
• multiple data structures configured to hold the waveform data, each data structure being associated with an audio sample;
• an audio rendering object configured to call the audio sample pre-processor to ascertain the waveform data associated with an audio sample that is currently being rendered by the renderer, wherein the audio rendering object is called by the renderer to render visualizations corresponding to the audio samples, and wherein the audio rendering object has an associated target frame rate defining how frequently the visualizations are drawn, redrawn, or painted;
• the audio sample pre-processor being configured to ascertain the waveform data by querying the renderer for a time associated with the currently-rendered audio sample, and then using the time queried to identify a data structure having a timestamp that is nearest in value to the time queried;
• one or more effects associated with the audio rendering object and configured to receive the waveform data that is associated with the data structure having the timestamp that is nearest in value to the time queried, and use the waveform data to render a visualization that is synchronized with the audio sample that is being rendered by the renderer;
• the audio sample pre-processor being configured to associate the timestamp with the waveform data based on a current rendering time of the audio sample and a number of other audio samples in a pipeline scheduled for having on the renderer;
• and
• wherein the audio sample pre-processor comprises a Fast Fourier Transform that it utilizes to process the audio samples to provide the waveform data associated with the audio samples.

A system for processing audio samples comprising:

• a memory;
• a processor coupled to the memory;
• means for providing a timestamp module for assigning timestamps to audio samples that are to be rendered by a media player renderer, wherein a timestamp is associated with a frequency data extracted from a corresponding audio sample based on a current rendering time of the corresponding audio sample and a number of other audio samples in a pipeline scheduled for playing on a media player;
• means for providing a spectrum analyzer for processing the audio samples to derive the frequency data from the audio samples;
• means for providing multiple data structures each of which being associated with an audio sample, the data structures each containing timestamp data and frequency data for its associated audio sample;
• the system being configured to use the timestamp data to ascertain a data structure associated with an audio sample that is currently being rendered by the media player renderer and provide the frequency data associated with that audio sample so that the frequency data can be used to render a visualization associated with that audio sample;
• an audio rendering object called by the media player to render visualizations corresponding to the audio samples provided by one or more audio sources, wherein the audio rendering object has an associated target frame rate defining how frequently the visualizations are drawn, redrawn, or painted;
• and
• wherein the means for providing the spectrum analyzer comprises means for providing a Fast Fourier Transform that is utilized to provide the frequency data.

One or more computer-readable storage media having computer-readable instructions thereon which, when executed by a computer, implement a method comprising:

• receiving multiple audio samples;
• pre-processing the audio samples before they are rendered by a media player renderer, the pre-processing deriving characterizing data from each sample, wherein the characterizing data comprises frequency data and a timestamp based upon when the audio sample is calculated to be rendered by the media player renderer;
• associating the timestamp with the frequency data based on a current rendering time of the audio sample and a number of other audio samples in a pipeline scheduled for playing on the media player renderer;
• maintaining characterizing data for each audio sample in a data structure, wherein each audio sample is associated with the data structure;
• determining when an audio sample is being rendered by the media player renderer, the determining comprising:
• ascertaining a time associated with a currently-rendered audio sample;
• and
• selecting a data structure having a timestamp that is nearest the time;
• providing characterizing data associated with the selected data structure to a component configured to provide the visualization;
• responsive to the determining, using the characterizing data that is associated with the audio sample that is being rendered to provide a visualization by calling an audio rendering object which renders visualizations corresponding to the audio samples, wherein the audio rendering object has an associated target frame rate defining how frequently the visualizations are drawn, redrawn, or painted;
• and
• wherein said pre-processing comprises using a Fast Fourier Transform to provide the frequency data associated with the audio samples.

A system comprising:

• a memory;
• a processor coupled to the memory;
• means for receiving multiple audio samples;
• means for pre-processing the audio samples before they are rendered by a media player renderer, the pre-processing comprising at least (1) using a Fast Fourier Transform to derive frequency data from the samples, and (2) associating a timestamp with each sample;
• wherein the pre-processing is configured to associate the timestamp with the frequency data based on a current rendering time of the audio sample and a number of other audio samples in a pipeline scheduled for playing on the media player renderer;
• means for maintaining the frequency data and the timestamp for each sample in a data structure;
• means for determining when an audio sample is being rendered by the media player renderer by:
• ascertaining a time associated with a currently-rendered sample;
• and
• selecting a data structure having a timestamp that is nearest the time;
• means for providing characterizing data associated with the selected data structure to a component configured to provide the visualization;
• and
• means for using the characterizing data that is associated with the audio sample that is being rendered, responsive to the determining when the audio sample is being rendered, to provide a visualization by calling an audio rendering object which renders visualizations corresponding to the audio samples, wherein the audio rendering object has an associated target frame rate defining how frequently the visualizations are drawn, redrawn, or painted.