Performance retiming effects on synchronized data in an editing system

A computer program product, comprising:

• a computer readable medium;
• computer program instructions stored in the computer readable medium that, when executed by a computer, instruct the computer to perform a method for producing a retiming effect on a clip of synchronized audio data and video data to produce a retimed clip of synchronized audio and video data in the audiovisual work, comprising:
• associating a definition of a retiming function for a rampable retiming effect that maps output times to input times with the clip of synchronized audio data and video data;
• processing the synchronized audio data and video data according to the retiming function to produce the retimed clip, wherein processing comprises:
• for each output time for an audio sample,
• determining a corresponding input time from the output time and the retiming function;
• and
• computing an output audio sample at the output time based on at least the audio data in the neighborhood of the corresponding input time using a resampling function;
• and
• for each output time for a video sample,
• determining a corresponding input time from the output time and the retiming function, such that input times determined for output times for video samples correspond to input times determined for the same output times for audio samples;
• and
• computing an output video sample at the output time based on at least the video data in the neighborhood of the corresponding input time using a resampling function.

An editing system for editing an audiovisual work and for producing a retiming effect on a clip of synchronized audio data and video data to produce a retimed clip of synchronized audio and video data in the audiovisual work, comprising:

• means for relating a definition of a retiming function for a rampable retiming effect that maps output times to input times with the clip of synchronized audio data and video data;
• means for processing the synchronized audio data and video data according to the retiming function to produce the retimed clip, comprising:
• means for computing, for each output time for an audio sample, a corresponding input time from the output time and the retiming function;
• means for computing an output audio sample at the output time based on at least the audio data in the neighborhood of the corresponding input time using a resampling function;
• means for computing, for each output time for a video sample, a corresponding input time from the output time and the retiming function, such that input times determined for output times for video samples correspond to input times determined for the same output times for audio samples;
• and
• means for computing an output video sample at the output time based on at least the video data in the neighborhood of the corresponding input time using a resampling function.

An editing system for editing an audiovisual work and for producing a retiming effect on a clip of synchronized audio data and video data to produce a retimed clip of synchronized audio and video data in the audiovisual work, comprising:

• an editing interface allowing a user to associate a definition of a retiming function for a rampable retiming effect that maps output times to input times with the clip of synchronized audio data and video data;
• an audio retiming module having an input for receiving the definition of the retiming function and an input for receiving the audio data, and an output providing retimed audio data such that, for each output time for an audio sample, an output audio sample is computed using a resampling function based on at least the audio data in a neighborhood of a corresponding input time according to the retiming function;
• and
• a video retiming module having an input for receiving the definition of the retiming function and an input for receiving the video data, and an output providing retimed video data such that, for each output time for a video sample, an output video sample is computed using a resampling function based on at least the video data in a neighborhood of a corresponding input time according to the retiming function, wherein input times determined for output times for video samples correspond to input times determined for the same output times for audio samples.

A digital entertainment product, comprising:

• a computer readable medium;
• information stored on the computer readable medium that, when read by a computer, indicates to the computer a retimed clip of synchronized audio data and video data, produced according to a process for retiming a clip of synchronized audio data and video data to produce a retimed clip of synchronized audio and video data in the audiovisual work, the process comprising:
• associating a definition of a retiming function for a rampable retiming effect that maps output times to input times with the clip of synchronized audio data and video data;
• processing the synchronized audio data and video data according to the retiming function to produce the retimed clip, wherein processing comprises:
• for each output time for an audio sample,
• determining a corresponding input time from the output time and the retiming function;
• and
• computing an output audio sample at the output time based on at least the audio data in the neighborhood of the corresponding input time using a resampling function;
• and
• for each output time for a video sample,
• determining a corresponding input time from the output time and the retiming function, such that input times determined for output times for video samples correspond to input times determined for the same output times for audio samples;
• and
• computing an output video sample at the output time based on at least the video data in the neighborhood of the corresponding input time using a resampling function.

In a system for editing an audiovisual work including a sequence of a plurality of clips of audiovisual data, a method for producing a retiming effect on a clip of synchronized audio data and video data to produce a retimed clip of synchronized audio and video data in the audiovisual work, comprising:

• associating a definition of a retiming function for a rampable retiming effect that maps output times to input times with the clip of synchronized audio data and video data;
• processing the video data of the clip according to the retiming function to produce a retimed video clip, wherein processing comprises:
• for each output time for a video sample,
• determining a corresponding input time from the output time and the retiming function;
• and
• computing an output video sample at the output time based on at least the video data in the neighborhood of the corresponding input time using a resampling function;
• placing the retimed video clip in the audiovisual work;
• and
• exporting the audiovisual work to an audio editing system, including the definition of the retiming function, for processing the audio data of the clip according to the retiming function, such that an input time determined for each output time for video samples corresponds to an input time determined for the same output time for audio samples.

In a system for editing an audiovisual work including a sequence of a plurality of clips of audiovisual data, a method for producing a retiming effect on a clip of synchronized audio data and video data to produce a retimed clip of synchronized audio and video data in the audiovisual work, comprising:

• receiving a definition of a retiming function for a rampable retiming effect that maps output times to input times associated with the clip of synchronized audio data and video data;
• receiving the audio data;
• receiving the audiovisual work including a retimed video clip processed according to the retiming function;
• processing the audio data according to the retiming function to produce a retimed audio clip, wherein processing comprises:
• for each output time for an audio sample,
• determining a corresponding input time from the output time and the retiming function, such that an input time determined for each output time for video samples corresponds to an input time determined for the same output time for audio samples;
• and
• computing an output audio sample at the output time based on at least the audio data in the neighborhood of the corresponding input time using a resampling function;
• and
• synchronizing the retimed audio clip with the retimed video clip in the audiovisual work.

In a system for editing an audiovisual work including a sequence of a plurality of clips of audiovisual data, a method for producing a retiming effect on a clip of synchronized audio data and video data to produce a retimed clip of synchronized audio and video data in the audiovisual work, comprising:

• receiving a definition of a retiming function for a rampable retiming effect that maps output times to input times associated with the clip of synchronized audio data and video data;
• receiving the video data;
• receiving the audiovisual work including a retimed audio clip processed according to the retiming function;
• processing the video data according to the retiming function to produce a retimed video clip, wherein processing comprises:
• for each output time for a video sample,
• determining a corresponding input time from the output time and the retiming function, such that an input time determined for each output time for video samples corresponds to an input time determined for the same output time for audio samples;
• and
• computing an output video sample at the output time based on at least the video data in the neighborhood of the corresponding input time using a resampling function;
• and
• synchronizing the retimed video clip with the retimed audio clip in the audiovisual work.

In a system for editing an audiovisual work, a method for defining a retiming effect applied to audio data and video data to produce a retimed clip of synchronized audio and video data in the audiovisual work, comprising:

• associating a mapping of a plurality of video events in the video data and a corresponding plurality of audio events in the audio data to a corresponding plurality of output times in the retimed clip;
• processing the audio data according to the retiming function to produce retimed audio data;
• processing the video data according to the retiming function to produce retimed video data;
• placing the retimed audio data and retimed video data in the audiovisual work as the retimed clip of synchronized audio and video data.

A computer program product, comprising:

• a computer readable medium;
• computer program instructions stored on the computer readable medium that, when executed by a computer, instructs the computer to perform a method for defining a retiming effect applied to audio data and video data to produce a retimed clip of synchronized audio and video data in the audiovisual work, comprising:
• associating a mapping of a plurality of video events in the video data and a corresponding plurality of audio events in the audio data to a corresponding plurality of output times in the retimed clip;
• processing the audio data according to the retiming function to produce retimed audio data;
• processing the video data according to the retiming function to produce retimed video data;
• placing the retimed audio data and retimed video data in the audiovisual work as the retimed clip of synchronized audio and video data.

An editing system for defining a retiming effect applied to audio data and video data to produce a retimed clip of synchronized audio and video data in the audiovisual work, comprising:

• means for mapping a plurality of video events in the video data and a corresponding plurality of audio events in the audio data to a corresponding plurality of output times in the retimed clip;
• means for processing the audio data according to the retiming function to produce retimed audio data;
• means for processing the video data according to the retiming function to produce retimed video data;
• and
• means for inserting the retimed audio data and retimed video data in the audiovisual work as the retimed clip of synchronized audio and video data.

A method for applying a rampable retiming effect to synchronized streams of temporal media data, include a first stream of a first media type and a second stream of a second media type different from the first media type, comprising:

• associating a retiming function for the rampable retiming effect with the synchronized streams, wherein the retiming function maps output times to input times;
• processing the synchronized streams according to the retiming function, wherein processing comprises:
• for each output time for the first stream,
• determining a corresponding input time from the output time and the retiming function;
• and
• computing an output sample for the first stream at the output time from at least data in a neighborhood of the corresponding input time, using a resampling function for the first media type;
• and
• for each output time for the second stream,
• determining a corresponding input time from the output time and the retiming function, such that an input time determined for an output time for a sample of the second stream corresponds to an input time determined for the same output times for a sample of the first stream;
• and
• computing an output sample for the second stream at the output time from at least data in a neighborhood of the corresponding input time, using a resampling function for the second media type.

A method for applying a rampable retiming effect to synchronized streams of temporal media data, wherein the streams include a first stream of a first media type and a second stream of a second media type, comprising:

• associating a retiming function for the rampable retiming effect with the synchronized streams, wherein the retiming function defines a mapping of output times to input times, wherein the mapping has a corresponding temporal resolution of greater than or equal to a temporal resolution of both of the synchronized streams;
• processing the synchronized streams according to the retiming function, wherein processing comprises:
• for each output time for the first stream,
• determining a corresponding input time from the output time and the mapping defined by the retiming function;
• and
• computing an output sample for the first stream at the output time from at least data in a neighborhood of the corresponding input time, using a resampling function for the first media type;
• and
• for each output time for the second stream,
• determining a corresponding input time from the output time and the mapping defined by the retiming function;
• and
• computing an output sample for the second stream at the output time from at least data in a neighborhood of the corresponding input time, using a resampling function for the second media type.

A method for applying a rampable retiming effect to temporal media data synchronized with metadata, comprising:

• associating a retiming function for the rampable retiming effect with the temporal media data, wherein the retiming function maps output times to input times;
• processing the temporal media data according to the retiming function, wherein processing comprises, for each output time,
• determining a corresponding input time from the output time and the retiming function;
• and
• computing an output sample of the temporal media data at the output time from at least temporal media data in a neighborhood of the corresponding input time, using a resampling function for the temporal media data;
• and
• processing the metadata to synchronize the metadata with the retimed temporal media data.