Methods and apparatus for use in sound replacement with automatic synchronization to images

A data processing system for audio and video data, comprising a non-transitory computer readable medium having thereon:

• digitized audio and moving picture video data for providing a first audio data stream synchronized with corresponding visual events in a moving picture video data stream;
• timing data representative of a plurality of selected times in a running of said synchronized first audio data stream and said video data stream;
• audio feature data for providing a data stream of time-varying features abstracted from at least a selected portion of said first audio data stream and representative of audible time-varying acoustic features;
• an audio feature analyzer for abstracting a corresponding stream of time-varying features from an input second audio data stream;
• a timing analysis and waveform editing processor adapted to determine timing differences between said streams of time-varying features and to utilize said timing differences to edit said input second audio data stream and produce edited input audio data having acoustic features aligned with the acoustic features of said selected portion of said first audio data stream;
• and
• a playback control module adapted to control running of said synchronized audio data and video data streams with said edited input second audio data such that said edited input second audio data replaces said selected portion and is synchronized with said corresponding visual events in said video data stream.

A method of providing a processing system for audio and video data, comprising the steps of:

• storing in a non-transitory computer readable medium digitized audio and moving picture video data for providing a first audio data stream synchronized with corresponding visual events in a moving picture video data stream;
• storing in the non-transitory computer readable medium timing data representative of a plurality of selected times in a running of said synchronized audio and video data streams;
• selecting at least a portion of said first audio data stream;
• abstracting from the selected portion of said first audio data stream, using a programmable processor executing computer-executable instructions, audio feature data for providing a data stream of time-varying features, said audio feature data being representative of audible time-varying acoustic features;
• storing the abstracted audio feature data in the non-transitory computer readable medium;
• storing in the non-transitory computer readable medium an audio feature analyzer for abstracting a corresponding stream of time-varying features from an input second audio data stream;
• storing in the non-transitory computer readable medium a timing analysis and waveform editing processor adapted to determine timing differences between said data stream of time-varying features and corresponding features abstracted from said input second audio data stream and to produce an editing input second audio data stream having acoustic features aligned with the acoustic features of said selected portion of said first audio data stream;
• and
• storing in the non-transitory computer readable medium a playback control module for controlling running of said synchronized audio data and video data streams with edited second input audio data from said processor such that said edited input second audio data replaces said selected portion and is synchronized with said corresponding visual events in said moving picture video data stream.

A method of processing audio data, comprising the steps of:

• providing an original first audio data stream synchronized with corresponding visual events in a moving picture video data stream;
• selecting at least a portion of said original first audio data stream;
• storing in a non-transitory computer readable medium an input second audio data stream substantially in synchronization with a portion of said video data stream corresponding to the selected portion of said original audio data stream;
• abstracting from said input second audio data stream a stream of time-varying features of the input second audio data stream, said time-varying features being representative of audible time-varying acoustic features;
• comparing, using a programmable processor executing computer-executable instructions, the abstracted stream of time-varying features from said input second audio data stream with a corresponding stream of time-varying features abstracted from said selected portion of said original first audio data stream and determining timing differences between said streams of time-varying features;
• editing said second audio data stream utilizing said timing differences to produce edited input second audio data having acoustic features aligned with the acoustic features of said selected portion of said first audio data stream;
• and
• running said portion of said video data stream with said edited input second audio data such that said edited input second audio data replaces said selected portion and is synchronized with said corresponding visual events in said moving picture video data stream.

A non-transitory computer readable medium having thereon audio and video data processing software comprising:

• a feature analysis program adapted to derive from audio data feature data representative of audible time-varying acoustic features of the audio data;
• a comparison and timing program adapted to compare first feature data derived from first audio data synchronized with corresponding visual events in moving picture video data with second feature data derived from second audio data and to determine timing differences between the first and second feature data;
• an editing program adapted to edit the second audio data in dependence upon said timing differences such as to provide edited second audio data in a synchronous relation to said first audio data;
• and
• a streaming program adapted to synchronously output said video data and said edited second audio data while muting said first audio data whereby said edited second audio data replaces said first audio data and is synchronized with said corresponding visual events in said video data.

A method of producing an audio signal as a replacement for a pre-recorded audio signal synchronized with a pre-recorded video signal, the method comprising the steps of:

• populating a database in a non-transitory computer readable medium with data representing the pre-recorded audio signal synchronized with the pre-recorded video signal, and data indicative of the timing of the pre-recorded audio signal relative to the pre-recorded video signal;
• receiving and storing in the non-transitory computer readable medium an input audio signal having timing similar to that of the pre-recorded audio signal;
• providing first feature data that is derived from the pre-recorded audio signal and encodes audible time-varying acoustic features of a predetermined type and the timing thereof within the pre-recorded audio signal;
• providing second feature data that is derived from the input audio signal and encodes audible time-varying acoustic features of the predetermined type and the timing thereof within the input audio signal;
• determining timing differences existing between the first feature data and the second feature data;
• altering the timing of the acoustic features of the stored input audio signal to match the timing of the corresponding acoustic features of the pre-recorded audio signal to produce thereby an edited replacement audio signal;
• and
• selecting a start time for starting outputting of the edited replacement audio signal to coincide with a corresponding start time of the pre-recorded audio signal relative to the pre-recorded video signal and for starting muting of the pre-recorded audio signal.

An apparatus for producing an audio signal as a replacement for a pre-recorded audio signal synchronized with a pre-recorded video signal, the apparatus comprising:

• one or more non-transitory computer readable mediums;
• a database populated with data representing the pre-recorded audio signal synchronized with the pre-recorded video signal, and data indicative of the timing of the pre-recorded audio signal relative to the pre-recorded video signal;
• means for deriving from the pre-recorded audio signal first feature data encoding audible time-varying acoustic features of a predetermined type and the timing thereof in the pre-recorded audio signal;
• means for receiving and storing an input audio signal having timing similar to that of the pre-recorded audio signal;
• means for deriving from the input audio signal second feature data encoding audible time-varying acoustic features of said predetermined type and the timing thereof within the input audio signal;
• means for determining timing differences existing between the first feature data and the second feature data;
• means for altering the timing of the acoustic features of the stored input audio signal to match the timing of the corresponding acoustic features of the pre-recorded audio signal and producing thereby an edited replacement audio signal;
• and means for selecting a start time for starting outputting of the edited replacement audio signal to coincide with a corresponding start time of the pre-recorded audio signal relative to the pre-recorded video signal and for starting muting of the pre-recorded audio signal.

An apparatus for producing an audio signal as a replacement audio signal for a pre-recorded audio signal synchronized with a pre-recorded video signal, the apparatus comprising:

• one or more non-transitory computer readable mediums;
• a database populated with data representing the pre-recorded audio signal synchronized with the pre-recorded video signal, and data indicative of the timing of the pre-recorded audio signal relative to the pre-recorded video signal;
• an audio system for receiving and storing an input audio signal having timing similar to that of the pre-recorded audio signal;
• a comparing unit deriving from the pre-recorded audio signal first feature data encoding audible time-varying acoustic features of a predetermined type and the timing thereof in the pre-recorded audio signal and deriving from the input audio signal second feature data encoding audible time-varying acoustic features of the predetermined type and the timing thereof within the input audio signal, and determining timing differences existing between the first feature data and the second feature data;
• an editing unit for altering the timing of the acoustic features of the stored input audio signal to match the timing of the corresponding acoustic features of the pre-recorded audio signal to produce thereby an edited replacement audio signal;
• and a timer for selecting a start time for starting outputting of the edited replacement audio signal to coincide with a corresponding start time of the pre-recorded audio signal relative to the pre-recorded video signal and for starting muting of the pre-recorded audio signal.

An apparatus for producing an audio signal as a replacement audio signal for a pre-recorded audio signal synchronized with a pre-recorded video signal, the apparatus comprising:

• one or more non-transitory computer readable mediums;
• a database populated with data representing the pre-recorded audio signal synchronized with the pre-recorded video signal, and data indicative of the timing of the pre-recorded audio signal relative to the pre-recorded video signal;
• an audio system for receiving and storing an input audio signal having timing similar to that of the pre-recorded audio signal;
• a first acoustic feature extracting unit for deriving from the pre-recorded audio signal first feature data encoding audible time-varying acoustic features of a predetermined type and the timing thereof in the pre-recorded audio signal;
• a second acoustic feature extracting unit for deriving from the input audio signal second feature data encoding audible time-varying acoustic features of the predetermined type and the timing thereof within the input audio signal;
• a comparator for determining timing differences existing between the first feature data and the second feature data;
• an editing unit for altering the timing of the acoustic features of the stored input audio signal to match the timing of the corresponding acoustic features of the pre-recorded audio signal to produce thereby an edited replacement audio signal;
• and a timer for selecting a start time for starting outputting of the edited replacement audio signal to coincide with a corresponding start time of the pre-recorded audio signal relative to the pre-recorded video signal and for starting muting of the pre-recorded audio signal.

A method of producing an audio signal as a replacement for a pre-recorded audio signal synchronized with a pre-recorded video signal, the method comprising the steps of:

• populating a database in a non-transitory computer readable medium with data representing the pre-recorded audio signal synchronized with the pre-recorded video signal, and data indicative of the timing of the pre-recorded audio signal relative to the pre-recorded video signal;
• receiving and storing in a non-transitory computer readable medium an input audio signal having timing similar to that of the pre-recorded audio signal;
• providing first feature data that is derived from the pre-recorded audio signal and encodes audible time-varying acoustic features of a predetermined type and the timing thereof within the pre-recorded audio signal,
• providing second feature data that is derived from the input audio signal and encodes audible time-varying acoustic features of the predetermined type and the timing thereof within the input audio signal;
• determining timing differences existing between the first feature data and the second feature data;
• altering acoustic features of the stored input audio signal to produce thereby an edited replacement audio signal such that the timing of the acoustic features of the predetermined type of the edited replacement audio signal matches the timing of the acoustic features of the pre-recorded audio signal;
• and
• selecting a start time for starting outputting of the edited replacement audio signal to coincide with a corresponding start time of the pre-recorded audio signal relative to the pre-recorded video signal and for starting muting of the pre-recorded audio signal.

An apparatus for producing an audio signal as a replacement audio signal for a pre-recorded audio signal synchronized with a pre-recorded video signal, the apparatus comprising:

• one or more non-transitory computer readable mediums;
• a database populated with data representing the pre-recorded audio signal synchronized with the pre-recorded video signal, and data indicative of the timing of the pre-recorded audio signal relative to the pre-recorded video signal;
• an audio system for receiving and storing an input audio signal having timing similar to that of the pre-recorded audio signal;
• a comparing unit deriving from the pre-recorded audio signal first feature data encoding audible time-varying acoustic features of a predetermined type and the timing thereof in the pre-recorded audio signal and deriving from the input audio signal second feature data encoding audible time-varying acoustic features of the predetermined type and the timing thereof within the input audio signal, and determining timing differences existing between the first feature data and the second feature data;
• an editing unit for altering the acoustic features of the stored input audio signal to produce thereby an edited replacement audio signal having acoustic features of the predetermined type with timing matching the timing of the corresponding acoustic features of the pre-recorded audio signal;
• and a timer for selecting a start time for starting outputting of the edited replacement audio signal to coincide with a corresponding start time of the pre-recorded audio signal relative to the pre-recorded video signal and for starting muting of the pre-recorded audio signal.

An apparatus for producing an audio signal as a replacement audio signal for a pre-recorded audio signal synchronized with a pre-recorded video signal, the apparatus comprising:

• one or more non-transitory computer readable mediums;
• a database populated with data representing the pre-recorded audio signal synchronized with the pre-recorded video signal, and data indicative of the timing of the pre-recorded audio signal relative to the pre-recorded video signal;
• an audio system for receiving and storing an input audio signal having timing similar to that of the pre-recorded audio signal;
• a first acoustic feature extracting unit for deriving from the pre-recorded audio signal first feature data encoding audible time-varying acoustic features of a predetermined type and the timing thereof in the pre-recorded audio signal;
• a second acoustic feature extracting unit for deriving from the input audio signal second feature data encoding audible time-varying acoustic features of the predetermined type and the timing thereof within the input audio signal;
• a comparator for determining timing differences existing between the first feature data and the second feature data;
• an editing unit for altering the acoustic features of the stored input audio signal to produce thereby an edited replacement audio signal having acoustic features of the predetermined type with timing matching the timing of the corresponding acoustic features of the pre-recorded audio signal;
• and a timer for selecting a start time for starting outputting of the edited replacement audio signal to coincide with a corresponding start time of the pre-recorded audio signal relative to the pre-recorded video signal and for starting muting of the pre-recorded audio signal.