Back to results
US 9,756,281 B2Granted
Apparatus and method for audio based video synchronization
AssigneeGoPro, Inc.
InventorDavid K. Tcheng
Life of this patent
Estimating the term…
Abstract
Multiple video recordings may be synchronized using audio features of the recordings. A synchronization process may compare energy tracks of each recording within a multi-resolution framework to correlate audio features of one recording to another.
Claims21
A method for synchronizing audio tracks, the method comprising:
- (a) obtaining a first feature track for the first audio track including information representing sound captured from a live occurrence, the first feature track representing individual feature samples associated with the sound captured on the first audio track from the live occurrence, the first feature track conveying feature magnitude for individual sampling periods of a first sampling period length;
- (b) obtaining a second feature track for the first audio track, the second feature track representing individual feature samples associated with the sound captured on the first audio track from the live occurrence, the second feature track conveying feature magnitude for individual sampling periods of a second sampling period length, the second sampling period length being different than the first sampling period length;
- (c) obtaining a third feature track for the second audio track including information representing sound captured from the live occurrence, the third feature track representing individual feature samples associated with the sound captured on the second audio track from the live occurrence, the third feature track conveying feature magnitude for individual sampling periods of the first sampling period length;
- (d) obtaining a fourth feature track for the second audio track, the fourth feature track representing individual feature samples associated with the sound captured on the second audio track from the live occurrence, the fourth feature track conveying feature magnitude for individual sampling periods of the second sampling period length;
- (e) comparing at least a portion of the first feature track against at least a portion of the third feature track to correlate one or more features in the first feature track with one or more features in the third feature track, the correlated features being identified as potentially representing features in the same sounds from the live occurrence;
- (f) comparing a portion of the second feature track against a portion of the fourth feature track to correlate one or more features in the second feature track with one or more features in the fourth feature track, the portion of the second feature track and the portion of the fourth feature track being selected for the comparison based on the correlation of features between the first feature track and the third feature track;
- (g) determining, from the correlations of features in the feature tracks, a temporal alignment estimate between the first audio track and the second audio track, the temporal alignment estimate reflecting an offset in time between commencement of sound capture for the first audio track and commencement of sound capture for the second audio track;
- (h) obtaining a temporal alignment threshold;
- (i) determining whether to continue comparing feature tracks associated with the first audio track and the second audio track based on the comparison of the temporal alignment estimate and the temporal alignment threshold, including determining to not continue comparing feature tracks associated with the first audio track and the second track in response to the temporal alignment estimate being smaller than the alignment threshold;
- (j) responsive to the determining to not continue comparing feature tracks associated with the first audio track and the second audio track, using the temporal alignment estimate to synchronize the first audio track with the second audio track;
- (k) responsive to the determining to continue comparing feature tracks associated with the first audio track and the second audio track, iterating after operations (a) through (k).
8 more elements
A system for synchronizing audio tracks, the system comprising:
- one or more physical computer processors is further configured by computer readable instructions to:
- (a) obtain a first feature track for the first audio track including information representing sound captured from a live occurrence, the first feature track representing individual feature samples associated with the sound captured on the first audio track from the live occurrence, the first feature track conveying feature magnitude for individual sampling periods of a first sampling period length;
- (b) obtain a second feature track for the first audio track, the second feature track representing individual feature samples associated with the sound captured on the first audio track from the live occurrence, the second feature track conveying feature magnitude for individual sampling periods of a second sampling period length, the second sampling period length being different than the first sampling period length;
- (c) obtain a third feature track for the second audio track, the third feature track representing individual feature samples associated with the sound captured on the second audio track from the live occurrence, the third feature track conveying feature magnitude for individual sampling periods of the first sampling period length;
- (d) obtain a fourth feature track for the second audio track, the fourth feature track representing individual feature samples associated with the sound captured on the second audio track from the live occurrence, the fourth feature track conveying feature magnitude for individual sampling periods of the second sampling period length;
- (e) compare at least a portion of the first feature track against at least a portion of the third feature track to correlate one or more features in the first feature track with one or more features in the third feature track, the correlated features being identified as potentially representing features in the same sounds from the live occurrence;
- (f) compare a portion of the second feature track against a portion of the fourth feature track to correlate one or more features in the second feature track with one or more features in the fourth feature track, the portion of the second feature track and the portion of the fourth feature track being selected for the comparison based on the correlation of features between the first feature track and the third feature track;
- (g) determine, from the correlations of features in the feature tracks, a temporal alignment estimate between the first audio track and the second audio track, the temporal alignment estimate reflecting an offset in time between commencement of sound capture for the first audio track and commencement of sound capture for the second audio track;
- (h) obtain a temporal alignment threshold;
- (i) determine whether to continue comparing feature tracks associated with the first audio track and the second audio track based on the comparison of the temporal alignment estimate and the temporal alignment threshold, including determining to not continue comparing feature tracks associated with the first audio track and the second track in response to the temporal alignment estimate being smaller than the alignment threshold;
- (j) responsive to the determining to not continue comparing feature tracks associated with the first audio track and the second audio track, using the temporal alignment estimate to synchronize the first audio track with the second audio track;
- (k) responsive to the determining to continue comparing feature tracks associated with the first audio track and the second audio track, iterating after operations (a) through (k).
9 more elements
Dependencies are inferred from the claim text.