Back to results
US 10,762,887 B1Granted
Smart voice enhancement architecture for tempo tracking among music, speech, and noise
AssigneeDialpad, Inc.
InventorsQian-Yu Tang, John Rector
Life of this patent
Estimating the term…
Abstract
Audio data describing an audio signal may be received and used to determine a set of frames of the audio signal. A plurality of note onsets in the set of frames may be identified based on spectral energy of the audio signal in the set of frames. One or more tempos may be computed based on the identified plurality of note onsets. The one or more tempos may be validated based on a tempo validation condition. One or more music states of the audio signal may be determined based on the validated one or more tempos. Audio enhancement of the audio signal may be modified based on the one or more determined states of the audio signal.
Claims25
A computer-implemented method, comprising:
- receiving, by a computing device, audio data describing an audio signal;
- determining, by the computing device, a set of frames of the audio signal using the audio data;
- identifying, by the computing device, a plurality of note onsets in the set of frames based on spectral energy of the audio signal in the set of frames;
- computing, by the computing device, one or more tempos based on the identified plurality of note onsets;
- validating, by the computing device, the one or more tempos based on a tempo validation condition;
- determining, by the computing device, one or more music states of the audio signal based on the validated one or more tempos;
- and
- modifying, by the computing device, audio enhancement of the audio signal based on the one or more music states, wherein modifying the audio enhancement of the audio signal comprises ceasing noise cancelation of the audio signal.
5 more elements
A computer system comprising:
- at least one processor;
- and
- a non-transitory computer memory storing instructions that, when executed by the at least one processor, cause the computer system to perform operations comprising:
- receiving audio data describing an audio signal;
- determining a set of frames of the audio signal using the audio data;
- identifying a plurality of note onsets in the set of frames based on spectral energy of the audio signal in the set of frames;
- computing one or more tempos based on the identified plurality of note onsets;
- validating the one or more tempos based on a tempo validation condition;
- and
- determining one or more music states of the audio signal based on the validated one or more tempos comprising:
- detecting that a state duration of a particular music state of the one or more music states has elapsed;
- transitioning a finite state machine from the particular music state to a legato tempo detection state;
- detecting a note onset while in the legato tempo detection state;
- computing a potential legato tempo based on the detected note onset;
- and
- validating the potential legato tempo based on a multiple of an initial tempo of the one or more tempos.
13 more elements
A computer system, comprising:
- at least one processor;
- a computer memory;
- a Fast Fourier Transform module receiving audio data describing an audio signal, and determining a set of frames of the audio signal using the audio data;
- a smart music detection module communicatively coupled with the Fast Fourier Transform module to receive frequency domain data describing the set of frames of the audio signal from the Fast Fourier Transform module, the smart music detection module performing operations comprising:
- receiving audio data describing an audio signal,
- determining a set of frames of the audio signal using the audio data,
- identifying a plurality of note onsets in the set of frames based on spectral energy of the audio signal in the set of frames,
- computing one or more tempos based on the identified plurality of note onsets,
- validating the one or more tempos based on a tempo validation condition, and
- determining one or more music states of the audio signal based on the validated one or more tempos;
- and
- a smart noise cancelation module modifying audio enhancement of the audio signal using the one or more music states of the audio signal determined by the smart music detection module, the smart noise cancelation module communicatively coupled with the smart music detection module to receive the one or more determined music states of the audio signal from the smart music detection module.
9 more elements
A computer-implemented method, comprising:
- receiving, by a computing device, audio data describing an audio signal;
- determining, by the computing device, a set of frames of the audio signal using the audio data;
- identifying, by the computing device, a plurality of note onsets in the set of frames based on spectral energy of the audio signal in the set of frames;
- computing, by the computing device, one or more tempos based on the identified plurality of note onsets;
- validating, by the computing device, the one or more tempos based on a tempo validation condition;
- determining, by the computing device, one or more music states of the audio signal based on the validated one or more tempos;
- determining, by the computing device, one or more states of a finite state machine based on the validated one or more tempos;
- and
- declaring, by the computing device, that the audio signal includes music based on a transition of the one or more states to a final state of the finite state machine, the one or more music states including the final state.
6 more elements
A computer-implemented method, comprising:
- receiving, by a computing device, audio data describing an audio signal;
- determining, by the computing device, a set of frames of the audio signal using the audio data;
- identifying, by the computing device, a plurality of note onsets in the set of frames based on spectral energy of the audio signal in the set of frames;
- validating, by the computing device, at least one of the identified plurality of note onsets, wherein validating the at least one of the identified plurality of note onsets comprises:
- determining a quantity of the set of frames between a first frame of a particular state of one or more music states and a frame in which a particular note onset is detected;
- determining that the quantity of the set of frames between the first frame and the frame of the particular note onset satisfies a defined threshold;
- and
- responsive to determining that the quantity of frames satisfies the defined threshold, setting the one or more music states to an initial state, the initial state indicating that music has not been detected in the audio signal;
- computing, by the computing device, one or more tempos based on the identified plurality of note onsets;
- validating, by the computing device, the one or more tempos based on a tempo validation condition;
- and
- determining, by the computing device, the one or more music states of the audio signal based on the validated one or more tempos.
9 more elements
A computer-implemented method, comprising:
- receiving, by a computing device, audio data describing an audio signal;
- determining, by the computing device, a set of frames of the audio signal using the audio data;
- identifying, by the computing device, a plurality of note onsets in the set of frames based on spectral energy of the audio signal in the set of frames;
- computing, by the computing device, one or more tempos based on the identified plurality of note onsets;
- validating, by the computing device, the one or more tempos based on a tempo validation condition comprising:
- computing a quantity of the identified plurality of note onsets during a period of a valid tempo of the one or more tempos;
- determining that the quantity of the identified plurality of note onsets satisfies a defined threshold;
- and
- responsive to determining that the quantity of the identified plurality of note onsets satisfies the defined threshold, setting one or more music states to an initial state, the initial state indicating that music has not been detected in the audio signal;
- and
- determining, by the computing device, the one or more music states of the audio signal based on the validated one or more tempos.
8 more elements
A computer-implemented method, comprising:
- receiving, by a computing device, audio data describing an audio signal;
- determining, by the computing device, a set of frames of the audio signal using the audio data;
- identifying, by the computing device, a plurality of note onsets in the set of frames based on spectral energy of the audio signal in the set of frames;
- computing, by the computing device, one or more tempos based on the identified plurality of note onsets;
- validating, by the computing device, the one or more tempos based on a tempo validation condition, wherein the tempo validation condition comprises a tempo swing condition and validating the one or more tempos based on the tempo swing condition comprises:
- determining a tempo swing condition threshold based on an initial tempo of the one or more tempos and a defined multiplier;
- determining a maximum tempo of the one or more tempos;
- determining a minimum tempo of the one or more tempos;
- determining a difference between the maximum tempo and the minimum tempo;
- determining that the difference satisfies the tempo swing condition threshold;
- and
- responsive to determining that the difference satisfies the tempo swing condition threshold, setting one or more music states to an initial state, the initial state indicating that music has not been detected in the audio signal;
- and
- determining, by the computing device, the one or more music states of the audio signal based on the validated one or more tempos.
11 more elements
A computer-implemented method, comprising:
- receiving, by a computing device, audio data describing an audio signal;
- determining, by the computing device, a set of frames of the audio signal using the audio data;
- identifying, by the computing device, a plurality of note onsets in the set of frames based on spectral energy of the audio signal in the set of frames;
- computing, by the computing device, one or more tempos based on the identified plurality of note onsets;
- validating, by the computing device, the one or more tempos based on a tempo validation condition comprising determining that a plurality of the one or more tempos satisfy a tempo swing condition, the tempo swing condition indicating whether a range of the plurality of the one or more tempos satisfy a determined tempo swing condition threshold;
- and
- determining, by the computing device, one or more music states of the audio signal based on the validated one or more tempos comprising transitioning a finite state machine from a first of the one or more music states to a second of the one or more music states based on the determination that the plurality of the one or more tempos satisfy the tempo swing condition.
4 more elements
A computer-implemented method, comprising:
- receiving, by a computing device, audio data describing an audio signal;
- determining, by the computing device, a set of frames of the audio signal using the audio data;
- identifying, by the computing device, a plurality of note onsets in the set of frames based on spectral energy of the audio signal in the set of frames;
- computing, by the computing device, one or more tempos based on the identified plurality of note onsets;
- validating, by the computing device, the one or more tempos based on a tempo validation condition comprising:
- determining that a second tempo differs from a first tempo by a threshold amount;
- and
- responsive to determining that the second tempo differs from the first tempo by greater than the threshold amount:
- searching for an additional note onset,
- determining a tempo for the additional note onset, and
- determining whether the tempo for the additional note onset satisfies a tempo swing condition, the tempo swing condition indicating whether a range of the first tempo and the second tempo satisfy a determined tempo swing condition threshold;
- and
- determining, by the computing device, one or more music states of the audio signal based on the validated one or more tempos.
10 more elements
A computer-implemented method, comprising:
- receiving, by a computing device, audio data describing an audio signal;
- determining, by the computing device, a set of frames of the audio signal using the audio data;
- identifying, by the computing device, a plurality of note onsets in the set of frames based on spectral energy of the audio signal in the set of frames;
- computing, by the computing device, one or more tempos based on the identified plurality of note onsets;
- validating, by the computing device, the one or more tempos based on a tempo validation condition;
- and
- determining, by the computing device, one or more music states of the audio signal based on the validated one or more tempos comprising:
- computing an average quantity of the set of frames between note onsets in the identified plurality of note onsets;
- determining whether the average quantity of the set of frames between the note onsets satisfies a threshold length;
- and
- responsive to determining that the average quantity does not satisfy the threshold length, setting the one or more music states to an initial state, the initial state indicating that music has not been detected in the audio signal.
8 more elements
A computer-implemented method, comprising:
- receiving, by a computing device, audio data describing an audio signal;
- determining, by the computing device, a set of frames of the audio signal using the audio data;
- identifying, by the computing device, a plurality of note onsets in the set of frames based on spectral energy of the audio signal in the set of frames;
- computing, by the computing device, one or more tempos based on the identified plurality of note onsets;
- validating, by the computing device, the one or more tempos based on a tempo validation condition;
- and
- determining, by the computing device, one or more music states of the audio signal based on the validated one or more tempos comprising:
- determining whether a total onset score satisfies a first threshold, an onset score of the total onset score representing a signal energy of a note onset of the plurality of note onsets;
- determining whether a total music factor score satisfies a second threshold, a music factor score representing a high-frequency component of the note onset of the plurality of note onsets;
- transitioning a finite state machine to a final state based on the total onset score satisfying the first threshold and the total music factor score satisfying the second threshold, the one or more music states including the final state;
- and
- declaring that the audio signal includes music based on the final state.
9 more elements
Dependencies are inferred from the claim text.