Tracking beats and downbeats of voices in real time

A system of extracting a melody, comprising:

• at least one processor;
• and
• at least one memory communicatively coupled to the at least one processor and comprising computer-readable instructions that upon execution by the at least one processor cause the at least one processor to perform operations comprising:
• receiving audio in real time;
• splitting the audio into a sequence of segments;
• extracting a sequence of audio features corresponding to the sequence of segments of the audio;
• generating in real time a continuous sequence of activations indicative of probabilities of beats or downbeats occurring in the sequence of segments of the audio using a machine learning model with causal mechanisms, wherein the machine learning model is pre-trained to generate beat or downbeat activations based on audio features;
• and
• determining timings of the beats or the downbeats occurring in the sequence of segments of the audio based on the continuous sequence of activations by fusing local rhythmic information with respect to each instant segment among the sequence of segments of the audio with information indicative of beats or downbeats in previous segments among the sequence of segments.

A non-transitory computer-readable storage medium, storing computer-readable instructions that upon execution by a processor cause the processor to implement operations comprising:

• receiving audio in real time;
• splitting the audio into a sequence of segments;
• extracting a sequence of audio features corresponding to the sequence of segments of the audio;
• generating in real time a continuous sequence of activations indicative of probabilities of beats or downbeats occurring in the sequence of segments of the audio using a machine learning model with causal mechanisms, wherein the machine learning model is pre-trained to generate beat or downbeat activations based on audio features;
• and
• determining timings of the beats or the downbeats occurring in the sequence of segments of the audio based on the continuous sequence of activations by fusing local rhythmic information with respect to each instant segment among the sequence of segments of the audio with information indicative of beats or downbeats in previous segments among the sequence of segments.