Method and system for real-time and low latency synthesis of audio using neural networks and differentiable digital signal processors

A non-transitory computer-readable device having instructions thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations comprising:

• generating a frame by sampling audio input in increments, which are based on a first buffer size associated with an input/output buffer of a host device, until a threshold buffer size, corresponding to a frame size used to train a machine learning model, is reached, wherein the first buffer size does not match the threshold buffer size;
• extracting, from the frame, amplitude information, pitch information, and pitch status information;
• determining, by the machine learning model, control information for audio reproduction based on the amplitude information, the pitch information, and the pitch status information, the control information including pitch control information and noise magnitude control information;
• generating filtered noise information by inverting the noise magnitude control information using an overlap and add technique, including:
• receiving the noise magnitude control information according to the frame size from the machine learning model;
• rendering the filtered noise information in a block size not equal to the frame size:
• writing, via the overlap and add technique, the filtered noise information to a circular buffer;
• and
• reading, in the first buffer size, the filtered noise information from the circular buffer;
• generating, based on the pitch control information, additive harmonic information by combining a plurality of scaled wavetables;
• and
• rendering audio output based on the filtered noise information and the additive harmonic information.

A system comprising:

• an audio capture device;
• a speaker,
• a memory storing instructions thereon;
• and
• at least one processor coupled with the memory and configured by the instructions to:
• capture audio input via the audio capture device,
• generate a frame by sampling the audio input in increments, which are based on a first buffer size associated with an input/output buffer of a host device, until a threshold buffer size, corresponding to a frame size used to train a machine learning model, is reached, wherein the first buffer size does not match the threshold buffer size;
• extract, from the frame, amplitude information, pitch information, and pitch status information;
• determine, by the machine learning model, control information for audio reproduction based on the amplitude information, the pitch information, and the pitch status information, the control information including pitch control information and noise magnitude control information;
• filter the noise magnitude control information using an overlap and add technique to generate filtered noise information, including
• receiving the noise magnitude control information according to the frame size from a machine learning model:
• rendering the filtered noise information in a block size not equal to the frame size:
• writing, via the overlap and add technique, the filtered noise information to a circular buffer;
• and
• reading, in the first buffer size, the filtered noise information from the circular buffer:
• generate, based on the pitch control information, additive harmonic information by combining a plurality of scaled wavetables;
• render audio output based on the filtered noise information and the additive harmonic information;
• and
• reproduce the audio output via the speaker.