Systems and methods for classifying beamformed signals for binaural audio playback

A system comprising:

• a reception module, stored in memory, that receives a signal for each channel of an audio transducer array on a wearable device, wherein the received signal is captured by the wearable device when worn by a user;
• a calculation module, stored in memory, that calculates a beamformed signal for each beam direction of a set of beamforming filters for the wearable device;
• a classification module, stored in memory, that classifies a first beamformed signal from the calculated beamformed signals into a first class of sound and a second beamformed signal from the calculated beamformed signals into a second class of sound;
• an adjustment module, stored in memory, that adjusts, based on the classifying, a gain of the first beamformed signal relative to the second beamformed signal to enable sound mixing;
• a conversion module, stored in memory, that:
• converts the calculated and adjusted beamformed signals into spatialized binaural audio based on a position of the user wearing the wearable device and a position of a virtual user independent of the position of the user wearing the wearable device, wherein the spatialized binaural audio is updated in real time based on a new position of the user wearing the wearable device and a new position of the virtual user;
• and
• rotates a corresponding video and recalculating a timing of the corresponding video based on the new position of the user wearing the wearable device and the new position of the virtual user;
• a transmitting module, stored in memory, that transmits the spatialized binaural audio and the corresponding video to a playback device of the virtual user, wherein the spatialized binaural audio and the corresponding video replicates a real-world user experience shared from the wearable device to the playback device as the virtual user's point-of-view;
• and
• at least one processor that executes the reception module, the calculation module, the classification module, the adjustment module, the conversion module, and the transmitting module.

A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:

• receive a signal for each channel of an audio transducer array on a wearable device, wherein the received signal is captured by the wearable device when worn by a user;
• calculate a beamformed signal for each beam direction of a set of beamforming filters for the wearable device;
• classify a first beamformed signal from the calculated beamformed signals into a first class of sound and a second beamformed signal from the calculated beamformed signals into a second class of sound;
• adjust, based on the classifying, a gain of the first beamformed signal relative to the second beamformed signal to enable sound mixing;
• convert the calculated and adjusted beamformed signals into spatialized binaural audio based on a position of the user wearing the wearable device and a position of a virtual user independent of the position of the user wearing the wearable device, wherein the spatialized binaural audio is updated in real time based on a new position of the user wearing the wearable device and a new position of the virtual user;
• rotate a corresponding video and recalculating a timing of the corresponding video based on the new position of the user wearing the wearable device and the new position of the virtual user;
• and
• transmit the spatialized binaural audio and the corresponding video to a playback device of the virtual user, wherein the spatialized binaural audio and the corresponding video replicates a real-world user experience shared from the wearable device to the playback device as the virtual user's point-of-view.