Mobile audio beamforming using sensor fusion

A head mounted display (HMD) device supporting audio receive beamforming, the device comprising:

• a microphone array;
• a set of inertial sensors;
• a set of one or more forward-facing cameras;
• a logic machine;
• and
• a storage machine having instructions stored thereon executable by the logic machine to:
• obtain a set of audio signals via the microphone array;
• obtain a set of inertial signals via the set of inertial sensors;
• obtain a camera feed captured via the set of one or more forward-facing cameras;
• identify a location of a targeted object to beamform within the camera feed relative to a coordinate system of the HMD device;
• determine a parameter of a beamforming function that defines a beamforming region containing the targeted object based on the set of inertial signals, the location of the targeted object, and an indicia of a changing position of the HMD device, the parameter including at least a beam vector originating at the HMD device and intersecting the targeted object, wherein the beamforming function is one of a plurality of beamforming functions that define respective beamforming regions for a plurality of targeted objects;
• apply the beamforming function to the set of audio signals using the parameter to obtain a set of processed audio signals that increases a signal-to-noise ratio of an audio source within the beamforming region relative to the set of audio signals;
• output the set of processed audio signals;
• and
• upon two of the plurality of targeted objects approaching each other, adjust a beam vector and/or a beam width of at least one of the respective beamforming regions away from at least another of the respective beamforming regions to maintain a threshold distance between the respective beamforming regions and/or reduce overlap between the respective beamforming regions.

A method performed by a computing system for audio receive beamforming, comprising:

• obtaining a set of audio signals via a microphone array of a mobile device;
• obtaining a set of inertial signals via a set of inertial sensors of the mobile device, the microphone array and the set of inertial sensors having a shared coordinate system;
• identifying a location of a targeted object to beamform within a camera feed captured via a set of one or more cameras imagining an environment of the mobile device;
• determining a parameter of a beamforming function that defines a beamforming region containing the targeted object based on the set of inertial signals, the location of the targeted object, and an indicia of a changing position of the mobile device, the parameter including at least a beam vector originating at the microphone array and intersecting the targeted object;
• applying the beamforming function to the set of audio signals using the parameter to obtain a set of processed audio signals that increases a signal-to-noise ratio of an audio source within the beamforming region relative to the set of audio signals;
• outputting the set of processed audio signals to the mobile device;
• determine a change in positioning of the mobile device relative to the targeted object by estimating a future location of the targeted object at a future time relative to the mobile device based on motion of the targeted object observed across individual frames of the camera feed;
• and
• vary one or more of a beam vector and/or a beam width of the beamforming function responsive to the future location prior to identifying an updated location of the targeted object within the camera feed at the future time.