Speaker beam-steering based on microphone array and depth camera assembly input

A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:

• identify an object within a local area of a device from data captured by one or more sensors included in the device, the captured data including audio data and video data;
• determine a depth from the device to the identified object from video data captured by a sensor included in the device;
• determine a distance relative to an axis perpendicular to a center of the device of a location in the local area from which one or more microphones included in the device captured the audio data;
• and direct audio output by each of a plurality of speakers of the device to the identified object at the determined depth and at the distance relative to an axis perpendicular to a center of the device of a location in the local area from which one or more microphones included in the device captured the audio data by
• determining a shading function based on a product of a probability determined by the processor and an angle from an axis normal to a center of a speaker located in a center of the plurality of speakers to the determined depth and a determined location of arrival or the audio data captured by the one or more microphones;
• and
• applying the shading function to the audio output by the plurality of speakers.

A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:

• determine, from audio data captured by a plurality of microphones included in a device, a distance relative to an axis perpendicular to a center of the device of a location in a local area from which the plurality of microphones included in the device captured the audio data;
• obtain video data of the local area from a depth camera assembly and depth information identifying depths from the depth camera assembly to locations within the local area from the depth camera assembly identify an object within the location in the local area from which the plurality of microphones captured audio data from data captured by the depth camera assembly;
• determine a depth from the depth camera assembly to the identified object from data captured by the depth camera assembly;
• and
• direct audio output by each of a plurality of speakers of the device to the identified object at the distance relative to the axis perpendicular to the center of the device of the location in the local area from which the plurality of microphones captured audio data and to the depth from the depth camera assembly to the identified object by
• determining a shading function based on a product of a probability determined by the processor and an angle from an axis normal to a center of a speaker located in a center of the plurality of speakers to the location in the local area from which the plurality of microphones captured audio data;
• and
• applying the shading function to the audio output by the plurality of speakers.

A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:

• determine, from audio data captured by a plurality of microphones included in a device, a distance relative to an axis perpendicular to a center of the device of a location in a local area from which the plurality of microphones included in the device captured the audio data;
• obtain video data of the local area from a depth camera assembly and depth information identifying depths from the depth camera assembly to locations within the local area from the depth camera assembly
• identify an object within the location in the local area from which the plurality of microphones captured audio data from data captured by the depth camera assembly;
• determine a depth from the depth camera assembly to the identified object from data captured by the depth camera assembly;
• and
• direct audio output by each of a plurality of speakers of the device to the identified object at the distance relative to the axis perpendicular to the center of the device of the location in the local area from which the plurality of microphones captured audio data and to the depth from the depth camera assembly to the identified object by
• for each of the plurality of speakers, determine an angle from an axis perpendicular to a center of a speaker to the location in the local area from which the plurality of microphones captured audio data;
• determine a weighting function for each of the plurality of speakers, the weighting function for a speaker based on the angle determined for the speaker and having a maximum at the determined distance and at the determined depth;
• and
• for each of the plurality of speakers, apply the weighting function corresponding the speaker to audio output by the speaker.