Neural network for recognition of signals in multiple sensory domains

An apparatus for determining an identity of a speaker, comprising:

• a memory;
• and
• a processor communicably coupled to the memory, the processor configured to execute instructions to:
• extract a first audio feature from a first audio content of a first video clip that includes a prescribed utterance of a first speaker who is identified by a speaker identifier;
• extract a first video feature from a first video content of the first video clip that includes an image of the first speaker;
• obtain an authentication signature based on the first audio feature and the first video feature;
• extract a second audio feature from a second audio content of a second video clip that includes an utterance of a second speaker who is not pre-identified;
• extract a second video feature from a second video content of the second video clip that includes an image of the second speaker;
• obtain a signature of the second speaker based on the second audio feature and the second video feature;
• and
• determine whether the second speaker in the second video clip is the same as the first speaker in the first video clip based on a comparison between the signature of the second speaker and the authentication signature.

A method of estimating a direction of a sound, comprising:

• extracting a first audio feature from a first audio content of a first video clip;
• extracting a first video feature from a first video content of the first video clip;
• determining a label indicating at least a direction of a first sound from a sound source in the first video clip based on the first audio feature and the first video feature;
• extracting a second audio feature from a second audio content of a second video clip that includes a second sound from the sound source, wherein the direction of the second sound is not pre-identified;
• extracting a second video feature from a second video content of the second video clip;
• and
• obtaining a probable direction of the second sound based on a comparison of the second audio feature to the first audio feature and a comparison of the second video feature to the first video feature.

An apparatus for estimating a direction of a sound, comprising:

• a memory;
• and
• a processor communicably coupled to the memory, the processor configured to execute instructions to:
• extract a first audio feature from a first audio content of a first video clip;
• extract a first video feature from a first video content of the first video clip;
• determine a label indicating at least a direction of a first sound from a sound source in the first video clip based on the first audio feature and the first video feature;
• extract a second audio feature from a second audio content of a second video clip that includes a second sound from the sound source, wherein the direction of the second sound is not pre-identified;
• extract a second video feature from a second video content of the second video clip;
• and
• obtain a probable direction of the second sound based on a comparison of the second audio feature to the first audio feature and a comparison of the second video feature to the first video feature.