Attention mechanism for coping with acoustic-lips timing mismatch in audiovisual processing

A system for determining a context-dependent time shift of audio and video features in an audiovisual stream or file, the system comprising a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor to perform:

• receiving audio information and video information of the audiovisual stream or file;
• processing the audio information and video information separately to generate a new representation of the audio information, including information relating to features of the audio information, and a new representation of the video information, including information relating to features of the video information;
• and
• mapping features of the audio information and features of the video information using an attention mechanism modeling the context-dependent time shift to identify pairs of audio and video features, wherein the pairs of audio and video features are identified as being synchronized (true) features that contain a recording of a speaker in which the audio information of the recording and the video information of lips of the speaker are synchronized or unsynchronized (false) features otherwise, and wherein the attention mechanism uses a weighted sum of a plurality of consecutive audio frames and a weighted sum of a plurality of consecutive video frames and weights of the attention mechanism are based on a content and context of the audio information and on a content and context of the video information.

A computer program product for determining a context-dependent time shift of audio and video features in an audiovisual stream or file, the computer program product comprising a non-transitory computer readable storage having program instructions embodied therewith, the program instructions executable by a computer, to cause the computer to perform a method comprising:

• receiving audio information and video information of the audiovisual stream or file;
• processing the audio information and video information separately to generate a new representation of the audio information, including information relating to features of the audio information, and a new representation of the video information, including information relating to features of the video information;
• and
• mapping features of the audio information and features of the video information using an attention mechanism modeling a context-dependent time shift to identify pairs of audio and video features, wherein the pairs of audio and video features are identified as being synchronized (true) features that contain a recording of a speaker in which the audio information of the recording and the video information of lips of the speaker are synchronized or unsynchronized (false) features otherwise, and wherein the attention mechanism uses a weighted sum of a plurality of consecutive audio frames and a weighted sum of a plurality of consecutive video frames and weights of the attention mechanism are based on a content and context of the audio information and on a content and context of the video information.