US 2025/0232122 A1Published application

Artificial intelligence device for emotion recognition using joint and unimodal representation learning with late fusion and method thereof

AssigneeLG Electronics Inc.

InventorsAmirhossein HAJAVI, Homa FASHANDI, Harmanpreet SINGH

Life of this patent

Estimating the term…

Loading patent…

Abstract

A method for controlling an artificial intelligence (AI) device can include receiving a video and an audio signal, generating a visual embedding and an audio embedding by visual and audio encoders, processing the visual embedding, by a unimodal visual transformer encoder, to generate an independent visual embedding vector, and processing the audio embedding, by a unimodal audio transformer encoder, to generate an independent audio embedding vector. The method can further include processing the visual and audio embeddings, by respective joint transformer encoders, to generate a joint visual embedding vector using cross-attention and a joint audio embedding vector using cross-attention, processing the independent visual and audio embedding vectors and the joint visual and audio embedding vectors to generate a global fused embedding vector, generating an emotion prediction using a classifier module that analyzes the global fused embedding vector, and outputting an emotion prediction based on the global fused embedding vector.

Claims20

Dependent claims

A method for controlling an artificial intelligence (AI) device, the method comprising:

receiving, by a processor in the AI device, a video segment including a plurality of frames and an audio signal corresponding to the video segment;
processing the video segment, by a visual encoder, to generate a visual embedding;
processing the audio signal, by an audio encoder, to generate an audio embedding;
processing the visual embedding, by a unimodal visual transformer encoder, to generate an independent visual embedding vector based on self-attention;
processing the audio embedding, by a unimodal audio transformer encoder, to generate an independent audio embedding vector based on self-attention;
processing the visual embedding, by a joint visual transformer encoder, to generate a joint visual embedding vector using cross-attention based on information from the audio embedding;
processing the audio embedding, by a joint audio transformer encoder, to generate a joint audio embedding vector using cross-attention based on information from the video embedding;
processing the independent visual embedding vector, the independent audio embedding vector, the joint visual embedding vector and the joint audio embedding vector, by a fusion module, to generate a global fused embedding vector;
and
generating an emotion prediction using a classifier module that analyzes the global fused embedding vector, and outputting the emotion prediction.

7 more elements

An artificial intelligence (AI) device, comprising:

a memory configured to store video and audio information;
and
a controller configured to:
receive a video segment including a plurality of frames and an audio signal corresponding to the video segment,
process the video segment, by a visual encoder, to generate a visual embedding,
process the audio signal, by an audio encoder, to generate an audio embedding,
process the visual embedding, by a unimodal visual transformer encoder, to generate an independent visual embedding vector based on self-attention,
process the audio embedding, by a unimodal audio transformer encoder, to generate an independent audio embedding vector based on self-attention,
process the visual embedding, by a joint visual transformer encoder, to generate a joint visual embedding vector using cross-attention based on information from the audio embedding,
process the audio embedding, by a joint audio transformer encoder, to generate a joint audio embedding vector using cross-attention based on information from the video embedding,
process the independent visual embedding vector, the independent audio embedding vector, the joint visual embedding vector and the joint audio embedding vector, by a fusion module, to generate a global fused embedding vector, and
generate an emotion prediction using a classifier module that analyzes the global fused embedding vector, and output the emotion prediction.

9 more elements

Dependencies are inferred from the claim text.