Apparatuses and methods for recognizing object and facial expression robust against change in facial expression, and apparatuses and methods for training

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform a method comprising:

• acquiring an object image comprising an object;
• extracting a speech map from the object image based on a speech model;
• extracting a facial expression feature from the object image based on a facial expression model;
• and
• recognizing a facial expression of the object based on the speech map and the facial expression feature.

A method of recognizing an object based on a speech, the method comprising:

• acquiring an object image comprising an object;
• extracting a speech map from the object image based on a speech model;
• extracting an object feature from the object image based on an object model;
• and
• recognizing the object based on the speech map and the object feature.

A method of training a facial expression based on a speech, the method comprising:

• acquiring a reference image and a reference facial expression;
• and
• training, by a processor, a parameter of a facial expression model and a parameter of a speech model to output the reference facial expression, in response to an input of the reference image, based on the facial expression model and the speech model,
• wherein the facial expression model is configured to extract a facial expression feature from an object image, and the speech model is configured to extract a speech map from the object image.

An apparatus to recognize a facial expression based on a speech, the apparatus comprising:

• a memory configured to store a facial expression model, speech model, and instructions;
• and
• a processor configured to execute the instructions to:
• acquire an image of an object,
• extract a speech map from the image based on the speech model,
• extract a facial expression feature from the image based on the facial expression model, and
• recognize a facial expression of the object based on the speech map and the facial expression feature.