System and method for personalization in intelligent multi-modal personal assistants

A method, by a user device, comprising:

• receiving an input from a user;
• obtaining at least one of audio information, video information, and relation information of the user;
• identifying the user based on the audio information or the video information of the user and a set of facial embeddings and speech embeddings that is correlated with the user, the set of facial embeddings and speech embeddings being generated using a facial embedding model, a speech embedding model, and a sound source localization model;
• and
• performing an action based on the input and at least one of identifying the user and the relation information of the user.

A user device comprising:

• a memory configured to store instructions;
• and
• a processor configured to execute the instructions to:
• receive an input from a user;
• obtain at least one of audio information, video information, and relation information of the user;
• identify the user based on the audio information or the video information of the user and a set of facial embeddings and speech embeddings that is correlated with the user, the set of facial embeddings and speech embeddings being generated using a facial embedding model, a speech embedding model, and a sound source localization model;
• and
• perform an action based on the input and at least one of identifying the user and the relation information of the user.

A non-transitory computer-readable medium storing instructions that, when executed, cause at least one processor of a user device to:

• receive an input from a user;
• obtain at least one of audio information, video information, and relation information of the user;
• identify the user based on the audio information or the video information of the user and a set of facial embeddings and speech embeddings that is correlated with the user, the set of facial embeddings and speech embeddings being generated using a facial embedding model, a speech embedding model, and a sound source localization model;
• and
• perform an action based on the input and at least one of identifying the user and the relation information of the user.