Artificial intelligence device for a digital avatar with 3D interaction capabilities and control method thereof

An artificial intelligence (AI) device for implementing a digital avatar, the AI device comprising:

• a memory configured to store information for controlling a digital avatar;
• and
• a controller configured to:
• receive an audio signal corresponding to a user query,
• convert, by a speech-to-text neural network model, the audio signal into a text query,
• input the text query into a large language gesture instruction model to generate high level movement instructions for controlling a digital avatar to move or manipulate a 3D object,
• input the text query and the high level movement instructions into an information retrieval model to generate a text response including at least one sentence and digital avatar control information based on the high level movement instructions,
• input the text response into a text-to-speech neural network model to generate an audio response corresponding to the at least one sentence,
• input the audio response into an audio-to-facial animation model and an audio-to-conversational gesture model to generate updated digital avatar control information including gesture information, and
• output the audio response and the updated digital avatar control information for controlling the digital avatar.