Personalized Gesture Recognition for User Interaction with Assistant Systems

One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

• receive, from a client system associated with a first user, a user request from the first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user;
• determine, by a personalized gesture-classification model associated with the first user, an intent corresponding to the user request based on the gesture-input;
• execute one or more tasks based on the determined intent and the speech-input;
• and
• send, to the client system, instructions for presenting execution results of the one or more tasks responsive the user request.

A system comprising:

• one or more processors;
• and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
• receive, from a client system associated with a first user, a user request from the first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user;
• determine, by a personalized gesture-classification model associated with the first user, an intent corresponding to the user request based on the gesture-input;
• execute one or more tasks based on the determined intent and the speech-input;
• and
• send, to the client system, instructions for presenting execution results of the one or more tasks responsive the user request.