Personalized gesture recognition for user interaction with assistant systems
AssigneeMeta Platforms Technologies, LLC
InventorsPaul Anthony Crook, Xiaohu Liu, Francislav P. Penov, Rajen Subba
Life of this patent
Estimating the term…
Abstract
In one embodiment, a method includes receiving a user request from a first user from a client system associated with a first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user, determining an intent corresponding to the user request based on the gesture-input by a personalized gesture-classification model associated with the first user, executing one or more tasks based on the determined intent and the speech-input, and sending instructions for presenting execution results of the one or more tasks to the client system responsive the user request.
Claims20
A method comprising, by a computer system:
- receiving a first gesture-input of a user and a speech-input of the user;
- determining a user intent based at least in part on the speech-input;
- determining, based at least in part on the first gesture-input, a gesture identifier associated with the first gesture-input;
- associating the gesture identifier with the user intent;
- receiving a second gesture-input of the user;
- determining if the second gesture-input corresponds to the gesture identifier;
- and
- in response to determining that the second gesture-input corresponds to the gesture identifier, performing a task based at least in part on the user intent associated with the gesture identifier.
A non-transitory, computer-readable medium storing instructions that, when executed by one or more processors of a computing system, cause the computing system to:
- receive a first gesture-input of a user and a speech-input of the user;
- determine a user intent based at least in part on the speech-input;
- determine, based at least in part on the first gesture-input, a gesture identifier associated with the first gesture-input;
- associate the gesture identifier with the user intent;
- receive a second gesture-input of the user;
- determine if the second gesture-input corresponds to the gesture identifier;
- and
- in response to determining that the second gesture-input corresponds to the gesture identifier, perform a task based at least in part on the user intent associated with the gesture identifier.
A smart glasses system, comprising:
- a microphone;
- a camera;
- one or more processors communicatively coupled to the microphone and the camera;
- and
- one or more non-transitory, computer-readable media storing instructions that, when executed by at least one of the one or more processors, cause the smart glasses system to:
- receive from the camera a first gesture-input of a user and from the microphone a speech-input of the user;
- determine a user intent based at least in part on the speech-input;
- determine, based at least in part on the first gesture-input, a gesture identifier associated with the first gesture-input;
- associate the gesture identifier with the user intent;
- receive from the camera a second gesture-input of the user;
- determine if the second gesture-input corresponds to the gesture identifier;
- and
- in response to determining that the second gesture-input corresponds to the gesture identifier, perform a task based at least in part on the user intent associated with the gesture identifier.
Dependencies are inferred from the claim text.