Personalized gesture recognition for user interaction with assistant systems
AssigneeMeta Platforms Technologies, LLC
InventorsPaul Anthony Crook, Francislav P. Penov, Rajen Subba, Xiaohu Liu
Life of this patent
Estimating the term…
Abstract
In one embodiment, a method includes receiving a user request from a first user from a client system associated with a first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user, determining an intent corresponding to the user request based on the gesture-input by a personalized gesture-classification model associated with the first user, executing one or more tasks based on the determined intent and the speech-input, and sending instructions for presenting execution results of the one or more tasks to the client system responsive the user request.
Claims21
(canceled)
A system comprising:
- one or more processors;
- a microphone, operably coupled to at least one of the one or more processors, configured to generate a speech-input based on an audible signal from a user;
- a camera, operably coupled to at least one of the one or more processors, configured to generate a gesture-input based on an action performed by the user;
- and
- one or more non-transitory memories, operably coupled to at least one of the one or more processors, comprising instructions that, when executed by at least one of the one or more processors, cause the system to:
- communicate a user input comprising at least one of speech information or gesture information to an external assistant system to cause the external assistant system to determine an output user intent from the user input, wherein the speech information is based at least in part upon the speech-input, the gesture information is based at least in part upon the gesture-input, and the output user intent is based at least in part upon the speech information or gesture information;
- in response to communicating the user input, receive, from the external assistant system, information associated with the output user intent;
- and
- execute a task based at least in part upon the received information, wherein the task is based at least in part upon the speech-input and the gesture-input.
A method comprising:
- generating a speech-input based on an audible signal from a user captured by a microphone;
- generating a gesture-input based on an action performed by the user captured by a camera;
- communicating a user input comprising at least one of speech information or gesture information to an external assistant system to cause the external assistant system to determine an output user intent from the user input, wherein the speech information is based at least in part upon the speech-input, the gesture information is based at least in part upon the gesture-input, and the output user intent is based at least in part upon the speech information or gesture information;
- in response to communicating the user input, receiving, from the external assistant system, information associated with the output user intent;
- and
- executing a task based at least in part upon the received information, wherein the task is based at least in part upon the speech-input and the gesture-input.
A non-transitory computer-readable medium operably coupled to one or more processors, comprising instructions that, when executed by at least one of the one or more processors, cause the at least one of the one or more processors to:
- generate a speech-input based on an audible signal from a user captured by a microphone;
- generate a gesture-input based on an action performed by the user captured by a camera;
- communicate a user input comprising at least one of speech information or gesture information to an external assistant system to cause the external assistant system to determine an output user intent from the user input, wherein the speech information is based at least in part upon the speech-input, the gesture information is based at least in part upon the gesture-input, and the output user intent is based at least in part upon the speech information or gesture information;
- in response to communicating the user input, receive, from the external assistant system, information associated with the output user intent;
- and
- execute a task based at least in part upon the received information, wherein the task is based at least in part upon the speech-input and the gesture-input.
Dependencies are inferred from the claim text.