# US 2020/0401422 A1 · Personalized Gesture Recognition for User Interaction with Assistant Systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401422 A1
- Document ID: US20200401422A1
- Status: Published application
- Kind: A1
- Application number: 17/010,619
- Filed: 2020-09-02
- Published: 2020-12-24
- Earliest claimed: 2018-04-20
- Assignees: Facebook, Inc.
- Applicants: Facebook, Inc.
- Inventors: Xiaohu Liu; Paul Anthony Crook; Francislav P. Penov; Rajen Subba
- Web page: https://patentgrep.com/patent/US20200401422A1

## CPC

- G06Q 10/00: Administration; Management
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 7/14: Merging, i.e. combining at least two sets of record carriers each arranged in the same ordered sequence to produce a single set having the same ordered sequence
- G06F 8/31: {Programming languages or programming paradigms}
- G06F 9/44505: {Configuring for program initiating, e.g. using registry, configuration files}
- G06F 9/4451: {User profiles; Roaming}
- G06F 9/453: {Help systems}
- G06F 16/176: Support for shared access to files; File sharing support
- G06F 16/2255: {Hash tables}
- G06F 16/2365: {Ensuring data consistency and integrity}
- G06F 16/243: {Natural language query formulation}
- G06F 16/24552: {Database cache management}
- G06F 16/24575: {using context}
- G06F 16/24578: {using ranking}
- G06F 16/248: Presentation of query results
- G06F 16/3322: {using system suggestions (G06F16/3325 takes precedence)}
- G06F 16/3323: {using document space presentation or visualization, e.g. category, hierarchy or range presentation and selection}
- G06F 16/3329: Natural language query formulation
- G06F 16/3344: {using natural language analysis}
- G06F 16/338: Presentation of query results
- G06F 16/90332: {Natural language query formulation or dialogue systems}
- G06F 16/90335: {Query processing}
- G06F 16/9038: Presentation of query results
- G06F 16/904: Browsing; Visualisation therefor (for navigating the web G06F16/954; browsing optimisation for the web G06F16/957)
- G06F 16/951: Indexing; Web crawling techniques
- G06F 16/9532: Query formulation
- G06F 16/9535: Search customisation based on user profiles and personalisation
- G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines
- G06F 21/6245: {Protecting personal data, e.g. for financial or medical purposes}
- G06F 40/205: Parsing
- G06F 40/274: Converting codes to words; Guess-ahead of partial word inputs
- G06F 40/295: Named entity recognition
- G06F 40/30: Semantic analysis
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06N 5/027: {Frames}
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06Q 10/40: Business processes related to social networking or social networking services
- G06Q 10/42: Determination of affinities or common interests between users
- G06Q 10/48: using social graphs
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/063: {Training}
- G10L 15/07: to the speaker
- G10L 15/16: using artificial neural networks
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 15/183: using context dependencies, e.g. language models
- G10L 15/187: Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 17/06: Decision making techniques; Pattern matching strategies
- G10L 17/22: Interactive procedures; Man-machine interfaces
- H04L 5/02: Channels characterised by the type of signal
- H04L 12/2816: {Controlling appliance services of a home automation network by calling their functionalities (arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00)}
- H04L 41/20: {Network management software packages}
- H04L 41/22: comprising specially adapted graphical user interfaces [GUI]
- H04L 43/0882: Utilisation of link capacity
- H04L 43/0894: Packet rate
- H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
- H04L 51/18: Commands or executable codes
- H04L 51/216: Handling conversation history, e.g. grouping of messages in sessions or threads
- H04L 51/52: for supporting social networking services
- H04L 63/102: {Entity profiles}
- H04L 67/306: User profiles
- H04L 67/535: {Tracking the activity of the user (network monitoring arrangements H04L43/00; recording of computer activity G06F11/34)}
- H04L 67/5651: Reducing the amount or size of exchanged application data
- H04L 67/75: Indicating network or usage conditions on the user display
- H04W 12/08: Access security
- G06F 2216/13: Prefetching
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 13/04: Details of speech synthesis systems, e.g. synthesiser structure or memory management
- G10L 17/00: Speaker identification or verification techniques
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2015/225: {Feedback of the input speech}
- H04L 51/046: Interoperability with other network applications or services
- H04L 51/222: using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area
- H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46)
- H04L 67/53: using third party service providers
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

In one embodiment, a method includes receiving a user request from a first user from a client system associated with a first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user, determining an intent corresponding to the user request based on the gesture-input by a personalized gesture-classification model associated with the first user, executing one or more tasks based on the determined intent and the speech-input, and sending instructions for presenting execution results of the one or more tasks to the client system responsive the user request.

## Claims

### Claim 1

A method comprising, by one or more computing systems:
receiving, from a client system associated with a first user, a user request from the first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user;
determining, by a personalized gesture-classification model associated with the first user, an intent corresponding to the user request based on the gesture-input;
executing one or more tasks based on the determined intent and the speech-input; and
sending, to the client system, instructions for presenting execution results of the one or more tasks responsive the user request.

### Claim 2 (depends on 1)

The method of claim 1, further comprising training the personalized gesture-classification model for the first user, wherein the training comprises:
accessing, from a data store, a plurality of input tuples associated with the first user, wherein each input tuple comprises a gesture-input and a corresponding speech-input;
determining, by a natural-language understanding (NLU) module, a plurality of intents corresponding to the plurality of speech-inputs, respectively;
generating, for the plurality of gesture-inputs, a plurality of feature representations based on one or more machine-learning models;
determining a plurality of gesture identifiers for the plurality of gesture-inputs, respectively, based on their respective feature representations;
associating the plurality of intents with the plurality of gesture identifiers, respectively; and
generating the personalized gesture-classification model based on the plurality of feature representations of their respective gesture-inputs and the associations between the plurality of intents and their respective gesture identifiers.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
accessing, from the data store, a general gesture-classification model corresponding to a general user population, wherein training the personalized gesture-classification model is further based on the general gesture-classification model.

### Claim 4 (depends on 3)

The method of claim 3, wherein the general gesture-classification model is trained based on a plurality of gesture-inputs from the general user population.

### Claim 5 (depends on 2)

The method of claim 2, wherein the one or more machine-learning models are based on one or more of a neural network model or a long-short term memory (LSTM) model.

### Claim 6 (depends on 1)

The method of claim 1, wherein training the personalized gesture-classification model is further based on user feedback data from the first user.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
generating, by one or more automatic speech recognition (ASR) modules, one or more text-inputs for the speech-input.

### Claim 8 (depends on 7)

The method of claim 7, wherein determining the intent corresponding to the user request is further based on the one or more text-inputs of the speech-input.

### Claim 9 (depends on 1)

The method of claim 1, wherein the personalized gesture-classification model is based on convolutional neural networks.

### Claim 10 (depends on 1)

The method of claim 1, further comprising generating a feature representation for the gesture-input by the personalized gesture-classification model, wherein generating the feature representation comprises:
dividing the gesture-input into one or more components; and
modeling the one or more components into the feature representation for the gesture-input.

### Claim 11 (depends on 1)

The method of claim 1, further comprising generating a feature representation for the gesture-input by the personalized gesture-classification model, wherein generating the feature representation comprises:
determining temporal information associated with the gesture-input; and
modeling the temporal information into the feature representation for the gesture-input.

### Claim 12

One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
receive, from a client system associated with a first user, a user request from the first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user;
determine, by a personalized gesture-classification model associated with the first user, an intent corresponding to the user request based on the gesture-input;
execute one or more tasks based on the determined intent and the speech-input; and
send, to the client system, instructions for presenting execution results of the one or more tasks responsive the user request.

### Claim 13 (depends on 12)

The media of claim 12, wherein the software is further operable when executed to train the personalized gesture-classification model for the first user, wherein the software is further operable when executed to:
access, from a data store, a plurality of input tuples associated with the first user, wherein each input tuple comprises a gesture-input and a corresponding speech-input;
determine, by a natural-language understanding (NLU) module, a plurality of intents corresponding to the plurality of speech-inputs, respectively;
generate, for the plurality of gesture-inputs, a plurality of feature representations based on one or more machine-learning models;
determine a plurality of gesture identifiers for the plurality of gesture-inputs, respectively, based on their respective feature representations;
associate the plurality of intents with the plurality of gesture identifiers, respectively; and
generate the personalized gesture-classification model based on the plurality of feature representations of their respective gesture-inputs and the associations between the plurality of intents and their respective gesture identifiers.

### Claim 14 (depends on 13)

The media of claim 13, wherein the software is further operable when executed to:
access, from the data store, a general gesture-classification model corresponding to a general user population, wherein training the personalized gesture-classification model is further based on the general gesture-classification model.

### Claim 15 (depends on 14)

The media of claim 14, wherein the general gesture-classification model is trained based on a plurality of gesture-inputs from the general user population.

### Claim 16 (depends on 13)

The media of claim 13, wherein the one or more machine-learning models are based on one or more of a neural network model or a long-short term memory (LSTM) model.

### Claim 17 (depends on 12)

The media of claim 12, wherein training the personalized gesture-classification model is further based on user feedback data from the first user.

### Claim 18 (depends on 12)

The media of claim 12, wherein the software is further operable when executed to:
generate, by one or more automatic speech recognition (ASR) modules, one or more text-inputs for the speech-input.

### Claim 19 (depends on 12)

The media of claim 12, wherein the personalized gesture-classification model is based on convolutional neural networks.

### Claim 20

A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
receive, from a client system associated with a first user, a user request from the first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user;
determine, by a personalized gesture-classification model associated with the first user, an intent corresponding to the user request based on the gesture-input;
execute one or more tasks based on the determined intent and the speech-input; and
send, to the client system, instructions for presenting execution results of the one or more tasks responsive the user request.