# US 12,182,883 B2 · In-call experience enhancement for assistant systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,883 B2
- Document ID: US12182883B2
- Status: Granted
- Kind: B2
- Application number: 17/500,679
- Filed: 2021-10-13
- Published: 2022-02-03
- Granted: 2024-12-31
- Earliest claimed: 2019-10-18
- Assignees: Meta Platforms Technologies, LLC
- Applicants: Meta Platforms Technologies, LLC
- Inventors: Xiaohu Liu; Baiyang Liu; Rajen Subba
- Web page: https://patentgrep.com/patent/US12182883B2

## CPC

- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 9/453: {Help systems}
- G06F 9/485: {Task life-cycle, e.g. stopping, restarting, resuming execution (G06F9/4881 takes precedence)}
- G06F 9/4862: {the task being a mobile agent, i.e. specifically designed to migrate}
- G06F 9/4881: {Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues}
- G06F 9/54: Interprogram communication
- G06F 9/547: {Remote procedure calls [RPC]; Web services}
- G06F 16/3329: Natural language query formulation
- G06F 16/90332: {Natural language query formulation or dialogue systems}
- G06F 16/9536: Search customisation based on social or collaborative filtering
- G06F 18/2321: using statistics or function optimisation, e.g. modelling of probability density functions
- G06F 40/126: Character encoding
- G06F 40/205: Parsing
- G06F 40/216: using statistical methods
- G06F 40/242: Dictionaries
- G06F 40/253: Grammatical analysis; Style critique
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 40/295: Named entity recognition
- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G06F 40/56: Natural language generation
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 3/098: Distributed learning, e.g. federated learning
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06N 20/00: Machine learning
- G06Q 10/00: Administration; Management
- G06Q 10/109: Time management, e.g. calendars, reminders, meetings or time accounting
- G06Q 10/1093: Calendar-based scheduling for persons or groups
- G06Q 10/40: Business processes related to social networking or social networking services
- G06Q 10/42: Determination of affinities or common interests between users
- G06Q 10/48: using social graphs
- G06Q 30/0603: {Catalogue creation or management}
- G06Q 30/0631: {Recommending goods or services}
- G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)}
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06V 20/20: in augmented reality scenes
- G06V 20/30: in albums, collections or shared content, e.g. social network photos or video
- G06V 40/16: Human faces, e.g. facial parts, sketches or expressions
- G06V 40/25: {Recognition of walking or running movements, e.g. gait recognition}
- G10L 15/063: {Training}
- G10L 15/08: Speech classification or search
- G10L 15/16: using artificial neural networks
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 15/32: Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
- H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
- H04L 51/18: Commands or executable codes
- H04L 51/212: using filtering or selective blocking
- H04L 51/214: using selective forwarding
- H04L 51/222: using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area
- H04L 51/224: providing notification on incoming messages, e.g. pushed notifications of received messages
- H04L 51/52: for supporting social networking services
- H04L 67/306: User profiles
- H04L 67/75: Indicating network or usage conditions on the user display
- H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 2209/541: Client-server
- G06N 3/048: Activation functions
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 40/174: {Facial expression recognition}
- G06V 2201/10: Recognition assisted with metadata
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 15/07: to the speaker
- G10L 2015/0631: {Creating reference templates; Clustering}
- G10L 2015/088: {Word spotting}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2015/227: {of the speaker; Human-factor methodology}
- G10L 2015/228: {of application context}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

In one embodiment, a method includes establishing a video call between a plurality of client systems, wherein access to an assistant system is persistently maintained during the video call, receiving, from a first client system of the plurality of client systems, a request by a first user to be performed by the assistant system during the video call, wherein the request references one or more activities associated with one or more users associated with the plurality of client systems, analyzing, by a context engine of the assistant system, images of a scene of the video call to identify the one or more activities within the scene, instructing the assistant system to execute the request based on the identified one or more activities, and sending, to one or more of the plurality of client systems, a response to the request while maintaining the video call between the plurality of client systems.

## Claims

### Claim 1

A method comprising, by one or more computing systems:
establishing a video call between a plurality of client systems, wherein access to an assistant system is persistently maintained during the video call;
receiving, from a first client system of the plurality of client systems, a request by a first user to be performed by the assistant system during the video call, wherein the request references one or more activities to be performed after the request and is associated with one or more users associated with the plurality of client systems, wherein the one or more activities referenced in the request are physical actions performed by the one or more users after the request is made;
analyzing, by a context engine of the assistant system after receiving the request from the first user, images of a scene of the video call to identify within the scene the one or more activities referenced in the request;
instructing the assistant system to execute the request responsive to the identification of one or more of the activities referenced in the request; and
sending, to one or more of the plurality of client systems, a response to the request while maintaining the video call between the plurality of client systems.

### Claim 2 (depends on 1)

The method of claim 1, wherein the request by the first user further references an instruction to perform a virtual activity with respect to one or more of the activities.

### Claim 3 (depends on 1)

The method of claim 1, wherein the request by the first user further references an instruction to identify within the scene one or more objects or one or more users with respect to one or more of the activities referenced in the request.

### Claim 4 (depends on 1)

The method of claim 1, wherein analyzing images of the scene of the video call to identify within the scene the one or more activities referenced in the request comprises:
identifying user activity of one or more users of the plurality of client systems, wherein the identified user activity is identified as being equivalent to one of the one or more activities referenced in the request based on a taxonomy of activity classes.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
accessing, from the context engine of the assistant system, context data associated with the video call, wherein the context data indicates properties of a scene of the video call to identify one or more activities or objects within the scene.

### Claim 6 (depends on 5)

The method of claim 5, wherein the context data comprises identifications of one or more objects within the scene of the video call, and wherein the request by the first user further references one or more of the identified objects.

### Claim 7 (depends on 5)

The method of claim 5, wherein the context data comprises identifications of one or more users within the scene of the video call, and wherein the request by the first user further references one or more of the identified users.

### Claim 8 (depends on 5)

The method of claim 5, wherein the context data comprises location information of one or more objects or one or more users within the scene of the video call, and wherein the request by the first user further references a location of one or more of the objects or users.

### Claim 9 (depends on 5)

The method of claim 5, wherein the context data comprises information indicating one or more context changes with respect to one or more objects within the scene of the video call, and wherein the request by the first user further references context of one or more of the objects.

### Claim 10 (depends on 5)

The method of claim 5, wherein the context data comprises information indicating context changes with respect to one or more users within the scene of the video call, and wherein the request by the first user further references context of one or more users.

### Claim 11 (depends on 5)

The method of claim 5, wherein the context data comprises information indicating context changes with respect to the location of one or more objects or one or more users within the scene of the video call, and wherein the request by the first user further references a location of one or more of the objects or users.

### Claim 12 (depends on 1)

The method of claim 1, wherein the assistant system comprises a scene understanding engine.

### Claim 13 (depends on 12)

The method of claim 12, further comprising:
accessing, from the scene understanding engine of the assistant system, relationship data associated with the scene of the video call, wherein the relationship data indicates relationships between one or more users or one or more objects within the scene of the video call,
wherein the request by the first user further references a relationship between one or more of the objects or users.

### Claim 14 (depends on 12)

The method of claim 12, further comprising:
determining the request by the first user further references a particular type of relationship data; and
activating the scene understanding engine in response to determining the request references the particular type of relationship data, wherein the scene understanding engine analyzes the scene of the video call to generate relationship data of the particular type of relationship data referenced in the request.

### Claim 15 (depends on 14)

The method of claim 14, wherein the scene understanding engine generates the relationship data in real time in response to being activated.

### Claim 16 (depends on 14)

The method of claim 14, further comprising:
deactivating the scene understanding engine after the relationship data has been generated.

### Claim 17 (depends on 12)

The method of claim 12, wherein the scene understanding engine accesses the relationship data in response to a subsequent request by the first user to be performed by the assistant system during the video call.

### Claim 18 (depends on 12)

The method of claim 12, further comprising:
activating the scene understanding engine in response to determining the request references a particular type of relationship, wherein the scene understanding engine analyzes the scene of the video call to generate relationship data of the particular type of relationship referenced in the request;
identifying one or more changes between the particular type of relationship in the scene of the video call; and
in response to determining the request references changes between the particular type of relationship in the scene of the video call, instructing the assistant system to shift the scene of the video call.

### Claim 19

One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
establish a video call between a plurality of client systems, wherein access to an assistant system is persistently maintained during the video call;
receive, from a first client system of the plurality of client systems, a request by a first user to be performed by the assistant system during the video call, wherein the request references one or more activities to be performed after the request and is associated with one or more users associated with the plurality of client systems, wherein the one or more activities are physical actions performed by the one or more users after the request is made;
analyze, by a context engine of the assistant system after receiving the request from the first user, images of a scene of the video call to identify within the scene the one or more activities referenced in the request;
instruct the assistant system to execute the request responsive to the identification of one or more of the activities referenced in the request; and
send, to one or more of the plurality of client systems, a response to the request while maintaining the video call between the plurality of client systems.

### Claim 20

A system comprising one or more processors and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable, when executed by one or more of the processors, to cause the system to:
establish a video call between a plurality of client systems, wherein access to an assistant system is persistently maintained during the video call;
receive, from a first client system of the plurality of client systems, a request by a first user to be performed by the assistant system during the video call, wherein the request references one or more activities to be performed after the request and is associated with one or more users associated with the plurality of client systems, wherein the one or more activities referenced in the request are physical actions performed by the one or more users after the request is made;
analyze, by a context engine of the assistant system after receiving the request from the first user, images of a scene of the video call to identify within the scene the one or more activities referenced in the request;
instruct the assistant system to execute the request responsive to the identification of one or more of the activities referenced in the request; and
send, to one or more of the plurality of client systems, a response to the request while maintaining the video call between the plurality of client systems.