# US 11,443,120 B2 · Multimodal entity and coreference resolution for assistant systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,120 B2 - Document ID: US11443120B2 - Status: Granted - Kind: B2 - Application number: 17/006,377 - Filed: 2020-08-28 - Published: 2021-04-22 - Granted: 2022-09-13 - Earliest claimed: 2019-10-18 - Assignees: Meta Platforms, Inc. - Applicants: Meta Platforms, Inc. - Inventors: Shivani Poddar; Seungwhan Moon; Paul Anthony Crook; Rajen Subba - Web page: https://patentgrep.com/patent/US11443120B2 ## CPC - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 9/453: {Help systems} - G06F 9/485: {Task life-cycle, e.g. stopping, restarting, resuming execution (G06F9/4881 takes precedence)} - G06F 9/4862: {the task being a mobile agent, i.e. specifically designed to migrate} - G06F 9/4881: {Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues} - G06F 9/54: Interprogram communication - G06F 9/547: {Remote procedure calls [RPC]; Web services} - G06F 16/3329: Natural language query formulation - G06F 16/90332: {Natural language query formulation or dialogue systems} - G06F 16/9536: Search customisation based on social or collaborative filtering - G06F 18/2321: using statistics or function optimisation, e.g. modelling of probability density functions - G06F 40/126: Character encoding - G06F 40/205: Parsing - G06F 40/216: using statistical methods - G06F 40/242: Dictionaries - G06F 40/253: Grammatical analysis; Style critique - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 40/295: Named entity recognition - G06F 40/30: Semantic analysis - G06F 40/35: Discourse or dialogue representation - G06F 40/56: Natural language generation - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/045: Combinations of networks - G06N 3/047: Probabilistic or stochastic networks - G06N 3/08: Learning methods - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06N 3/09: Supervised learning - G06N 3/098: Distributed learning, e.g. federated learning - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06N 20/00: Machine learning - G06Q 10/00: Administration; Management - G06Q 10/109: Time management, e.g. calendars, reminders, meetings or time accounting - G06Q 10/1093: Calendar-based scheduling for persons or groups - G06Q 10/40: Business processes related to social networking or social networking services - G06Q 10/42: Determination of affinities or common interests between users - G06Q 10/48: using social graphs - G06Q 30/0603: {Catalogue creation or management} - G06Q 30/0631: {Recommending goods or services} - G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)} - G06Q 30/0643: {graphically representing goods, e.g. 3D product representation} - G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes} - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60) - G06V 20/20: in augmented reality scenes - G06V 20/30: in albums, collections or shared content, e.g. social network photos or video - G06V 40/16: Human faces, e.g. facial parts, sketches or expressions - G06V 40/25: {Recognition of walking or running movements, e.g. gait recognition} - G10L 15/063: {Training} - G10L 15/08: Speech classification or search - G10L 15/16: using artificial neural networks - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - G10L 15/1822: {Parsing for meaning understanding} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications - G10L 15/32: Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems - H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages - H04L 51/18: Commands or executable codes - H04L 51/212: using filtering or selective blocking - H04L 51/214: using selective forwarding - H04L 51/222: using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area - H04L 51/224: providing notification on incoming messages, e.g. pushed notifications of received messages - H04L 51/52: for supporting social networking services - H04L 67/306: User profiles - H04L 67/75: Indicating network or usage conditions on the user display - H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 2209/541: Client-server - G06N 3/048: Activation functions - G06N 3/084: Backpropagation, e.g. using gradient descent - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 40/174: {Facial expression recognition} - G06V 2201/10: Recognition assisted with metadata - G10L 13/00: Speech synthesis; Text to speech systems - G10L 15/07: to the speaker - G10L 2015/0631: {Creating reference templates; Clustering} - G10L 2015/088: {Word spotting} - G10L 2015/223: {Execution procedure of a spoken command} - G10L 2015/227: {of the speaker; Human-factor methodology} - G10L 2015/228: {of application context} - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract In one embodiment, a method includes accessing visual data from a client system associated with a user, wherein the visual data comprises images portraying one or more objects, receiving, from the client system, a user request, wherein the user request comprises a coreference to a target object, resolving the coreference to the target object from among the one or more objects, resolving the target object to a specific entity, and sending, to the client system, instructions for providing a response to the user request, wherein the response comprises attribute information about the specific entity. ## Claims ### Claim 1 A method comprising: accessing visual data from a client system associated with a user, wherein the visual data comprises images portraying one or more objects; receiving, from the client system, a user request, wherein the user request comprises a coreference to a target object; resolving the coreference to the target object from among the one or more objects; resolving the target object to a specific entity; and sending, to the client system, instructions for providing a response to the user request, wherein the response comprises attribute information about the specific entity. ### Claim 2 (depends on 1) The method of claim 1, further comprising: analyzing, by a computer vision module, the visual data to identify the one or more objects portrayed in the images; parsing, by a natural-language understanding (NLU) module, the user request to identify an intent of the user request and references to one or more of the objects; and updating a dialog state to include the identified objects and the identified references. ### Claim 3 (depends on 2) The method of claim 2, further comprising: receiving gesture or gaze information from the user; and updating the dialog state to include the received gesture or gaze information. ### Claim 4 (depends on 2) The method of claim 2, wherein resolving the coreference to the target object from among the one or more objects comprises combining additional information with the dialog state. ### Claim 5 (depends on 4) The method of claim 4, wherein the additional information comprises a level of interestingness of one or more of the objects. ### Claim 6 (depends on 4) The method of claim 4, wherein the additional information comprises information of a user profile of the user. ### Claim 7 (depends on 4) The method of claim 4, wherein the additional information comprises a model of past interactions with one or more of the objects by other users. ### Claim 8 (depends on 1) The method of claim 1, further comprising: analyzing the visual data to identify the one or more objects portrayed in the images; assigning respective object identifiers to one or more of the identified objects; and storing one or more of the object identifiers as entities in a dialog state tracker. ### Claim 9 (depends on 8) The method of claim 8, wherein resolving the coreference to the target object from among the one or more objects comprises: analyzing the dialog state tracker to identify one or more of the entities; and identifying a result set of one or more of the identified entities based on a context of the user. ### Claim 10 (depends on 9) The method of claim 9, wherein resolving the target object to a specific entity comprises: selecting the specific entity from among the identified entities of the result set. ### Claim 11 (depends on 8) The method of claim 8, further comprising: in response to receiving the user request, generating, by a scene understanding engine, relational information between identified objects. ### Claim 12 (depends on 1) The method of claim 1, further comprising: accessing a knowledge graph; and retrieving the attribute information about the specific entity from the knowledge graph. ### Claim 13 (depends on 1) The method of claim 1, wherein one or more of the objects of the visual data are virtual objects in a virtual reality environment displayed to the user via the client system. ### Claim 14 (depends on 1) The method of claim 1, further comprising: receiving, from the client system, a second user request to perform a task with respect to the specific entity; executing the task; and sending, the client system, instructions for providing a second response to the user comprising results of the executed task. ### Claim 15 (depends on 1) The method of claim 1, wherein resolving the coreference to the target object from among the one or more objects comprises: analyzing, by a scene understanding engine, the visual data to identify the one or more objects portrayed in the images; and identifying the target object from among the identified objects based on its position within a field of view of the visual data. ### Claim 16 (depends on 1) The method of claim 1, wherein resolving the coreference to the target object from among the one or more objects comprises: sending, to the client system, a prompt for the user to confirm the target object; and receiving, from the client system, a user response to the prompt identifying the target object. ### Claim 17 (depends on 1) The method of claim 1, further comprising: accessing a set of images from the visual data; and selecting one or more images from among the set of images based on a timing of the user request, wherein the selected images portray the target object. ### Claim 18 (depends on 1) The method of claim 1, further comprising: accessing a memory graph associated with the user to retrieve context data associated with the user; and comparing the attribute information about the specific entity with the context data, wherein the response further comprises a result of the comparison. ### Claim 19 One or more computer-readable non-transitory storage media embodying software that is operable when executed to: access visual data from a client system associated with a user, wherein the visual data comprises images portraying one or more objects; receive, from the client system, a user request, wherein the user request comprises a coreference to a target object; resolve the coreference to the target object from among the one or more objects; resolve the target object to a specific entity; and send, to the client system, instructions for providing a response to the user request, wherein the response comprises attribute information about the specific entity. ### Claim 20 A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to: access visual data from a client system associated with a user, wherein the visual data comprises images portraying one or more objects; receive, from the client system, a user request, wherein the user request comprises a coreference to a target object; resolve the coreference to the target object from among the one or more objects; resolve the target object to a specific entity; and send, to the client system, instructions for providing a response to the user request, wherein the response comprises attribute information about the specific entity.