# US 2026/0073365 A1 · Multimodal entity and coreference resolution for assistant systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0073365 A1
- Document ID: US20260073365A1
- Status: Published application
- Kind: A1
- Application number: 19/390,440
- Filed: 2025-11-14
- Published: 2026-03-12
- Earliest claimed: 2019-10-18
- Assignees: Meta Platforms Technologies, LLC
- Applicants: Meta Platforms Technologies, LLC
- Inventors: Shivani Poddar; Seungwhan Moon; Paul Anthony Crook; Rajen Subba
- Web page: https://patentgrep.com/patent/US20260073365A1

## CPC

- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 9/453: {Help systems}
- G06F 9/485: {Task life-cycle, e.g. stopping, restarting, resuming execution (G06F9/4881 takes precedence)}
- G06F 9/4862: {the task being a mobile agent, i.e. specifically designed to migrate}
- G06F 9/4881: {Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues}
- G06F 9/54: Interprogram communication
- G06F 9/547: {Remote procedure calls [RPC]; Web services}
- G06F 16/3329: Natural language query formulation
- G06F 16/90332: {Natural language query formulation or dialogue systems}
- G06F 16/9536: Search customisation based on social or collaborative filtering
- G06F 18/2321: using statistics or function optimisation, e.g. modelling of probability density functions
- G06F 40/126: Character encoding
- G06F 40/205: Parsing
- G06F 40/216: using statistical methods
- G06F 40/242: Dictionaries
- G06F 40/253: Grammatical analysis; Style critique
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 40/295: Named entity recognition
- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G06F 40/56: Natural language generation
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 3/098: Distributed learning, e.g. federated learning
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06N 20/00: Machine learning
- G06Q 10/00: Administration; Management
- G06Q 10/109: Time management, e.g. calendars, reminders, meetings or time accounting
- G06Q 10/1093: Calendar-based scheduling for persons or groups
- G06Q 10/40: Business processes related to social networking or social networking services
- G06Q 10/42: Determination of affinities or common interests between users
- G06Q 10/48: using social graphs
- G06Q 30/0603: {Catalogue creation or management}
- G06Q 30/0631: {Recommending goods or services}
- G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)}
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06V 20/20: in augmented reality scenes
- G06V 20/30: in albums, collections or shared content, e.g. social network photos or video
- G06V 40/16: Human faces, e.g. facial parts, sketches or expressions
- G06V 40/25: {Recognition of walking or running movements, e.g. gait recognition}
- G10L 15/063: {Training}
- G10L 15/08: Speech classification or search
- G10L 15/16: using artificial neural networks
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 15/32: Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
- H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
- H04L 51/18: Commands or executable codes
- H04L 51/212: using filtering or selective blocking
- H04L 51/214: using selective forwarding
- H04L 51/222: using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area
- H04L 51/224: providing notification on incoming messages, e.g. pushed notifications of received messages
- H04L 51/52: for supporting social networking services
- H04L 67/306: User profiles
- H04L 67/75: Indicating network or usage conditions on the user display
- H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 2209/541: Client-server
- G06N 3/048: Activation functions
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 40/174: {Facial expression recognition}
- G06V 2201/10: Recognition assisted with metadata
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 15/07: to the speaker
- G10L 2015/0631: {Creating reference templates; Clustering}
- G10L 2015/088: {Word spotting}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2015/227: {of the speaker; Human-factor methodology}
- G10L 2015/228: {of application context}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

In one embodiment, a method includes receiving, at a client system, an audio input, where the audio input comprises a coreference to a target object, accessing visual data from one or more camera associated with the client system, where the visual data comprises images portraying one or more objects, resolving the coreference to the target object from among the one or more objects, resoling the target object to a specific entity, and providing, at the client system, a response to the audio input, where the response comprises information about the specific entity.

## Claims

### Claim 1

A method for operating a head-word device, the method comprising:
capturing, with a camera of the head-worn device, a plurality of images portraying a plurality of objects;
storing, in a memory, information associated with the plurality of images, wherein the information comprises timing information indicating when each object of the plurality of objects appears in the plurality of images;
receiving, by a microphone of the head-word device, an audio input from a user of the head-worn device, wherein the audio input comprises a request and a coreference to a target object;
identifying the target object from among the plurality of objects based at least in part on the coreference and the timing information;
determining an action associated with the request;
executing the action, based at least in part on the target object, to obtain a result; and
providing an output to the user based at least in part on the result.

### Claim 2 (depends on 1)

The method of claim 1, wherein the plurality of images are captured over a period of time, and the information comprises the plurality of images.

### Claim 3 (depends on 2)

The method of claim 2, wherein the plurality of images are part of a video captured by the camera.

### Claim 4 (depends on 2)

The method of claim 2, further comprising:
analyzing the information to identify attributes of the plurality of objects, wherein the attributes are used to identify the target object.

### Claim 5 (depends on 2)

The method of claim 2, further comprising:
analyzing the information to identify relational information between the plurality of objects, wherein the relational information is used to identify the target object.

### Claim 6 (depends on 1)

The method of claim 1, wherein the information comprises tags identifying two or more of the plurality of objects portrayed in the plurality of images.

### Claim 7 (depends on 6)

The method of claim 6, wherein the target object is identified based at least in part on the tags.

### Claim 8 (depends on 1)

The method of claim 1, further comprising performing speech recognition on the audio input to identify the request and the coreference.

### Claim 9 (depends on 1)

The method of claim 1, wherein the output is provided on a display of the head-worn device.

### Claim 10 (depends on 1)

The method of claim 1, wherein the output is provided by a speaker of the head-worn device.

### Claim 11

A non-transitory computer-readable storage medium including executable instructions that, when executed by one or more processors, causes the one or more processors to:
while a user is wearing a head-worn device:
cause a camera of the head-worn device to capture a plurality of images portraying a plurality of objects;
cause information associated with the plurality of images to be stored in a memory, wherein the information comprises timing information indicating when each object of the plurality of objects appears in the plurality of images;
receive, from a microphone of the head-word device, an audio input from the user of the head-worn device, wherein the audio input comprises a request and a coreference to a target object;
identify the target object from among the plurality of objects based at least in part on the coreference and the timing information;
determine an action associated with the request;
cause the action to be executed, based at least in part on the target object, to obtain a result; and
cause an output to be provided to the user based at least in part on the result.

### Claim 12 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the plurality of images are captured over a period of time, and the information comprises the plurality of images.

### Claim 13 (depends on 12)

The non-transitory computer-readable storage medium of claim 12, wherein the plurality of images are part of a video captured by the camera.

### Claim 14 (depends on 12)

The non-transitory computer-readable storage medium of claim 12, wherein the executable instructions further cause the one or more processors to:
analyze the information to identify attributes of the plurality of objects, wherein the attributes are used to identify the target object.

### Claim 15 (depends on 12)

The non-transitory computer-readable storage medium of claim 12, wherein the executable instructions further cause the one or more processors to:
analyze the information to identify relational information between the plurality of objects, wherein the relational information is used to identify the target object.

### Claim 16

A head-wearable device including a camera and a microphone, the head-wearable device configured to:
cause the camera of the head-worn device to capture a plurality of images portraying a plurality of objects;
cause information associated with the plurality of images to be stored in a memory, wherein the information comprises timing information indicating when each object of the plurality of objects appears in the plurality of images;
receiving, from the microphone of the head-word device, an audio input from a user of the head-worn device, wherein the audio input comprises a request and a coreference to a target object;
identifying the target object from among the plurality of objects based at least in part on the coreference and the timing information;
determining an action associated with the request;
cause the action to be executed, based at least in part on the target object, to obtain a result; and
cause an output to be provided to the user based at least in part on the result.

### Claim 17 (depends on 16)

The head-wearable device of claim 16, wherein the plurality of images are captured over a period of time, and the information comprises the plurality of images.

### Claim 18 (depends on 17)

The head-wearable device of claim 17, wherein the plurality of images are part of a video captured by the camera.

### Claim 19 (depends on 17)

The head-wearable device of claim 17, wherein the head-wearable device is further configured to:
analyze the information to identify attributes of the plurality of objects, wherein the attributes are used to identify the target object.

### Claim 20 (depends on 17)

The head-wearable device of claim 17, wherein the head-wearable device is further configured to:
analyze the information to identify relational information between the plurality of objects, wherein the relational information is used to identify the target object.
