# US 2025/0148784 A1 · Multimodal State Tracking via Scene Graphs for Assistant Systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2025/0148784 A1 - Document ID: US20250148784A1 - Status: Published application - Kind: A1 - Application number: 19/018,019 - Filed: 2025-01-13 - Published: 2025-05-08 - Earliest claimed: 2020-09-01 - Assignees: Meta Platforms, Inc. - Applicants: Meta Platforms, Inc. - Inventors: Satwik Kottur - Web page: https://patentgrep.com/patent/US20250148784A1 ## CPC - G06V 20/35: {Categorising the entire scene, e.g. birthday party or wedding scene} - G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)} - G06F 16/9536: Search customisation based on social or collaborative filtering - G06V 20/20: in augmented reality scenes - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract In one embodiment, a method includes receiving, from a client system associated with a user, a first user request that includes a reference to a target object and one or more of an attribute or a relationship of the target object. Visual data including one or more images portraying the target object may then be accessed, and the reference may be resolved to the target object portrayed in the one or more images. Object information of the target object that corresponds to the referenced attribute or relationship of the first user request may be determined based on a visual analysis of the one or more images. Finally, responsive to receiving the first user request, the object information of the target object may be stored in a multimodal dialog state. ## Claims ### Claim 1 A method comprising: receiving a first user request comprising a first reference to a target object and a first query associated with the target object; accessing visual data that includes the target object; resolving the first reference to the target object in the visual data; determining a first attribute to answer the first query by analyzing the visual data; storing the target object in association with the first attribute in a multimodal dialog state based on the first reference being resolved to the target object and the first attribute being determined as the answer; receiving a second user request comprising a second reference to the target object and a second query associated with the target object; determining that that the second user request is associated with the target object based on the target object being stored in the multimodal dialog state; determining a second attribute to answer the second query based on the visual data and the target object being stored in the multimodal dialog state; and storing the second attribute in association with the target object in the multimodal dialog state. ### Claim 2 (depends on 1) The method of claim 1, wherein: the second reference includes an identifying attribute of the target object; the visual data includes other objects that include the identifying attribute; and the multimodal dialog state omits data related the other objects. ### Claim 3 (depends on 1) The method of claim 1, wherein the first user request includes one or more of a gaze of a user or a gesture of the user. ### Claim 4 (depends on 1) The method of claim 1, wherein the determining the second attribute comprises: determining a visual attribute of the target object based on the second query; determining an identifier for the target object based on the visual attribute; and retrieving the second attribute based on the identifier. ### Claim 5 (depends on 4) The method of claim 4, wherein the identifier is an identifying name of the target object or an identifying number of the target object. ### Claim 6 (depends on 1) The method of claim 1, wherein the multimodal dialog state includes an attribute associated with audio data. ### Claim 7 (depends on 1) The method of claim 1, wherein the second user request includes relational information in the visual data. ### Claim 8 A system comprising: one or more processors; and a memory coupled to the one or more processors, the memory comprising instructions executable by the one or more processors, the one or more processors being operable when executing the instructions to: receive a first user request comprising a first reference to a target object and a first query associated with the target object; access visual data that includes the target object; resolve the first reference to the target object in the visual data; determine a first attribute to answer the first query by analyzing the visual data; store the target object in association with the first attribute in a multimodal dialog state based on the first reference being resolved to the target object and the first attribute being determined as the answer; receive a second user request comprising a second reference to the target object and a second query associated with the target object; determine that that the second user request is associated with the target object based on the target object being stored in the multimodal dialog state; determine a second attribute to answer the second query based on the visual data and the target object being stored in the multimodal dialog state; and store the second attribute in association with the target object in the multimodal dialog state. ### Claim 9 (depends on 8) The system of claim 8, wherein: the second reference includes an identifying attribute of the target object; the visual data includes other objects that include the identifying attribute; and the multimodal dialog state omits data related the other objects. ### Claim 10 (depends on 8) The system of claim 8, wherein the first user request includes one or more of a gaze of a user or a gesture of the user. ### Claim 11 (depends on 8) The system of claim 8, wherein to determine the second attribute, the one or more processors are further operable when executing the instructions to: determining a visual attribute of the target object based on the second query; determining an identifier for the target object based on the visual attribute; and retrieving the second attribute based on the identifier. ### Claim 12 (depends on 11) The system of claim 11, wherein the identifier is an identifying name of the target object or an identifying number of the target object. ### Claim 13 (depends on 8) The system of claim 8, wherein the multimodal dialog state includes an attribute associated with audio data. ### Claim 14 (depends on 8) The system of claim 8, wherein the second user request includes relational information in the visual data. ### Claim 15 At least one computer readable storage medium comprising a set of instructions, which when executed by a computing device, cause the computing device to: receive a first user request comprising a first reference to a target object and a first query associated with the target object; access visual data that includes the target object; resolve the first reference to the target object in the visual data; determine a first attribute to answer the first query by analyzing the visual data; store the target object in association with the first attribute in a multimodal dialog state based on the first reference being resolved to the target object and the first attribute being determined as the answer; receive a second user request comprising a second reference to the target object and a second query associated with the target object; determine that that the second user request is associated with the target object based on the target object being stored in the multimodal dialog state; determine a second attribute to answer the second query based on the visual data and the target object being stored in the multimodal dialog state; and store the second attribute in association with the target object in the multimodal dialog state. ### Claim 16 (depends on 15) The at least one computer readable storage medium of claim 15, wherein: the second reference includes an identifying attribute of the target object; the visual data includes other objects that include the identifying attribute; and the multimodal dialog state omits data related the other objects. ### Claim 17 (depends on 15) The at least one computer readable storage medium of claim 15, wherein the first user request includes one or more of a gaze of a user or a gesture of the user. ### Claim 18 (depends on 15) The at least one computer readable storage medium of claim 15, wherein to determine the second attribute, the instructions, when executed, cause the computing device to: determining a visual attribute of the target object based on the second query; determining an identifier for the target object based on the visual attribute; and retrieving the second attribute based on the identifier. ### Claim 19 (depends on 18) The at least one computer readable storage medium of claim 18, wherein the identifier is an identifying name of the target object or an identifying number of the target object. ### Claim 20 (depends on 15) The at least one computer readable storage medium of claim 15, wherein: the multimodal dialog state includes an attribute associated with audio data; and the second user request includes relational information in the visual data.