Multimodal State Tracking via Scene Graphs for Assistant Systems

A system comprising:

• one or more processors;
• and
• a memory coupled to the one or more processors, the memory comprising instructions executable by the one or more processors, the one or more processors being operable when executing the instructions to:
• receive a first user request comprising a first reference to a target object and a first query associated with the target object;
• access visual data that includes the target object;
• resolve the first reference to the target object in the visual data;
• determine a first attribute to answer the first query by analyzing the visual data;
• store the target object in association with the first attribute in a multimodal dialog state based on the first reference being resolved to the target object and the first attribute being determined as the answer;
• receive a second user request comprising a second reference to the target object and a second query associated with the target object;
• determine that that the second user request is associated with the target object based on the target object being stored in the multimodal dialog state;
• determine a second attribute to answer the second query based on the visual data and the target object being stored in the multimodal dialog state;
• and
• store the second attribute in association with the target object in the multimodal dialog state.

At least one computer readable storage medium comprising a set of instructions, which when executed by a computing device, cause the computing device to:

• receive a first user request comprising a first reference to a target object and a first query associated with the target object;
• access visual data that includes the target object;
• resolve the first reference to the target object in the visual data;
• determine a first attribute to answer the first query by analyzing the visual data;
• store the target object in association with the first attribute in a multimodal dialog state based on the first reference being resolved to the target object and the first attribute being determined as the answer;
• receive a second user request comprising a second reference to the target object and a second query associated with the target object;
• determine that that the second user request is associated with the target object based on the target object being stored in the multimodal dialog state;
• determine a second attribute to answer the second query based on the visual data and the target object being stored in the multimodal dialog state;
• and
• store the second attribute in association with the target object in the multimodal dialog state.