Multimodal entity and coreference resolution for assistant systems

One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

• access visual data from a client system associated with a user, wherein the visual data comprises images portraying one or more objects;
• receive, from the client system, a user request, wherein the user request comprises a coreference to a target object;
• resolve the coreference to the target object from among the one or more objects;
• resolve the target object to a specific entity;
• and
• send, to the client system, instructions for providing a response to the user request, wherein the response comprises attribute information about the specific entity.

A system comprising:

• one or more processors;
• and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
• access visual data from a client system associated with a user, wherein the visual data comprises images portraying one or more objects;
• receive, from the client system, a user request, wherein the user request comprises a coreference to a target object;
• resolve the coreference to the target object from among the one or more objects;
• resolve the target object to a specific entity;
• and
• send, to the client system, instructions for providing a response to the user request, wherein the response comprises attribute information about the specific entity.