AI-assisted remote guidance using augmented reality

A system comprising:

• one or more computing devices;
• and
• memory coupled to the one or more computing devices, the memory storing instructions, which when executed, cause the one or more computing devices to perform operations comprising:
• receiving a multi-modal input representing a query associated with a physical object, wherein the multi-modal input comprises one or more frames of a video and at least one of text or audio component inputs;
• detecting the physical object in the one or more frames using a machine learning model trained to perform object detection;
• determining based in part on an identification of the physical object, at least one response to the query associated with the physical object, the at least one response being identified by processing the at least one of the text or audio component inputs using a language processing model;
• determining a sequence of actions associated with the at least one response, the sequence of actions involving an interaction with at least one portion of the physical object;
• generating, a digital representation of the sequence of actions on a digital twin of the physical object, the digital representation comprising:
• the digital twin, and one or more gesture-icons representing the sequence of actions, each of the one or more gesture-icons being overlaid on corresponding portions of the digital twin of the physical object;
• and
• providing the digital representation to a user-device for presentation on a display, the digital representation being configured to be dynamically aligned to a changing view of the physical object on the display.

One or more non-transitory machine-readable storage media encoded with computer program instructions that when executed by one or more computing devices cause the one or more computing devices to perform operations comprising:

• receiving, a multi-modal input representing a query associated with a physical object, wherein the multi-modal input comprises one or more frames of a video and at least one of text or audio component inputs;
• detecting the physical object in the one or more frames using a machine learning model trained to perform object detection;
• determining based in part on an identification of the physical object, at least one response to the query associated with the physical object, the at least one response being identified by processing the at least one of the text or audio component inputs using a language processing model;
• determining, a sequence of actions associated with the at least one response, the sequence of actions involving an interaction with at least one portion of the physical object;
• generating, a digital representation of the sequence of actions on a digital twin of the physical object, the digital representation comprising:
• the digital twin, and one or more gesture-icons representing the sequence of actions, each of the one or more gesture-icons being overlaid on corresponding portions of the digital twin of the physical object;
• and
• providing the digital representation to a user-device for presentation on a display, the digital representation being configured to be dynamically aligned to a changing view of the physical object on the display.