On demand contextual support agent with spatial awareness

A method for toggling a visibility of holograms and for correlating tagged spatial data with output of a large language model (LLM), said method comprising:

• accessing scene data describing a real-world scene in which a three-dimensional (3D) object is located;
• accessing a digital file that models the 3D object, wherein the digital file includes tagged spatial data associated with the 3D object;
• receiving, in real time and from a user, user input directed to the 3D object located in the real-world scene, wherein the user input includes a gesture input, wherein a hand-pointing vector is generated based on the gesture input, and wherein the hand-pointing vector is used to determine that the user input is directed to the 3D object;
• providing the scene data, the digital file, and the user input as inputs to an LLM, wherein the LLM is tasked with correlating the user input with the tagged spatial data of the digital file, and wherein the LLM is further tasked with generating a response to the user input;
• and
• while the LLM's response is being provided to the user, toggling display of a visual cue that overlays at least a portion of the 3D object.

A computer system for toggling a visibility of tagged spatial data and for correlating the tagged spatial data with output of a large language model (LLM), said computer system comprising:

• a processor system;
• and
• a storage system comprising instructions that are executable by the processor system to cause the computer system to:
• access scene data describing a real-world scene in which a three-dimensional (3D) object is located, wherein the scene data is tagged with spatial data describing at least the 3D object;
• receive, in real time and from a user, user input directed to the 3D object located in the real-world scene, wherein the user input includes a query related to the 3D object;
• provide the scene data and the user input as inputs to an LLM, wherein the LLM is tasked with correlating the user input with the tagged spatial data, and wherein the LLM is further tasked with generating a response to the user input;
• and
• while the LLM's response is being provided to the user, toggle display of a visual cue that overlays at least a portion of the 3D object.