Location and navigation assistant

A system for navigation assistance in an indoor environment comprising:

• a processing system;
• and
• a computer-readable medium having encoded thereon, computer-readable instructions that, when executed by the processing system, cause the system to perform operations comprising:
• receiving a reference image depicting a location within the indoor environment, wherein the reference image includes a natural language annotation describing the location depicted by the reference image;
• calculating a numerical representation of the reference image capturing a semantic content of the reference image and the natural language annotation;
• storing the numerical representation in a reference database that is configured to store a plurality of numerical representations of reference images and annotations associated with the indoor environment;
• receiving an input image from a user device, the input image depicting a current position of the user device within the indoor environment;
• in response to receiving the input image from the user device, identifying, by a multimodal language model, the current position of the user device based on a comparison of a numerical representation of the input image against the plurality of numerical representations stored by the reference database;
• receiving a natural language input from the user device, wherein the natural language input includes a user-defined objective;
• in response to receiving the natural language input from the user device:
• identifying, by the multimodal language model, a destination within the indoor environment from the user-defined objective;
• and
• outputting, by the multimodal language model, a sequence of natural language directions defining a path from the current position of the user device to the destination identified from the user-defined objective.

A computer-readable storage medium for navigation assistance in an indoor environment, the computer-readable storage medium having encoded thereon, computer-readable instructions that, when executed by a system, cause the system to perform operations comprising:

• receiving a reference image depicting a location within the indoor environment, wherein the reference image includes a natural language annotation describing the location depicted by the reference image;
• calculating a numerical representation of the reference image capturing a semantic content of the reference image and the natural language annotation;
• storing the numerical representation in a reference database that is configured to store a plurality of numerical representations of reference images and annotations associated with the indoor environment;
• receiving an input image from a user device, the input image depicting a current position of the user device within the indoor environment;
• in response to receiving the input image from the user device, identifying, by a multimodal language model, the current position of the user device based on a comparison of a numerical representation of the input image against the plurality of numerical representations stored by the reference database;
• receiving a natural language input from the user device, wherein the natural language input includes a user-defined objective;
• in response to receiving the natural language input from the user device:
• identifying, by the multimodal language model, a destination within the indoor environment from the user-defined objective;
• and
• outputting, by the multimodal language model, a sequence of natural language directions defining a path from the current position of the user device to the destination identified from the user-defined objective.