Geometrically grounded large language models for zero-shot human activity forecasting in human-aware task planning

An apparatus comprising:

• a memory;
• processing circuitry coupled to the memory, the processing circuitry configured to:
• receive data from one or more sensors corresponding to a first time period,
• detect, from the received data, one or more activities of a person in a space,
• convert the detected one or more activities into a natural language narration;
• combine the natural language narration with sequence text that corresponds to an action that is sequential to the detected one or more activities,
• extract one or more items from a map corresponding to the space,
• determine a relevancy score for each of the extracted one or more items based on an output of a language model that receives as input the combination of the narration with the sequence text,
• correlate the extracted one or more items with one or more locations on the map based on the determined relevancy for each of the extracted one or more items, and
• a control signal for controlling a movement of one more devices based on the correlation of the extracted one or more items with the one or more locations on the map.

A non-transitory computer readable medium having instructions stored therein, which when executed by a processor cause the processor to execute a method comprising:

• receiving data from one or more sensors corresponding to a first time period,
• detecting, from the received data, one or more activities of a person in a space,
• converting the detected one or more activities into a natural language narration;
• combining the natural language narration with sequence text that corresponds to an action that is sequential to the detected one or more activities;
• extracting one or more items from a map corresponding to the space;
• determining a relevancy score for each of the extracted one or more items based on an output of a language model that receives as input the combination of the narration with the sequence text;
• correlating the extracted one or more items with one or more locations on the map based on the determined relevancy for each of the extracted one or more items;
• and
• outputting a control signal for controlling a movement of one more devices based on the correlation of the extracted one or more items with the one or more locations on the map.