Method for finetuning a large language model for a visual navigation task

A method for finetuning a language model, the method comprising:

• providing, in a memory, a scene graph representing a scene, the scene graph comprising a plurality of nodes and a plurality of edges interconnecting the plurality of nodes, the plurality of nodes representing a plurality of rooms in the scene and a plurality of objects in the scene;
• selecting, with a processor, (i) a starting room from the plurality of rooms and (ii) a target object, the target object being an object type or a particular object from the plurality of objects;
• determining, with the processor, based on the scene graph, a ground truth shortest path from the starting room to the target object in the scene, the ground truth shortest path being defined by a first sequence of connected nodes in the scene graph that represent a first sequence of navigation steps to navigate the scene from the starting room to the target object;
• generating, with the processor, a natural language prompt configured to prompt a language model to predict, given a current room and the target object in the scene, a next node in the first sequence of connected nodes that define the ground truth shortest path from the starting room to the target object in the scene;
• and
• finetuning the language model using the natural language prompt and the ground truth shortest path.