Augmenting of driving scenarios using contrastive learning

A system comprising:

• one or more processors;
• and
• a memory storing instructions that, when executed by the one or more processors, cause the system to perform:
• obtaining a textual prompt;
• encoding the textual prompt;
• obtaining candidate sequences of sensor data from different modalities, each sequence comprising a plurality of sequential frames, each of the candidate sequences being evaluated against the textual prompt based on a contrastive loss;
• encoding the candidate sequences of sensor data;
• embedding position information, within the encoded candidate sequences, indicating relative timestamps associated with each of the sequential frames of the sensor data;
• concatenating the encoded candidate sequences of sensor data, including the embedded position information;
• transforming the concatenated and encoded frames of sensor data to form transformed candidate sequences;
• determining a particular candidate sequence as a match between the transformed candidate sequences and the encoded textual prompt based on the contrastive loss;
• and
• generating a hierarchical structure that encapsulates navigation data of the particular candidate sequence.