Systems and methods for assisting a driver using a foundation model in a shared-autonomy driving mode of a vehicle

A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to:

• process, in a vehicle operating in a shared-autonomy driving mode, inputs including a driver vehicle-control input that includes at least one of turning a steering wheel of the vehicle, operating a throttle of the vehicle, or operating a brake of the vehicle, vehicle state information, external-road-agent state information, vehicle environmental sensor data, and map data using one or more encoder neural networks that have been trained to extract features for a large language model (LLM);
• input the extracted features to the LLM;
• predict, using the LLM based on the extracted features, an objective of a driver of the vehicle;
• and
• execute, based on an output from the LLM, one or more actions to assist the driver in meeting the predicted objective, wherein the one or more actions include controlling, at least in part, operation of the vehicle.

A method, comprising:

• processing, in a vehicle operating in a shared-autonomy driving mode, inputs including a driver vehicle-control input that includes at least one of turning a steering wheel of the vehicle, operating a throttle of the vehicle, or operating a brake of the vehicle, vehicle state information, external-road-agent state information, vehicle environmental sensor data, and map data using one or more encoder neural networks that have been trained to extract features for a large language model (LLM) and inputting the extracted features to the LLM;
• predicting, using the LLM based on the extracted features, an objective of a driver of the vehicle;
• and
• executing, based on an output from the LLM, one or more actions to assist the driver in meeting the predicted objective, wherein the one or more actions include controlling, at least in part, operation of the vehicle.