Systems and methods for providing query-based triggers in real-time videos associated with a vehicle

A device, comprising:

• one or more processors configured to:
• receive a query identifying one or more events to be captured by a camera of a vehicle, video data generated by the camera of the vehicle, and location data associated with the vehicle,
• wherein the query is a free-text query received by the camera associated with the vehicle;
• generate a prompt requesting captions of video frames included in the video data;
• process the video data, with a large language model and based on the prompt, to generate the captions of the video frames included in the video data;
• process the query and the captions, with a phrase model, to determine categories of words provided in the query and the captions;
• calculate matching scores between the query and the captions based on the categories of words and a dictionary of words associated with driving events;
• identify an event, of the one or more events associated with the vehicle, based on the matching scores and one or more thresholds;
• and
• perform one or more actions based on the event, wherein the one or more processors, to perform the one or more actions, are configured to one or more of:
• provide an alert identifying the event to a driver of the vehicle;
• provide a report identifying the event to a fleet manager of the vehicle;
• or
• provide the video data associated with the event for display by the device.

A non-transitory computer-readable medium storing a set of instructions, the set of instructions comprising:

• one or more instructions that, when executed by one or more processors of a device, cause the device to:
• receive a query identifying one or more events to be captured by a camera of a vehicle, video data generated by the camera of the vehicle, and location data associated with the vehicle,
• wherein the query includes a structured sentence or a list of keywords;
• generate a prompt requesting captions of video frames included in the video data;
• process the video data, with a large language model and based on the prompt, to generate the captions of the video frames included in the video data;
• process the query and the captions, with a phrase model, to determine categories of words provided in the query and the captions;
• calculate matching scores between the query and the captions based on the categories of words, a dictionary of words associated with driving events, and the location data associated with the vehicle;
• identify an event, of the one or more events associated with the vehicle, based on the matching scores and one or more thresholds;
• and
• perform one or more actions based on the event, wherein the instructions that cause the device to perform the one or more actions include instructions that cause the device to perform one or more of:
• provide an alert identifying the event to a driver of the vehicle;
• provide a report identifying the event to a fleet manager of the vehicle;
• or
• provide the video data associated with the event for display by the device.