In-Vehicle Object Queries with Large Multi-Modal Models
AssigneeNissan North America, Inc.
InventorsMarcell Jose Vazquez-Chanlatte, Corey Heath, Stefan Witwicki, Tomer Arnon
Life of this patent
Estimating the term…
Abstract
System and method for responding to queries about objects in a cabin of a vehicle. The system detects a trigger that causes an in-cabin camera to capture video of the cabin, and the system generates a history of captions for at least selected frames of the video by a large multi-modal model (LMM). The system converts a spoken query received by a microphone to a text-based prompt and generates, by the LMM, a response to the prompt based on the history of captions. The response is converted to speech that is output to a speaker.
Claims20
A method, comprising:
- detecting a trigger that causes one or more in-cabin cameras to capture one or more videos of a cabin environment of a vehicle;
- generating a first caption for a first frame of a first video of the one or more videos by a large multi-modal model (LMM);
- receiving a query concerning the cabin environment by a microphone;
- converting the query to a text-based prompt;
- generating a response to the prompt by the LMM based on the first caption;
- converting the response to speech;
- and
- causing a speaker to output the speech.
A system, comprising:
- one or more memories;
- and
- one or more processors configured to execute instructions stored in the one or more memories to:
- detect a trigger that causes one or more in-cabin cameras to capture one or more videos of a cabin environment of a vehicle;
- generate a first caption for a first frame of a first video of the one or more videos by a large multi-modal model (LMM);
- receive a query concerning the cabin environment by a microphone;
- convert the query to a text-based prompt;
- generate a response to the prompt by the LMM based on the first caption;
- convert the response to speech;
- and
- cause a speaker to output the speech.
A non-transitory computer-readable medium storing instructions operable to cause one or more processors to perform operations comprising:
- detecting a trigger that causes one or more in-cabin cameras to capture one or more videos of a cabin environment of a vehicle;
- generating a first caption for a first frame of a first video of the one or more videos by a large multi-modal model (LMM);
- receiving a query concerning the cabin environment by a microphone;
- converting the query to a text-based prompt;
- generating a response to the prompt by the LMM based on the first caption;
- converting the response to speech;
- and
- causing a speaker to output the speech.
Dependencies are inferred from the claim text.