Multi-frame capture system for enhanced visual reasoning in augmented reality devices

A method comprising:

• detecting, via a microphone of a device, user speech;
• upon detecting the user speech, initiating a multi-frame capture sequence by:
• capturing, via at least one camera of the device, a series of frames at a first frame capture rate;
• upon detecting a human hand within a frame of the series of frames, increasing a frame capture rate from the first frame capture rate to a second frame capture rate;
• and
• terminating the multi-frame capture sequence upon detecting an end of user speech;
• transcribing the user speech to text;
• constructing a first prompt for use as input with a multimodal large language model (MLLM), the first prompt comprising an instruction, a sequence of messages with each message containing a captured frame and a corresponding timestamp, and a user query represented by the text derived from transcribing the user speech;
• transmitting, via a network interface of the device, the first prompt to the MLLM for processing by the MLLM and receiving back a first output;
• constructing a second prompt for use as input with a large language model (LLM), the second prompt comprising an instruction, at least a portion of the first output from the MLLM, and a user query represented by the text derived from transcribing the user speech;
• transmitting, via the network interface, the second prompt to the LLM for processing by the LLM and receiving back a response;
• and
• outputting the response.

A device comprising:

• means for initiating a multi-frame capture sequence upon detecting user speech obtained via the microphone, by:
• capturing, via the at least one camera, a series of frames at a first frame capture rate;
• upon detecting a human hand within a frame of the series of frames, increasing a frame capture rate from the first frame capture rate to a second frame capture rate;
• and
• terminating the multi-frame capture sequence upon detecting an end of user speech;
• transcribing the user speech to text;
• means for constructing a first prompt for use as input with a multimodal large language model (MLLM), the first prompt comprising an instruction, a sequence of messages with each message containing a captured frame and a corresponding timestamp, and a user query represented by the text derived from transcribing the user speech;
• means for transmitting, via the network interface, the first prompt to the MLLM for processing by the MLLM and receiving back a first output;
• means for constructing a second prompt for use as input with a large language model (LLM), the second prompt comprising an instruction, at least a portion of the first output from the MLLM, and a user query represented by the text derived from transcribing the user speech;
• means for transmitting, via the network interface, the second prompt to the LLM for processing by the the LLM and receiving back a response;
• and
• means for outputting the response.