# US 2026/0065602 A1 · Multi-frame capture system for enhanced visual reasoning in augmented reality devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0065602 A1
- Document ID: US20260065602A1
- Status: Published application
- Kind: A1
- Application number: 18/824,621
- Filed: 2024-09-04
- Published: 2026-03-05
- Earliest claimed: 2024-09-04
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Viktoria Hwang; Mitchell Kuppersmith; Karen Stolzenberg
- Web page: https://patentgrep.com/patent/US20260065602A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06V 40/107: {Static hand or arm}
- G10L 15/18: using natural language modelling
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G02B 2027/014: {comprising information/image processing systems}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

A device and system for visual reasoning in augmented reality environments employs adaptive multi-frame capture triggered by detection of user speech. Upon detecting speech, the device or system captures image frames at an initial frame capture rate, increasing capture frequency when a hand is detected in a captured image. Timestamped frames and transcribed speech form a prompt for a multimodal large language model, which extracts relevant details with constrained output. A separate language model then generates a final response. This two-stage approach optimizes processing efficiency and accuracy while preserving privacy by limiting continuous visual data collection. The system enables more natural and context-aware interactions in AR settings without complex gesture recognition algorithms.

## Claims

### Claim 1

A device comprising:
at least one processor;
at least one camera;
a display component;
a microphone;
a network interface; and
memory storing instructions that, when executed by the at least one processor, cause the device to perform operations comprising:
upon detecting user speech obtained via the microphone, initiating a multi-frame capture sequence, by:
capturing, via the at least one camera, a series of frames at a first frame capture rate;
upon detecting a human hand within a frame of the series of frames, increasing a frame capture rate from the first frame capture rate to a second frame capture rate; and
terminating the multi-frame capture sequence upon detecting an end of user speech;
transcribing the user speech to text;
constructing a first prompt for use as input with a multimodal large language model (MLLM), the first prompt comprising an instruction, a sequence of messages with each message containing a captured frame and a corresponding timestamp, and a user query represented by the text derived from transcribing the user speech;
transmitting, via the network interface, the first prompt to the MLLM for processing by the MLLM and receiving back a first output;
constructing a second prompt for use as input with a large language model (LLM), the second prompt comprising an instruction, at least a portion of the first output from the MLLM, and a user query represented by the text derived from transcribing the user speech;
transmitting, via the network interface, the second prompt to the LLM for processing by the the LLM and receiving back a response; and
outputting the response.

### Claim 2 (depends on 1)

The device of claim 1, wherein the operations further comprise:
subjecting each frame captured at the first frame capture rate to a computer vision model trained to detect the presence of a human hand.

### Claim 3 (depends on 1)

The device of claim 1, wherein the first frame capture rate is one frame every three, four, or five seconds, and wherein the second frame capture rate is at least twice the first frame capture rate.

### Claim 4 (depends on 1)

The device of claim 1, wherein the instruction included in the first prompt directs the MLLM to analyze the sequence of messages and a user query represented by the text derived from transcribing the user speech, and to provide details for understanding the intent of the user query.

### Claim 5 (depends on 1)

The device of claim 1, wherein the instruction included in the second prompt for use as input with the LLM directs the LLM to generate the response to the user query based on the first output from the MLLM while focusing on extracted details relevant to the user query about objects or actions depicted in the captured frames, and to format the response for appropriate presentation via an output capability of the device.

### Claim 6 (depends on 1)

The device of claim 1, wherein outputting the generated response comprises at least one of:
presenting the response visually on the display component;
outputting the response as audio via a speaker of the device; or
presenting the response as a combination of visual output on the display component and audio output via the speaker.

### Claim 7 (depends on 1)

The device of claim 1, wherein the operations further comprise:
cropping each captured frame to a field of view corresponding to a view of the user, before including the frame as a message in the first prompt.

### Claim 8 (depends on 1)

The device of claim 1, wherein constructing the first prompt further comprises:
arranging the sequence of messages containing captured frames and corresponding timestamps in chronological order to represent an entire interaction over time.

### Claim 9 (depends on 1)

The device of claim 1, wherein the MLLM is constrained to output a maximum of thirty tokens, thereby reducing processing time and focusing the response generated by the MLLM on the most relevant details extracted from the frames in the sequence of messages.

### Claim 10

A method comprising:
detecting, via a microphone of a device, user speech;
upon detecting the user speech, initiating a multi-frame capture sequence by:
capturing, via at least one camera of the device, a series of frames at a first frame capture rate;
upon detecting a human hand within a frame of the series of frames, increasing a frame capture rate from the first frame capture rate to a second frame capture rate; and
terminating the multi-frame capture sequence upon detecting an end of user speech;
transcribing the user speech to text;
constructing a first prompt for use as input with a multimodal large language model (MLLM), the first prompt comprising an instruction, a sequence of messages with each message containing a captured frame and a corresponding timestamp, and a user query represented by the text derived from transcribing the user speech;
transmitting, via a network interface of the device, the first prompt to the MLLM for processing by the MLLM and receiving back a first output;
constructing a second prompt for use as input with a large language model (LLM), the second prompt comprising an instruction, at least a portion of the first output from the MLLM, and a user query represented by the text derived from transcribing the user speech;
transmitting, via the network interface, the second prompt to the LLM for processing by the LLM and receiving back a response; and
outputting the response.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
subjecting each frame captured at the first frame capture rate to a computer vision model trained to detect the presence of a human hand.

### Claim 12 (depends on 10)

The method of claim 10, wherein the first frame capture rate is one frame every three, four, or five seconds, and wherein the second frame capture rate is at least twice the first frame capture rate.

### Claim 13 (depends on 10)

The method of claim 10, wherein the instruction included in the first prompt directs the MLLM to analyze the sequence of messages and a user query represented by the text derived from transcribing the user speech, and to provide details for understanding the intent of the user query.

### Claim 14 (depends on 10)

The method of claim 10, wherein the instruction included in the second prompt for use as input with the LLM directs the LLM to generate the response to the user query based on the first output from the MLLM while focusing on extracted details relevant to the user query about objects or actions depicted in the captured frames, and to format the response for appropriate presentation via an output capability of the device.

### Claim 15 (depends on 10)

The method of claim 10, wherein outputting the generated response comprises at least one of:
presenting the response visually on a display component of the device;
outputting the response as audio via a speaker of the device; or
presenting the response as a combination of visual output on the display component and audio output via the speaker.

### Claim 16 (depends on 10)

The method of claim 10, further comprising:
cropping each captured frame to a field of view corresponding to a view of the user, before including the frame as a message in the first prompt.

### Claim 17 (depends on 10)

The method of claim 10, wherein constructing the first prompt further comprises:
arranging the sequence of messages containing captured frames and corresponding timestamps in chronological order to represent an entire interaction over time.

### Claim 18 (depends on 10)

The method of claim 10, wherein the MLLM is constrained to output a maximum of thirty tokens, thereby reducing processing time and focusing the response generated by the MLLM on the most relevant details extracted from the frames in the sequence of messages.

### Claim 19

A device comprising:
means for initiating a multi-frame capture sequence upon detecting user speech obtained via the microphone, by:
capturing, via the at least one camera, a series of frames at a first frame capture rate;
upon detecting a human hand within a frame of the series of frames, increasing a frame capture rate from the first frame capture rate to a second frame capture rate; and
terminating the multi-frame capture sequence upon detecting an end of user speech;
transcribing the user speech to text;
means for constructing a first prompt for use as input with a multimodal large language model (MLLM), the first prompt comprising an instruction, a sequence of messages with each message containing a captured frame and a corresponding timestamp, and a user query represented by the text derived from transcribing the user speech;
means for transmitting, via the network interface, the first prompt to the MLLM for processing by the MLLM and receiving back a first output;
means for constructing a second prompt for use as input with a large language model (LLM), the second prompt comprising an instruction, at least a portion of the first output from the MLLM, and a user query represented by the text derived from transcribing the user speech;
means for transmitting, via the network interface, the second prompt to the LLM for processing by the the LLM and receiving back a response; and
means for outputting the response.

### Claim 20 (depends on 19)

The device of claim 19, further comprising:
means for subjecting each frame captured at the first frame capture rate to a computer vision model trained to detect the presence of a human hand.
