# US 2026/0311694 A1 · Streaming audio and video for real-time voice interactions generative inference

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0311694 A1
- Document ID: US20260311694A1
- Status: Published application
- Kind: A1
- Application number: 19/098,795
- Filed: 2025-04-02
- Published: 2026-10-08
- Earliest claimed: 2025-04-02
- Assignees: OpenAI OpCo, LLC
- Applicants: OpenAI OpCo, LLC.
- Inventors: Mianna Chen; Tomer Kaftan; Alex Kirillov; Edede Oiwoh; Olaoluwa Okelola; Brendan Quinn; Jacqueline Shannon; Daniel Smith, II; Tao Xu; Rowan Zellers; Yu Zhang
- Web page: https://patentgrep.com/patent/US20260311694A1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 25/57: for processing of video signals
- G10L 25/93: Discriminating between voiced and unvoiced parts of speech signals (G10L25/90 takes precedence)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Disclosed are systems, apparatuses, processes, and computer-readable media for real-time modes of a generative response engine. The present technology includes different technologies to improve real-time operation to create natural and dynamic interactions to user experiences. A method of the present technology includes receiving, by a computing service, first audio frames associated with a speech instance at a client device and first video frames, wherein the first audio frames are provided at a first rate and the first video frames are provided at a variable rate; providing keyframes from the first video frames to send to a generative response engine for inference operation; transmitting the first audio frames and the keyframes to the generative response engine; and transmitting, to the client device, a stream of generated data generated by the generative response engine based on the first audio frames and the keyframes.

## Claims

### Claim 1

A method comprising:
receiving, by a computing service, first audio frames associated with a speech instance at a client device and first video frames, wherein the first audio frames are provided at a first rate and the first video frames are provided at a variable rate;
providing keyframes from the first video frames to send to a generative response engine for inference operation;
transmitting the first audio frames and the keyframes to the generative response engine; and
transmitting, to the client device, a stream of generated data generated by the generative response engine based on the first audio frames and the keyframes.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining an end of the speech instance based on identification of an independent clause in the first audio frames and an identification of silence; and
transmitting a signal to the generative response engine to complete the inference operation based on detection of the end of the speech instance.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
receiving the identification of the silence from the client device.

### Claim 4 (depends on 2)

The method of claim 2, further comprising:
determining an audio frame in the first audio frames corresponds to the silence at the client device.

### Claim 5 (depends on 1)

The method of claim 1, wherein at least one video frame of the keyframes is provided by the client device before the first audio frames.

### Claim 6 (depends on 1)

The method of claim 1, wherein at least one video frame of the keyframes is provided during silence associated with the first audio frames.

### Claim 7 (depends on 1)

The method of claim 1, wherein providing keyframes from the first video frames comprises:
selecting a keyframe from a plurality of the first video frames based on a static frame rate to provide to the generative response engine during the speech instance.

### Claim 8 (depends on 1)

The method of claim 1, wherein selecting the keyframes from the first video frames comprises:
selecting the keyframes from the first video frames to send to the generative response engine for the inference operation after detecting an end to the speech instance.

### Claim 9 (depends on 1)

The method of claim 1, wherein a keyframe detector model is configured to identify the keyframes to provide to the generative response engine.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
while transmitting the stream of generated data, receiving second audio frames from the client device indicating an interruption;
identifying a portion of the stream of generated data not presented at the client device; and
transmitting a signal to the generative response engine identifying the portion of the stream of generated data not presented at the client device.

### Claim 11 (depends on 10)

The method of claim 10, wherein the generative response engine is configured to drop tokens associated with the portion of the stream of generated data not presented at the client device from a context window.

### Claim 12 (depends on 10)

The method of claim 10, further comprising:
determining whether the second audio frames correspond to an interruption.

### Claim 13

A computing device, comprising:
at least one memory; and
at least one processor coupled to the at least one memory and configured to:
receive, by a computing service, first audio frames associated with a speech instance at a client device and first video frames, wherein the first audio frames are provided at a first rate and the first video frames are provided at a variable rate;
provide keyframes from the first video frames to send to a generative response engine for inference operation;
transmit the first audio frames and the keyframes to the generative response engine; and
transmit, to the client device, a stream of generated data generated by the generative response engine based on the first audio frames and the keyframes.

### Claim 14 (depends on 13)

The computing device of claim 13, wherein the at least one processor is configured to:
determine an end of the speech instance based on identification of an independent clause in the first audio frames and an identification of silence; and
transmit a signal to the generative response engine to complete the inference operation based on detection of the end of the speech instance.

### Claim 15 (depends on 14)

The computing device of claim 14, wherein the at least one processor is configured to:
receive the identification of the silence from the client device.

### Claim 16 (depends on 14)

The computing device of claim 14, wherein the at least one processor is configured to:
determine an audio frame in the first audio frames corresponds to the silence at the client device.

### Claim 17 (depends on 13)

The computing device of claim 13, wherein at least one video frame of the keyframes is provided by the client device before the first audio frames, or at least one video frame of the keyframes is provided during silence associated with the first audio frames.

### Claim 18 (depends on 13)

The computing device of claim 13, wherein the at least one processor is configured to:
select a keyframe from a plurality of the first video frames based on a static frame rate to provide to the generative response engine during the speech instance.

### Claim 19 (depends on 13)

The computing device of claim 13, wherein the at least one processor is configured to:
select the keyframes from the first video frames to send to the generative response engine for the inference operation after detecting an end to the speech instance.

### Claim 20 (depends on 13)

The computing device of claim 13, wherein the at least one processor is configured to:
while transmitting the stream of generated data, receive second audio frames from the client device indicating an interruption;
identify a portion of the stream of generated data not presented at the client device; and
transmit a signal to the generative response engine identifying the portion of the stream of generated data not presented at the client device.
