# US 2026/0228444 A1 · Speaker modules in an audio-visual real-time transcription system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0228444 A1
- Document ID: US20260228444A1
- Status: Published application
- Kind: A1
- Application number: 19/044,674
- Filed: 2025-02-04
- Published: 2026-08-06
- Earliest claimed: 2025-02-04
- Assignees: RTC Vision Ltd.
- Applicants: RTC Vision Ltd.
- Inventors: Ron WEIN; Joseph RUBNER; Nir AVRAHAMI; Nir TSUK; Kfir GEDALYAHU
- Web page: https://patentgrep.com/patent/US20260228444A1

## CPC

- G06F 40/35: Discourse or dialogue representation
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The present invention provides a system and method for real-time target speaker recognition and transcription, enhancing accuracy in noisy and multi-speaker environments. Using audio and video inputs from a microphone and camera, the system processes features through neural networks, including cross-attention mechanisms, to isolate speaker-specific data and suppress noise. A Speaker Module tracks and identifies active speakers via face detection, recurrent neural networks, or decision mechanisms, ensuring transcription continuity even with temporary visual occlusions. Outputs generated by the invention include transcribed text and speaker labels, displayed on an augmented reality display or a mobile device as real-time subtitles. The system can suppress the display of the user's own speech for improved usability and supports multilingual transcription. This invention offers a robust solution for individuals with hearing impairments, enabling accurate comprehension in challenging acoustic settings.

## Claims

### Claim 1

A method of target speaker recognition and transcription of speech, comprising:
receiving an audio stream from a first input;
receiving a video stream from a second input;
computing a plurality of initial audio features from the audio stream;
computing a plurality of initial video features from the video stream;
applying a neural network to the plurality of initial video features, thereby generating a plurality of extended video features;
determining a correlation between the plurality of initial audio features and the plurality of extended video features, thereby generating a plurality of enhanced audio features;
processing the plurality of initial audio features, the plurality of initial video features and the plurality of enhanced audio features, thereby generating a plurality of target speaker features; and
generating an output.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising:
receiving a data stream from a third input; and
computing a plurality of initial auxiliary features from the data stream,
wherein generating a plurality of target speaker features comprises processing the plurality of initial audio features, the plurality of initial video features, the plurality of enhanced audio features and the plurality of initial auxiliary features.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the neural network comprises at least one convolution layer.

### Claim 4 (depends on 1)

The method according to claim 1, wherein generating the plurality of enhanced audio features comprises performing a computation solely on the plurality of extended video features.

### Claim 5 (depends on 1)

The method according to claim 1, wherein generating the plurality of enhanced audio features comprises one or more of:
applying at least one cross-attention layer to the plurality of initial audio features and the plurality of extended video features;
applying a recurrent neural network to the plurality of initial audio features, the plurality of initial video features and a plurality of previously computed target speaker features; and
utilizing a decision process to the plurality of initial audio features, the plurality of initial video features and a plurality of previously computed target speaker features.

### Claim 6 (depends on 1)

The method according to claim 1, wherein generating the plurality of target speaker features comprises applying a recurrent neural network to the plurality of initial audio features, the plurality of initial video features and the plurality of enhanced audio features.

### Claim 7 (depends on 1)

The method according to claim 1, wherein generating an output comprises:
applying a machine learning model to the plurality of target speaker features, thereby generating a target speaker label; and
outputting the target speaker label.

### Claim 8 (depends on 1)

The method according to claim 1, wherein generating an output comprises:
applying positional encoding to the plurality of enhanced audio features;
applying at least one transformer encoder layer, thereby generating a sequence of encoded vectors;
applying a transformer decoder layer to the sequence of encoded vectors, thereby generating a sequence of textual tokens;
concatenating the sequence of textual tokens, thereby generating a text transcription; and
outputting the text transcription.

### Claim 9 (depends on 8)

The method according to claim 8, further comprising:
applying a machine learning model to the plurality of target speaker features and at least one of a previously obtained text transcription and a previously obtained plurality of textual tokens, thereby generating a target speaker label; and
outputting the target speaker label.

### Claim 10 (depends on 9)

The method according to claim 9, further comprising:
determining whether a condition is met, wherein the condition comprises checking whether the target speaker label equals to a predetermined speaker label; and
responsive to determining that the condition is met, skipping the step of outputting the text transcription.

### Claim 11 (depends on 1)

The method according to claim 1, wherein computing a plurality of initial audio features comprises:
receiving an audio signal from a microphone device;
computing a spectrogram of the audio signal; and
applying at least one convolution layer to the spectrogram.

### Claim 12 (depends on 1)

The method according to claim 1, wherein computing a plurality of initial video features comprises:
receiving a frame sequence from a camera device;
responsive to detection of one or more faces within a frame in the frame sequence, determining a bounding box for each detected face; and
extracting from the frame a 2-dimensional (2D) array of pixels limited by the respective bounding box for each detected face.

### Claim 13 (depends on 12)

The method according to claim 12, further comprising:
responsive to detecting absence of a target speaker in the plurality of video streams:
skipping the step of generating a plurality of extended video features;
skipping the step of generating a plurality of enhanced audio features; and
processing the plurality of initial audio features, thereby generating a plurality of target speaker features; and
generating an output.

### Claim 14 (depends on 12)

The method according to claim 12, further comprising:
detecting a particular face at the center of each frame in the frame sequence of a configured duration;
tracking the detected face across subsequent frames; and
prioritizing the target speaker features corresponding to the tracked face across subsequent frames.

### Claim 15 (depends on 12)

The method according to claim 12, further comprising:
receiving a signal to select a particular face from a third input;
tracking the selected face across subsequent frames; and
prioritizing the target speaker features corresponding to the tracked face across subsequent frames.

### Claim 16

A computer system, comprising
one or more microphone devices;
one or more camera devices;
a processor configured to execute stored executable instructions; and
a non-transitory computer readable medium storing executable instructions that, when executed by a processor, cause the computer system to perform a method of target speaker recognition and transcription, the method comprising:
receiving one or more audio streams as a first input;
receiving one or more video streams a second input;
computing a plurality of initial audio features from the one or more audio streams;
computing a plurality of initial video features from the one or more video streams;
applying a neural network to the plurality of initial video features, thereby generating a plurality of extended video features;
determining a correlation between the plurality of initial audio features and the plurality of extended video features, thereby generating a plurality of enhanced audio features;
processing the plurality of initial audio features, the plurality of initial video features and the plurality of enhanced audio features, thereby generating a plurality of target speaker features; and
generating an output.

### Claim 17 (depends on 16)

The computer system according to claim 16, further comprising one of an augmented reality display and virtual reality display, wherein generating an output comprises rendering a text transcription on the one of an augmented reality display, an auxiliary screen, an auxiliary projector or a virtual reality display.

### Claim 18 (depends on 16)

The computer system according to claim 16, wherein at least one of the one or more microphone devices and the one or more camera devices is coupled to the processor via at least in part a wireless link.

### Claim 19 (depends on 18)

The computer system according to claim 18, wherein the wireless link comprises one of a Bluetooth, Wi-Fi, Wi-Fi Direct, Wi-Fi HaLow, Ultra-Wideband, mmWave, 5G, and LiFi connection.
