# US 12,118,123 B2 · Privacy-aware meeting room transcription from audio-visual stream

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,123 B2
- Document ID: US12118123B2
- Status: Granted
- Kind: B2
- Application number: 17/755,892
- Filed: 2019-11-18
- Published: 2022-12-01
- Granted: 2024-10-15
- Earliest claimed: 2019-11-18
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Oliver Siohan; Takaki Makino; Richard Rose; Otavio Braga; Hank Liao; Basilio Garcia Castillo
- Web page: https://patentgrep.com/patent/US12118123B2

## CPC

- G06F 21/6254: {by anonymising data, e.g. decorrelating personal data from the owner's identification}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06F 21/32: using biometric data, e.g. fingerprints, iris scans or voiceprints
- G06F 21/6245: {Protecting personal data, e.g. for financial or medical purposes}
- G10L 17/02: Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
- H04L 12/1831: {Tracking arrangements for later retrieval, e.g. recording contents, participants activities or behavior, network status}
- H04N 7/155: {involving storage of or access to video conference sessions (tracking arrangements for later retrieval of a computer conference content or participants activities H04L12/1831)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method for a privacy-aware transcription includes receiving audio-visual signal including audio data and image data for a speech environment and a privacy request from a participant in the speech environment where the privacy request indicates a privacy condition of the participant. The method further includes segmenting the audio data into a plurality of segments. For each segment, the method includes determining an identity of a speaker of a corresponding segment of the audio data based on the image data and determining whether the identity of the speaker of the corresponding segment includes the participant associated with the privacy condition. When the identity of the speaker of the corresponding segment includes the participant, the method includes applying the privacy condition to the corresponding segment. The method also includes processing the plurality of segments of the audio data to determine a transcript for the audio data.

## Claims

### Claim 1

A method comprising:
receiving, at data processing hardware, an audio-visual signal comprising audio data and image data, the audio data corresponding to speech utterances from a plurality of participants in a speech environment and the image data representing faces of the plurality of participants in the speech environment;
receiving, at the data processing hardware, a privacy request from a participant of the plurality of participants, the privacy request indicating a privacy condition associated with the participant in the speech environment;
segmenting, by the data processing hardware, the audio data into a plurality of segments;
for a respective segment of the plurality of segments of the audio data:
determining, by the data processing hardware, from among the plurality of participants, an identity of a speaker of the respective segment of the audio data based on the image data;
determining, by the data processing hardware, that the identity of the speaker of the respective segment comprises the participant associated with the privacy condition indicated by the received privacy request; and
based on determining that the identity of the speaker of the respective segment comprises the participant, omitting transcribing the respective segment based on the privacy condition indicated by the received privacy request; and
processing, by the data processing hardware, the plurality of segments of the audio data to determine a transcript for the audio data, the transcript comprising text corresponding to each segment of the plurality of segments other than the respective segment and omitting text corresponding to the respective segment.

### Claim 2 (depends on 1)

The method of claim 1, further comprising, based on determining that the identity of the speaker of the respective segment comprises the participant, deleting, by the data processing hardware, the respective segment of the audio data after determining the transcript.

### Claim 3 (depends on 1)

The method of claim 1, further comprising, based on determining that the identity of the speaker of the respective segment comprises the participant, augmenting, by the data processing hardware, the corresponding segment of the image data to visually conceal the identity of the speaker of the respective segment of the audio data.

### Claim 4 (depends on 1)

The method of claim 1, further comprising, based on determining that the identity of the speaker of the respective segment comprises the participant, modifying, by the data processing hardware, the transcript to not include the identity of the speaker.

### Claim 5 (depends on 1)

The method of claim 1, wherein each other segment of the plurality of segments comprises respective speech utterances not spoken by the participant associated with the privacy condition indicated by the received privacy request.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
obtaining, by the data processing hardware, a content-specific condition indicating a type of content to exclude from the transcript,
wherein omitting transcribing the respective segment is further based on the content-specific condition.

### Claim 7 (depends on 1)

The method of claim 1, wherein determining, from among the plurality of participants, the identity of the speaker of the respective segment of the audio data comprises:
determining a plurality of candidate identities for the speaker based on the image data;
for each candidate identity of the plurality of candidate identities, generating a confidence score indicating a likelihood that a face of a corresponding candidate identity based on the image data comprises a speaking face of the respective segment of the audio data; and
selecting the identity of the speaker of the respective segment of the audio data as the candidate identity of the plurality of candidate identifies associated with the highest confidence score.

### Claim 8 (depends on 1)

The method of claim 1, wherein the data processing hardware resides on a device that is local to at least one participant of the plurality of participants.

### Claim 9 (depends on 1)

The method of claim 1, wherein the image data comprises high-definition video processed by the data processing hardware.

### Claim 10 (depends on 1)

The method of claim 1, wherein processing the plurality of segments of the audio data to determine the transcript for the audio data further comprises processing the image data to determine the transcript.

### Claim 11

A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving an audio-visual signal comprising audio data and image data, the audio data corresponding to speech utterances from a plurality of participants in a speech environment and the image data representing faces of the plurality of participants in the speech environment;
receiving a privacy request from a participant of the plurality of participants, the privacy request indicating a privacy condition associated with the participant in the speech environment;
segmenting the audio data into a plurality of segments;
for a respective segment of the plurality of segments of the audio data:
determining from among the plurality of participants, an identity of a speaker of the respective segment of the audio data based on the image data;
determining that the identity of the speaker of the respective segment comprises the participant associated with the privacy condition indicated by the received privacy request; and
based on determining that the identity of the speaker of the respective segment comprises the participant, omitting transcribing the respective segment based on the privacy condition indicated by the received privacy request; and
processing the plurality of segments of the audio data to determine a transcript for the audio data, the transcript comprising text corresponding to each segment of the plurality of segments other than the respective segment and omitting text corresponding to the respective segment.

### Claim 12 (depends on 11)

The system of claim 11, wherein, based on determining that the identity of the speaker of the respective segment comprises the participant, the operations further comprise deleting the respective segment of the audio data after determining the transcript.

### Claim 13 (depends on 11)

The system of claim 11, wherein, based on determining that the identity of the speaker of the respective segment comprises the participant, the operations further comprise, augmenting the corresponding segment of the image data to visually conceal the identity of the speaker of the respective segment of the audio data.

### Claim 14 (depends on 11)

The system of claim 11, wherein, based on determining that the identity of the speaker of the respective segment comprises the participant, the operations further comprise, modifying the transcript to not include the identity of the speaker.

### Claim 15 (depends on 11)

The system of claim 11, wherein each other segment of the plurality of segments comprises respective speech utterances not spoken by the participant associated with the privacy condition indicated by the received privacy request.

### Claim 16 (depends on 11)

The system of claim 11, wherein the operations further comprise:
obtaining a content-specific condition indicating a type of content to exclude from the transcript,
wherein omitting transcribing the respective segment is further based on the content-specific condition.

### Claim 17 (depends on 11)

The system of claim 11, wherein determining, from among the plurality of participants, the identity of the speaker of the respective segment of the audio data comprises:
determining a plurality of candidate identities for the speaker based on the image data;
for each candidate identity of the plurality of candidate identities, generating a confidence score indicating a likelihood that a face of a corresponding candidate identity based on the image data comprises a speaking face of the respective segment of the audio data; and
selecting the identity of the speaker of the respective segment of the audio data as the candidate identity of the plurality of candidate identifies associated with the highest confidence score.

### Claim 18 (depends on 11)

The system of claim 11, wherein the data processing hardware resides on a device that is local to at least one participant of the plurality of participants.

### Claim 19 (depends on 11)

The system of claim 11, wherein the image data comprises high-definition video processed by the data processing hardware.

### Claim 20 (depends on 11)

The system of claim 11, wherein processing the plurality of segments of the audio data to determine the transcript for the audio data further comprises processing the image data to determine the transcript.