# US 2022/0044667 A1 · Speaker identity and content de-identification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044667 A1
- Document ID: US20220044667A1
- Status: Published application
- Kind: A1
- Application number: 17/452,563
- Filed: 2021-10-27
- Published: 2022-02-10
- Earliest claimed: 2020-04-28
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Aris Gkoulalas-Divanis; Xu Wang; Paul R. Bastide; Rohit Ranchal
- Web page: https://patentgrep.com/patent/US20220044667A1

## CPC

- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 21/00: Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility (G10L19/00 takes precedence)
- G06F 21/6254: {by anonymising data, e.g. decorrelating personal data from the owner's identification}
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 17/02: Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
- G06F 40/279: Recognition of textual entities
- G06F 2221/2113: Multi-level security, e.g. mandatory access control
- G10L 13/033: Voice editing, e.g. manipulating the voice of the synthesiser
- G10L 2021/0135: {Voice conversion or morphing}
- H04K 1/00: Secret communication
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04K: SECRET COMMUNICATION; JAMMING OF COMMUNICATION

## Abstract

One embodiment of the invention provides a method for speaker identity and content de-identification under privacy guarantees. The method comprises receiving input indicative of privacy protection levels to enforce, extracting features from a speech recorded in a voice recording, recognizing and extracting textual content from the speech, parsing the textual content to recognize privacy-sensitive personal information about an individual, generating de-identified textual content by anonymizing the personal information to an extent that satisfies the privacy protection levels and conceals the individual's identity, and mapping the de-identified textual content to a speaker who delivered the speech. The method further comprises generating a synthetic speaker identity based on other features that are dissimilar from the features to an extent that satisfies the privacy protection levels, and synthesizing a new speech waveform based on the synthetic speaker identity to deliver the de-identified textual content. The new speech waveform conceals the speaker's identity.

## Claims

### Claim 1

A method for speaker identity and content de-identification under data privacy guarantees, comprising:
receiving a set of voice recordings of different speeches delivered by different speakers;
for each of the different speakers, extracting features corresponding to the speaker from at least one of the different speeches, wherein the at least one speech is delivered by the speaker;
clustering all extracted features into different clusters based on similarity;
extracting textual content from a first speech of the different speeches, wherein the first speech is delivered by a first speaker of the different speakers;
generating de-identified textual content by anonymizing privacy-sensitive personal information included in the textual content;
generating a synthetic speaker identity corresponding to the first speaker based on extracted features included in a first cluster of the different clusters, wherein the first cluster is different from a second cluster of the different clusters, and the second cluster includes extracted features corresponding to the first speaker; and
synthesizing a new speech waveform based on the synthetic speaker identity to deliver the de-identified textual content, wherein the new speech waveform is different from a speech waveform of the first speech.

### Claim 2 (depends on 1)

The method of claim 1, wherein each cluster comprises extracted features corresponding to one or more of the plurality of speakers, and the one or more speakers have similar speech waveforms.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
receiving input indicative of at least one level of privacy protection, wherein the synthetic speaker identity and the de-identified textual content satisfy the at least one level of privacy protection.

### Claim 4 (depends on 3)

The method of claim 3, wherein the extracted features included in the first cluster are dissimilar from the extracted features corresponding to the first speaker to an extent that satisfies the at least one level of privacy protection.

### Claim 5 (depends on 1)

The method of claim 1, wherein the first cluster lies furthest away from the second cluster.

### Claim 6 (depends on 1)

The method of claim 1, wherein generating the synthetic speaker identity comprises:
applying an aggregation function over the extracted features included in the first cluster, wherein the synthetic speaker identity is further based on aggregated features resulting from applying the aggregation function.

### Claim 7

A system for speaker identity and content de-identification under privacy guarantees, comprising:
at least one processor; and
a non-transitory processor-readable memory device storing instructions that when executed by the at least one processor causes the at least one processor to perform operations including:
receiving a set of voice recordings of different speeches delivered by different speakers;
for each of the different speakers, extracting features corresponding to the speaker from at least one of the different speeches, wherein the at least one speech is delivered by the speaker;
clustering all extracted features into different clusters based on similarity;
extracting textual content from a first speech of the different speeches, wherein the first speech is delivered by a first speaker of the different speakers;
generating de-identified textual content by anonymizing privacy-sensitive personal information included in the textual content;
generating a synthetic speaker identity corresponding to the first speaker based on extracted features included in a first cluster of the different clusters, wherein the first cluster is different from a second cluster of the different clusters, and the second cluster includes extracted features corresponding to the first speaker; and
synthesizing a new speech waveform based on the synthetic speaker identity to deliver the de-identified textual content, wherein the new speech waveform is different from a speech waveform of the first speech.

### Claim 8 (depends on 7)

The system of claim 7, wherein each cluster comprises extracted features corresponding to one or more of the plurality of speakers, and the one or more speakers have similar speech waveforms.

### Claim 9 (depends on 7)

The system of claim 7, wherein the operations further include:
receiving input indicative of at least one level of privacy protection, wherein the synthetic speaker identity and the de-identified textual content satisfy the at least one level of privacy protection.

### Claim 10 (depends on 9)

The system of claim 9, wherein the extracted features included in the first cluster are dissimilar from the extracted features corresponding to the first speaker to an extent that satisfies the at least one level of privacy protection.

### Claim 11 (depends on 7)

The system of claim 7, wherein the first cluster lies furthest away from the second cluster.

### Claim 12 (depends on 7)

The system of claim 7, wherein generating the synthetic speaker identity comprises:
applying an aggregation function over the extracted features included in the first cluster, wherein the synthetic speaker identity is further based on aggregated features resulting from applying the aggregation function.

### Claim 13

A computer program product for speaker identity and content de-identification under privacy guarantees, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:
receive a set of voice recordings of different speeches delivered by different speakers;
for each of the different speakers, extract features corresponding to the speaker from at least one of the different speeches, wherein the at least one speech is delivered by the speaker;
cluster all extracted features into different clusters based on similarity;
extract textual content from a first speech of the different speeches, wherein the first speech is delivered by a first speaker of the different speakers;
generate de-identified textual content by anonymizing privacy-sensitive personal information included in the textual content;
generate a synthetic speaker identity corresponding to the first speaker based on extracted features included in a first cluster of the different clusters, wherein the first cluster is different from a second cluster of the different clusters, and the second cluster includes extracted features corresponding to the first speaker; and
synthesize a new speech waveform based on the synthetic speaker identity to deliver the de-identified textual content, wherein the new speech waveform is different from a speech waveform of the first speech.

### Claim 14 (depends on 13)

The computer program product of claim 13, wherein each cluster comprises extracted features corresponding to one or more of the plurality of speakers, and the one or more speakers have similar speech waveforms.

### Claim 15 (depends on 13)

The computer program product of claim 13, wherein the program instructions executable by the processor further cause the processor to:
receive input indicative of at least one level of privacy protection, wherein the synthetic speaker identity and the de-identified textual content satisfy the at least one level of privacy protection.

### Claim 16 (depends on 15)

The computer program product of claim 15, wherein the extracted features included in the first cluster are dissimilar from the extracted features corresponding to the first speaker to an extent that satisfies the at least one level of privacy protection.

### Claim 17 (depends on 13)

The computer program product of claim 13, wherein the first cluster lies furthest away from the second cluster.

### Claim 18 (depends on 13)

The computer program product of claim 13, wherein the program instructions executable by the processor cause the processor to generate the synthetic speaker identity by:
applying an aggregation function over the extracted features included in the first cluster, wherein the synthetic speaker identity is further based on aggregated features resulting from applying the aggregation function.
