# US 12,190,888 B2 · System and method for secure training of speech processing systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,190,888 B2
- Document ID: US12190888B2
- Status: Granted
- Kind: B2
- Application number: 17/840,795
- Filed: 2022-06-15
- Published: 2023-12-21
- Granted: 2025-01-07
- Earliest claimed: 2022-06-15
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Nuance Communications, Inc.
- Inventors: Shou-Chun Yin; Junho Park; Dushyant Sharma; DoYeong Kim; Francesco Nespoli
- Web page: https://patentgrep.com/patent/US12190888B2

## CPC

- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 13/04: Details of speech synthesis systems, e.g. synthesiser structure or memory management
- G10L 15/063: {Training}
- G16H 10/60: for patient-specific data, e.g. for electronic patient records
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 15/16: using artificial neural networks
- G16H 15/00: ICT specially adapted for medical reports, e.g. generation or transmission thereof
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

A method, computer program product, and computing system for generating an obscured speech signal from an input speech signal and an obscured transcription from a transcription of the input speech signal. A speaker embedding may be extracted from the input speech signal. A speaker embedding delta may be generated based upon, at least in part, the extracted speaker embedding and a synthetic speaker embedding. A synthetic speech signal may be generated from the obscured speech signal using the synthetic speaker embedding. A residual signal may be generated based upon, at least in part, the obscured speech signal and the speaker embedding delta. A speech processing system may be trained using the obscured transcription, the synthetic speech signal, the speaker embedding delta, and the residual signal.

## Claims

### Claim 1

A computer-implemented method, executed on a computing device, comprising:
generating an obscured speech signal from an input speech signal and an obscured transcription from a transcription of the input speech signal, wherein the obscured speech signal and the obscured transcription include obscured representations of sensitive content from the input speech signal and the transcription of the input speech signal;
extracting a speaker embedding from the input speech signal;
generating a speaker embedding delta based upon, at least in part, the extracted speaker embedding and a synthetic speaker embedding;
generating a synthetic speech signal from the obscured speech signal using the synthetic speaker embedding;
generating a residual signal based upon, at least in part, the obscured speech signal and the speaker embedding delta; and
training a speech processing system using the obscured transcription, the synthetic speech signal, the speaker embedding delta, and the residual signal.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the sensitive content from the input speech signal and the transcription of the input speech signal includes personally identifiable information (PII) and/or protected health information (PHI).

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein generating the residual signal includes:
generating a resynthesized speech signal from the synthetic speech signal using the speaker embedding delta.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein generating the residual signal includes:
generating the residual signal as the difference between the obscured speech signal and the resynthesized speech signal.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein training the speech processing system includes:
generating a reconstructed speech signal using the synthetic speech signal, the speaker embedding delta, and the residual signal.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein training the speech processing system includes:
training the speech processing system with the reconstructed speech signal and the obscured transcription.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein generating the obscured speech signal and the obscured transcription includes:
marking each obscured representation of sensitive content in the obscured transcription, thus defining a plurality of obscured portion markings.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein training the speech processing system includes:
training the speech processing system with a reconstructed speech signal, the obscured transcription, and the plurality of obscured portion markings.

### Claim 9

A computing system comprising:
a memory; and
a processor configured to generate an obscured speech signal from an input speech signal and an obscured transcription from a transcription of the input speech signal, wherein the obscured speech signal and the obscured transcription include obscured representations of sensitive content from the input speech signal and the transcription of the input speech signal, wherein the processor is further configured to extract a speaker embedding from the input speech signal, wherein the processor is further configured to generate a speaker embedding delta based upon, at least in part, the extracted speaker embedding and a synthetic speaker embedding, wherein the processor is further configured to generate a synthetic speech signal from the obscured speech signal using the synthetic speaker embedding, wherein the processor is further configured to generate a residual signal based upon, at least in part, the obscured speech signal and the speaker embedding delta, and wherein the processor is further configured to train a speech processing system using the obscured transcription, the synthetic speech signal, the speaker embedding delta, and the residual signal.

### Claim 10 (depends on 9)

The computing system of claim 9, wherein the sensitive content from the input speech signal and the transcription of the input speech signal includes personally identifiable information (PII) and/or protected health information (PHI).

### Claim 11 (depends on 9)

The computing system of claim 9, wherein generating the residual signal includes:
generating a resynthesized speech signal from the synthetic speech signal using the speaker embedding delta.

### Claim 12 (depends on 11)

The computing system of claim 11, wherein generating the residual signal includes:
generating the residual signal as the difference between the obscured speech signal and the resynthesized speech signal.

### Claim 13 (depends on 9)

The computing system of claim 9, wherein training the speech processing system includes:
generating a reconstructed speech signal using the synthetic speech signal, the speaker embedding delta, and the residual signal.

### Claim 14 (depends on 12)

The computing system of claim 12, wherein training the speech processing system includes:
training the speech processing system with the reconstructed speech signal and the obscured transcription.

### Claim 15 (depends on 9)

The computing system of claim 9, wherein generating the obscured speech signal and the obscured transcription includes:
marking each obscured representation of sensitive content in the obscured transcription, thus defining a plurality of obscured portion markings.

### Claim 16

A computer program product residing on a non-transitory computer readable medium having a plurality of instructions stored thereon which, when executed by a processor, cause the processor to perform operations comprising:
generating an obscured speech signal from an input speech signal and an obscured transcription from a transcription of the input speech signal, wherein the obscured speech signal and the obscured transcription include obscured representations of sensitive content from the input speech signal and the transcription of the input speech signal;
extracting a speaker embedding from the input speech signal;
generating a speaker embedding delta based upon, at least in part, the extracted speaker embedding and a synthetic speaker embedding;
generating a synthetic speech signal from the obscured speech signal using the synthetic speaker embedding;
generating a resynthesized speech signal from the synthetic speech signal using the speaker embedding delta;
generating a residual signal based upon, at least in part, the obscured speech signal and the resynthesized speech signal; and
training a speech processing system using the obscured transcription, the synthetic speech signal, the speaker embedding delta, and the residual signal.

### Claim 17 (depends on 16)

The computer program product of claim 16, wherein the sensitive content from the input speech signal and the transcription of the input speech signal includes personally identifiable information (PII) and/or protected health information (PHI).

### Claim 18 (depends on 16)

The computer program product of claim 16, wherein training the speech processing system includes:
generating a reconstructed speech signal using the synthetic speech signal, the speaker embedding delta, and the residual signal.

### Claim 19 (depends on 18)

The computer program product of claim 18, wherein training the speech processing system includes:
training the speech processing system with the reconstructed speech signal and the obscured transcription.

### Claim 20 (depends on 16)

The computer program product of claim 16, wherein generating the obscured speech signal and the obscured transcription includes:
marking each obscured representation of sensitive content in the obscured transcription, thus defining a plurality of obscured portion markings.