Feature extraction for anonymized speech recognition

A computer-implemented method for processing speech data, the method comprising:

• receiving, by an application server operated by an entertainment service, a raw audio waveform containing speech data of a user, where the raw audio waveform is transmitted to the application server by a user device;
• generating a frequency representation of the raw audio waveform;
• removing a feature from the frequency representation that can be used to identify the user to produce a modified frequency representation;
• generating an anonymized representation of the raw audio waveform from the modified frequency representation;
• providing metadata and the anonymized representation to a speech processing module executing on the application server for processing,
• where the metadata identifies the feature that was removed from the frequency representation,
• where the speech processing module is operated by a third-party speech processing service, and
• where the speech processing module is allowed to copy elements of the anonymized representation to improve speech processing operations performed by the speech processing module;
• receiving a processed version of the anonymized representation from the speech processing module;
• and
• performing a system operation using the processed version of the anonymized representation.

A computer-implemented method for anonymizing speech data, the method comprising:

• receiving a raw audio waveform containing speech data of a user;
• segmenting the raw audio waveform into a plurality of intervals;
• generating a frequency domain representation of each interval;
• applying a post-processing function to smooth each frequency domain representation and remove at least a portion of speaker-identifying data from each frequency domain representation;
• identifying a frequency component in one of the plurality of intervals that represents a spoken word in the raw audio waveform and cannot be used to identify the user;
• extracting the frequency component from the corresponding interval;
• providing metadata and the frequency component to an analysis system for processing,
• where the metadata identifies the speaker-identifying data removed from each frequency domain representation by the post-processing function;
• receiving, from the analysis system, processed data;
• and
• determining, for a synthetic character, an appropriate emotional state or responsive action based at least in part on the processed data.

An application server for performing speech processing in accordance with a privacy regulation, the application server comprising:

• a processor;
• a first communication port configured to establish a first connection with a user device across a first network;
• a second communication port configured to establish a second connection with a speech processing server across a second network;
• and
• a memory coupled to the processor and configured to store instructions, which, when executed by the processor, cause the processor to perform operations including:
• receiving, via the first communication port, a raw audio waveform from the user device that includes speech data of a user;
• generating a frequency representation of the raw audio waveform;
• removing one or more frequency components from the frequency representation that can be used to identify the user to produce a modified frequency representation;
• generating an anonymized waveform from the modified frequency representation;
• transmitting, via the second communication port, metadata and the anonymized waveform to the speech processing server for processing,
• where the metadata identifies the one or more frequency components that were removed from the frequency representation, and
• where, upon completion of a speech processing operation, the speech processing server is allowed to copy elements of the anonymized waveform to improve an audio model employed by the speech processing server to identify speech within audio waveforms;
• receive, via the second communication port, a textual depiction of speech in the anonymized waveform from the speech processing server;
• determining a responsive action for a synthetic character based at least in part on the textual depiction, where the synthetic character is presented to the user by the user device;
• transmitting, via the first communication port, an instruction indicative of the responsive action to the user device;
• and
• causing the user device to execute the instruction, which prompts the synthetic character to perform the responsive action.

A computer-implemented method for processing speech data in accordance with a privacy regulation, the method comprising:

• receiving, from a user device executing an entertainment application, a raw audio waveform that includes speech data of a user, wherein the raw audio waveform is recorded by the user device in response to the user interacting with the entertainment application;
• providing metadata and the raw audio waveform to a speech processing environment controlled by a third party for processing,
• where the metadata identifies a frequency component of a frequency representation of the raw audio waveform that is to be modified;
• directing the third party to perform an anonymization operation on the raw audio waveform before performing a speech processing operation, wherein the anonymization operation includes
• generating the frequency representation of the raw audio waveform,
• modifying the frequency component of the frequency representation that could be used to identify the user in order to produce a modified frequency representation, and
• generating an anonymized representation of the raw audio waveform from the modified frequency representation;
• allowing the third-party speech processing system to copy an element of the anonymized representation to improve future instances of speech processing;
• receiving, from the speech processing environment, a textual depiction of speech in the anonymized representation of the raw audio waveform;
• determining a responsive action for a synthetic character based at least in part on the textual depiction;
• and
• transmitting, to the user device, an instruction indicative of the responsive action for execution by the user device, wherein execution of the instruction causes the synthetic character to perform the responsive action within the entertainment application.