Speech style transfer

A speech synthesizing apparatus, comprising:

• an interface system;
• and
• a control system comprising one or more processors and one or more non-transitory storage media operatively coupled to the one or more processors, the control system configured for implementing a speech synthesizer, the speech synthesizer including a content extractor, a first neural network and a second neural network, the first neural network comprising a bi-directional recurrent neural network, the second neural network comprising a hierarchy of modules, each module operating at a different temporal resolution, the first neural network and the second neural network having been trained according to a process comprising:
• (a) receiving, via the interface system and by the content extractor, first audio data corresponding to first speech of a first person;
• (b) producing, by the content extractor, a first time-stamped phoneme sequence and first pitch contour data corresponding to the first speech;
• (c) receiving, by the first neural network, the first time-stamped phoneme sequence and the first pitch contour data;
• (d) producing, by the first neural network, first neural network output corresponding to the first time-stamped phoneme sequence and the first pitch contour data, the first neural network output comprising a plurality of frame sizes, each of the frame sizes corresponding with a temporal resolution of a module of the second neural network;
• (e) receiving, by the second neural network, the first neural network output;
• (f) generating, by the second neural network, first predicted audio signals;
• (g) comparing the first predicted audio signals to first test data, the test data being audio data corresponding to speech of the first person;
• (h) determining a loss function value for the first predicted audio signals;
• and
• (i) repeating (a) through (h) until a difference between a current loss function value for the first predicted audio signals and a prior loss function value for the first predicted audio signals is less than or equal to a predetermined value, wherein the control system is configured for controlling the speech synthesizer module for speech generation and wherein the speech generation comprises:
• (j) receiving, by the content extractor and via the interface system, second audio data corresponding to second speech of a second person, second time-stamped text corresponding to the second speech and first identification data corresponding to speech of the first person;
• (k) producing, by the content extractor, a second time-stamped phoneme sequence and second pitch contour data corresponding to the second speech;
• (l) receiving, by the first neural network, the second time-stamped phoneme sequence and the second pitch contour data of (k);
• (m) producing, by the first neural network, first neural network output corresponding to the second time-stamped phoneme sequence and the second pitch contour data of (k);
• (n) receiving, by the second neural network, the first neural network output of (m) and the first identification data;
• and
• (o) generating, by the second neural network, synthesized audio data corresponding to the first neural network output of (m) and the first identification data.

A speech synthesizing apparatus, comprising:

• an interface system;
• and
• a control system comprising one or more processors and one or more non-transitory storage media operatively coupled to the one or more processors, the control system configured for implementing a speech synthesizer, the speech synthesizer including a content extractor, a first neural network and a second neural network, the first neural network comprising a bi-directional recurrent neural network, the second neural network comprising a hierarchy of modules, each module operating at a different temporal resolution, the first neural network and the second neural network having been trained according to a process comprising:
• (a) receiving, via the interface system and by the content extractor, first audio data corresponding to first speech of a target speaker;
• (b) producing, by the content extractor, a first time-stamped phoneme sequence and first pitch contour data corresponding to the first speech;
• (c) receiving, by the first neural network, the first time-stamped phoneme sequence and the first pitch contour data;
• (d) producing, by the first neural network, first neural network output corresponding to the first time-stamped phoneme sequence and the first pitch contour data, the first neural network output comprising a plurality of frame sizes, each of the frame sizes corresponding with a temporal resolution of a module of the second neural network;
• (e) receiving, by the second neural network, the first neural network output;
• (f) generating, by the second neural network, first predicted audio signals;
• (g) comparing the first predicted audio signals to first test data;
• (h) determining a loss function value for the first predicted audio signals;
• and
• (i) repeating (a) through (h) until a difference between a current loss function value for the first predicted audio signals and a prior loss function value for the first predicted audio signals is less than or equal to a predetermined value, wherein the control system is configured for controlling the speech synthesizer module for speech generation and wherein the speech generation comprises:
• (j) receiving, by the content extractor and via the interface system, second audio data corresponding to second speech of a source speaker, second time-stamped text corresponding to the second speech and first identification data corresponding to speech of the target speaker;
• (k) producing, by the content extractor, a second time-stamped phoneme sequence and second pitch contour data corresponding to the second speech;
• (l) receiving, by the first neural network, the second time-stamped phoneme sequence and the second pitch contour data of (k);
• (m) producing, by the first neural network, first neural network output corresponding to the second time-stamped phoneme sequence and the second pitch contour data of (k);
• (n) receiving, by the second neural network, the first neural network output of (m) and the first identification data;
• and
• (o) generating, by the second neural network, synthesized audio data corresponding to the first neural network output of (m) and the first identification data.