# US 11,443,731 B2 · Systems and methods for generating synthesized speech responses to voice inputs by training a neural network model based on the voice input prosodic metrics and training voice inputs > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,731 B2 - Document ID: US11443731B2 - Status: Granted - Kind: B2 - Application number: 15/931,074 - Filed: 2020-05-13 - Published: 2021-10-14 - Granted: 2022-09-13 - Earliest claimed: 2020-04-09 - Assignees: Rovi Guides, Inc. - Applicants: Rovi Guides, Inc. - Inventors: Ankur Aher; Jeffry Copps Robert Jose - Web page: https://patentgrep.com/patent/US11443731B2 ## CPC - G10L 13/0335: {Pitch control} - G10L 25/63: for estimating an emotional state - G10L 13/00: Speech synthesis; Text to speech systems - G10L 13/02: Methods for producing synthetic speech; Speech synthesisers - G10L 13/06: Elementary speech units used in speech synthesisers; Concatenation rules - G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination - G10L 15/00: Speech recognition (G10L17/00 takes precedence) - G10L 15/10: using distance or distortion measures between unknown speech and reference templates - G10L 15/16: using artificial neural networks - G10L 15/18: using natural language modelling - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract The system provides a synthesized speech response to a voice input, based on the prosodic character of the voice input. The system receives the voice input and calculates at least one prosodic metric of the voice input. The at least one prosodic metric can be associated with a word, phrase, grouping thereof, or the entire voice input. The system also determines a response to the voice input, which may include the sequence of words that form the response. The system generates the synthesized speech response, by determining prosodic characteristics based on the response, and on the prosodic character of the voice input. The system outputs the synthesized speech response, which includes a more natural, relevant, or both answer to the call of the voice input. The prosodic character of the voice input and/or response may include pitch, note, duration, prominence, timbre, rate, and rhythm, for example. ## Claims ### Claim 1 A computer-implemented method for providing a synthesized speech response to a voice input with a neural network model trained by: receiving a plurality of training voice inputs, each associated with at least one respective voice input prosodic metric; extracting training data from the plurality of training voice inputs and the associated voice input prosodic metrics; processing the extracted training data for input into the neural network model; receiving a plurality of responses, each associated with at least one respective response prosodic metric; training the neural network model based on the extracted training data including the processed plurality of training voice inputs, the plurality of responses, the associated and processed voice input prosodic metrics, and the associated response prosodic metrics; and outputting, with the trained neural network model, information used to generate the synthesized speech response to the voice input, the method comprising: receiving the voice input; calculating at least one prosodic metric of the voice input; determining a response to the voice input; generating the synthesized speech response based on the calculated at least one prosodic metric of the voice input and the information outputted from the trained neural network model; and causing to be output the synthesized speech response. ### Claim 2 (depends on 1) The method of claim 1, wherein a plurality of reference responses are associated with the voice input, and wherein the prosodic characteristics of the synthesized speech response are further based on a relationship between the at least one prosodic metric and the plurality of reference responses. ### Claim 3 (depends on 2) The method of claim 2, further comprising identifying which of the plurality of reference responses comprise respective reference prosodic characteristics that are most closely related to the at least one prosodic metric, wherein the prosodic characteristics of the synthesized speech response are further based on the identified reference response. ### Claim 4 (depends on 2) The method of claim 2, further comprising identifying which of the plurality of reference responses is most closely related to the at least one prosodic metric, wherein the prosodic characteristics of the synthesized speech response are further based on the identified reference response. ### Claim 5 (depends on 1) The method of claim 1, further comprising determining an emotion metric based on the voice input, wherein the prosodic characteristics are further based on the emotion metric. ### Claim 6 (depends on 1) The method of claim 1, wherein: the voice input comprises a plurality of words, and each prosodic metric of the at least one prosodic metric corresponds to at least one word of the plurality of words. ### Claim 7 (depends on 1) The method of claim 1, wherein the at least one prosodic metric is selected from the group comprising pitch, note, duration, prominence, timbre, rate, rhythm, and any combination thereof, and wherein the processing the extracted training data for input into the neural network model includes for at least a portion of each of the plurality of training voice inputs an association with at least one from the group comprising a pitch indicator, a pitch for each frame, a pitch interval, a note, a speaking speed, a number of syllables, an age of an entity associated with the voice input, and a sentiment. ### Claim 8 (depends on 1) The method of claim 1, wherein synthesizing the synthesized speech response comprises: determining predicted prosodic characteristics of the response using a model; and modifying the predicted prosodic characteristics to generate the prosodic characteristics of the synthesized speech response. ### Claim 9 (depends on 1) The method of claim 1, wherein the prosodic characteristics are further based on at least one selected from the group comprising user voice input history, user language, user characteristics, user location, user preferences, and metadata tags associated with the user. ### Claim 10 (depends on 1) The method of claim 1, wherein the prosodic characteristics are further based on an interpolation operation affecting transitions in the synthesized speech response. ### Claim 11 A system for providing a synthesized speech response to a voice input with a neural network model trained by: receiving a plurality of training voice inputs, each associated with at least one respective voice input prosodic metric; extracting training data from the plurality of training voice inputs and the associated voice input prosodic metrics; processing the extracted training data for input into the neural network model; receiving a plurality of responses, each associated with at least one respective response prosodic metric; training the neural network model based on the extracted training data including the processed plurality of training voice inputs, the plurality of responses, the associated and processed voice input prosodic metrics, and the associated response prosodic metrics; and outputting, with the trained neural network model, information used to generate the synthesized speech response to the voice input, the system comprising: control circuitry for: receiving the voice input; calculating at least one prosodic metric of the voice input; determining a response to the voice input; and generating the synthesized speech response based on the calculated at least one prosodic metric of the voice input and the information outputted from the trained neural network model; and an output device for outputting the synthesized speech response. ### Claim 12 (depends on 11) The system of claim 11, wherein a plurality of reference responses are associated with the voice input, and wherein the prosodic characteristics of the synthesized speech response are further based on a relationship between the at least one prosodic metric and the plurality of reference responses. ### Claim 13 (depends on 12) The system of claim 12, wherein the control circuitry is further configured to identify which of the plurality of reference responses comprise respective reference prosodic characteristics that are most closely related to the at least one prosodic metric, wherein the prosodic characteristics of the synthesized speech response are further based on the identified reference response. ### Claim 14 (depends on 12) The system of claim 12, wherein the control circuitry is further configured to identify which of the plurality of reference responses is most closely related to the at least one prosodic metric, wherein the prosodic characteristics of the synthesized speech response are further based on the identified reference response. ### Claim 15 (depends on 11) The system of claim 11, wherein the control circuitry is further configured to determine an emotion metric based on the voice input, wherein the prosodic characteristics are further based on the emotion metric. ### Claim 16 (depends on 11) The system of claim 11, wherein: the voice input comprises a plurality of words, and each prosodic metric of the at least one prosodic metric corresponds to at least one word of the plurality of words. ### Claim 17 (depends on 11) The system of claim 11, wherein the at least one prosodic metric is selected from the group comprising pitch, note, duration, prominence, timbre, rate, rhythm, and any combination thereof, and wherein the processing the extracted training data for input into the neural network model includes for at least a portion of each of the plurality of training voice inputs an association with at least one from the group comprising a pitch indicator, a pitch for each frame, a pitch interval, a note, a speaking speed, a number of syllables, an age of an entity associated with the voice input, and a sentiment. ### Claim 18 (depends on 11) The system of claim 11, wherein the control circuitry is further configured to synthesize the synthesized speech response by: determining predicted prosodic characteristics of the response using a model; and modifying the predicted prosodic characteristics to generate the prosodic characteristics of the synthesized speech response. ### Claim 19 (depends on 11) The system of claim 11, wherein the prosodic characteristics are further based on at least one selected from the group comprising user voice input history, user language, user characteristics, user location, user preferences, and metadata tags associated with the user. ### Claim 20 (depends on 11) The system of claim 11, wherein the prosodic characteristics are further based on an interpolation operation affecting transitions in the synthesized speech response.