# US 12,217,755 B2 · Voice conversion apparatus, voice conversion learning apparatus, image generation apparatus, image generation learning apparatus, voice conversion method, voice conversion learning method, image generation method, image generation learning method, and computer program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,755 B2 - Document ID: US12217755B2 - Status: Granted - Kind: B2 - Application number: 17/640,221 - Filed: 2020-09-04 - Published: 2022-10-20 - Granted: 2025-02-04 - Earliest claimed: 2019-09-06 - Assignees: Nippon Telegraph and Telephone Corporation - Applicants: Nippon Telegraph and Telephone Corporation - Inventors: Hirokazu Kameoka; Ko Tanaka; Yasunori Oishi; Takuhiro Kaneko; Aaron Valero Puche - Web page: https://patentgrep.com/patent/US12217755B2 ## CPC - G06V 40/168: {Feature extraction; Face representation} - G10L 15/25: using position of the lips, movement of the lips or face analysis - G06T 7/00: Image analysis - G06V 40/176: {Dynamic expression} - G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit - G10L 15/063: {Training} - G10L 21/007: characterised by the process used - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A voice conversion device is provided with a linguistic information extraction unit that extracts linguistic information corresponding to utterance content from a conversion source voice signal, an appearance feature extraction unit that extracts appearance features expressing features related to the look of a person's face from a captured image of the person, and a converted voice generation unit that generates a converted voice on a basis of the linguistic information and the appearance features. ## Claims ### Claim 1 A voice device comprising one or more processors configured to perform operations comprising: receiving a voice signal; extracting linguistic information corresponding to utterance content from the voice signal; receiving a captured image of a person; extracting appearance features expressing features related to the look of the person's face from the captured image; determining a target timbre conforming to the appearance features of the person in the captured image; and generating a converted voice based on the linguistic information and the appearance features, wherein the converted voice is in the target timbre that conforms to the appearance features of the person in the captured image. ### Claim 2 (depends on 1) The voice device according to claim 1, wherein the operations comprise: accepting linguistic information corresponding to the utterance content extracted from the voice signal and appearance features expressing features related to the look of the person's face extracted from the captured image of the person as input, and train parameters of a linguistic information extraction unit configured to extract the linguistic information, an appearance feature extraction unit configured to extract the appearance features, and a converted voice generation unit configured to generate the converted voice on a basis of the linguistic information and the appearance features. ### Claim 3 (depends on 2) The voice device according to claim 2, wherein the operations comprise: performing learning such that the converted voice obtained when the linguistic information and the appearance features are input is as close as possible to the voice signal from which the linguistic information is extracted. ### Claim 4 (depends on 1) A non-transitory computer readable medium storing one or more instructions causing a computer to function as the voice device according to claim 1. ### Claim 5 An image device comprising: a timbre feature extraction unit, including one or more processors, configured to extract timbre features expressing features related to vocal timbre from a voice signal; and an image generation unit, including one or more processors, configured to generate a face image on a basis of the timbre features and appearance features expressing features related to the look of a person's face obtained from a captured image of the person, wherein the generated face image conforms to the timbre features of the voice signal. ### Claim 6 (depends on 5) The image device according to claim 5, comprising: a learning unit, including one or more processors, configured to accept timbre features expressing features related to vocal timbre extracted from the voice signal, appearance features expressing features related to the look of a person's face obtained from the captured image of the person, and a captured image of a person as input, and train parameters of the timbre feature extraction unit configured to extract the timbre features. ### Claim 7 (depends on 6) The image device according to claim 6, wherein the learning unit is configured to perform learning on a basis of the appearance features and the captured image such that the face image generated when the appearance features from any given captured image are input is as close as possible to the captured image from which the appearance features are extracted. ### Claim 8 (depends on 6) The image device according to claim 6, wherein the learning unit is configured to perform learning on a basis of the timbre features and the captured image such that the timbre features are as close as possible to appearance features obtained from the captured image used to generate a face image. ### Claim 9 A method comprising: receiving a voice signal; extracting linguistic information corresponding to utterance content from the voice signal; receiving a captured image of a person; extracting appearance features expressing features related to the look of the person's face from the captured image of the person; determining a target timbre conforming to the appearance features of the person in the captured image; and generating a converted voice based on the linguistic information and the appearance features, wherein the converted voice is in the target timbre that conforms to the appearance features of the person in the captured image. ### Claim 10 (depends on 9) The method according to claim 9, comprising: performing learning that accepts the linguistic information corresponding to utterance content extracted from the voice signal and appearance features expressing features related to the look of a person's face extracted from the captured image of the person as input, and training parameters of a linguistic information extraction unit that extracts the linguistic information, an appearance feature extraction unit that extracts the appearance features, and a converted voice generation unit that generates a converted voice on a basis of the linguistic information and the appearance features. ### Claim 11 (depends on 9) The method according to claim 9, comprising: extracting timbre features expressing features related to vocal timbre from a voice signal; and generating a face image on a basis of the timbre features and appearance features expressing features related to the look of the person's face obtained from the captured image of the person. ### Claim 12 (depends on 9) The method according to claim 9, comprising: performing learning that accepts timbre features expressing features related to vocal timbre extracted from a voice signal, appearance features expressing features related to the look of a person's face obtained from the captured image of the person, and the captured image of a person as input, and training parameters of an image generation unit that generates a face image on a basis of the timbre features and the appearance features and a timbre feature extraction unit that extracts the timbre features. ### Claim 13 (depends on 10) The method according to claim 10, comprising: performing learning such that the converted voice obtained when the linguistic information and the appearance features are input is as close as possible to the voice signal from which the linguistic information is extracted.