# US 12,002,138 B2 · Speech-driven animation method and apparatus based on artificial intelligence > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,138 B2 - Document ID: US12002138B2 - Status: Granted - Kind: B2 - Application number: 17/497,622 - Filed: 2021-10-08 - Published: 2022-02-10 - Granted: 2024-06-04 - Earliest claimed: 2019-08-29 - Assignees: Tencent Technology (Shenzhen) Company Limited - Applicants: Tencent Technology (Shenzhen) Company Limited - Inventors: Shiyin Kang; Deyi Tuo; Kuongchi Lei; Tianxiao Fu; Huirong Huang; Dan Su - Web page: https://patentgrep.com/patent/US12002138B2 ## CPC - G06T 13/205: {driven by audio data} - G10L 21/10: Transforming into visible information - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0499: Feedforward networks - G06N 3/09: Supervised learning - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit - G10L 15/04: Segmentation; Word boundary detection - G10L 15/187: Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams - G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications - G10L 25/30: using neural networks - G10L 25/63: for estimating an emotional state - G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Embodiments of this application disclose a speech-driven animation method and apparatus based on artificial intelligence (AI). The method includes obtaining a first speech, the first speech comprising a plurality of speech frames; determining linguistics information corresponding to a speech frame in the first speech, the linguistics information being used for identifying a distribution possibility that the speech frame in the first speech pertains to phonemes; determining an expression parameter corresponding to the speech frame in the first speech according to the linguistics information; and enabling, according to the expression parameter, an animation character to make an expression corresponding to the first speech. ## Claims ### Claim 1 A speech-driven animation method, performed by an audio and video processing device, the method comprising: obtaining a first speech with an acoustic feature, the first speech comprising a plurality of speech frames; determining linguistics information corresponding to a speech frame in the first speech by applying a neural network mapping model to extract the acoustic feature, the linguistics information being used for identifying a distribution possibility that the speech frame in the first speech pertains to phonemes; determining an expression parameter corresponding to the speech frame in the first speech according to the linguistics information, wherein the expression parameters do not reflect pronunciation habits of different speakers; and enabling, according to the expression parameter, an animation character to make an expression corresponding to the first speech. ### Claim 2 (depends on 1) The method according to claim 1, wherein a target speech frame is a speech frame in the first speech, and the determining an expression parameter corresponding to the speech frame in the first speech according to the linguistics information comprises: determining a speech frame set in which the target speech frame is located, the speech frame set comprising the target speech frame and speech frames preceding and succeeding the target speech frame; and determining an expression parameter corresponding to the target speech frame according to linguistics information corresponding to each speech frame in the speech frame set. ### Claim 3 (depends on 2) The method according to claim 2, wherein a quantity of speech frames in the speech frame set is determined according to a neural network mapping model, or a quantity of speech frames in the speech frame set is determined according to a speech segmentation result of the first speech. ### Claim 4 (depends on 2) The method according to claim 2, wherein the speech frames preceding and succeeding the target speech frame are consecutive preceding and succeeding speech frames, or the speech frames preceding and succeeding the target speech frame are inconsecutive speech frames. ### Claim 5 (depends on 2) The method according to claim 2, wherein the determining an expression parameter corresponding to the target speech frame according to linguistics information corresponding to each speech frame in the speech frame set comprises: determining an undetermined expression parameter corresponding to each speech frame in the speech frame set according to the linguistics information corresponding to each speech frame in the speech frame set; and calculating the expression parameter corresponding to the target speech frame according to undetermined expression parameters of the target speech frame that are respectively determined in different speech frame sets. ### Claim 6 (depends on 1) The method according to claim 1, wherein the linguistics information comprises any one or a combination of two or more of a phonetic posterior gram (PPG), a bottleneck feature, and an embedding feature. ### Claim 7 (depends on 1) The method according to claim 1, wherein the determining an expression parameter corresponding to the speech frame in the first speech according to the linguistics information comprises: determining the expression parameter corresponding to the speech frame in the first speech according to the linguistics information by using the neural network mapping model, the neural network mapping model comprising a deep neural network (DNN) model, a long short-term memory (LSTM) model, or a bidirectional long short-term memory (BLSTM) model. ### Claim 8 (depends on 1) The method according to claim 1, wherein the determining an expression parameter corresponding to the speech frame in the first speech according to the linguistics information comprises: determining the expression parameter corresponding to the speech frame in the first speech according to the linguistics information and a sentiment vector corresponding to the first speech. ### Claim 9 (depends on 1) The method according to claim 1, wherein the determining linguistics information corresponding to a speech frame in the first speech comprises: determining an acoustic feature corresponding to the speech frame in the first speech; and determining linguistics information corresponding to the acoustic feature by using an automatic speech recognition (ASR) model. ### Claim 10 (depends on 9) The method according to claim 9, wherein the ASR model is obtained through training according to training samples that comprise correspondences between speech segments and phonemes. ### Claim 11 A speech-driven animation apparatus, deployed on an audio and video processing device, the apparatus comprising a processor and a memory, the memory being configured to store program code and transmit the program code to the processor; and when executing the program code, the processor being configured to: obtain a first speech with an acoustic feature, the first speech comprising a plurality of speech frames; determine linguistics information corresponding to a speech frame in the first speech by applying a neural network mapping model to extract the acoustic feature, the linguistics information being used for identifying a distribution possibility that the speech frame in the first speech pertains to phonemes; determine an expression parameter corresponding to the speech frame in the first speech according to the linguistics information, wherein the expression parameters do not reflect pronunciation habits of different speakers; and enable according to the expression parameter, an animation character to make an expression corresponding to the first speech. ### Claim 12 (depends on 11) The apparatus according to claim 11, wherein a target speech frame is a speech frame in the first speech, and for the target speech frame, the processor is further configured to: determine a speech frame set in which the target speech frame is located, the speech frame set comprising the target speech frame and preceding and succeeding speech frames of the target speech frame; and determine an expression parameter corresponding to the target speech frame according to linguistics information corresponding to each speech frame in the speech frame set. ### Claim 13 (depends on 1) The method according to claim 1, wherein the expression includes a mouth shape, a facial action, or a head posture. ### Claim 14 A non-transitory computer-readable storage medium, configured to store program code, the program code, when being executed by a processor, causing the processor to perform: obtaining a first speech with an acoustic feature, the first speech comprising a plurality of speech frames; determining linguistics information corresponding to a speech frame in the first speech by applying a neural network mapping model to extract the acoustic feature, the linguistics information being used for identifying a distribution possibility that the speech frame in the first speech pertains to phonemes; determining an expression parameter corresponding to the speech frame in the first speech according to the linguistics information, wherein the expression parameters do not reflect pronunciation habits of different speakers; and enabling, according to the expression parameter, an animation character to make an expression corresponding to the first speech. ### Claim 15 (depends on 14) The computer-readable storage medium according to claim 14, wherein a target speech frame is a speech frame in the first speech, and the determining an expression parameter corresponding to the speech frame in the first speech according to the linguistics information comprises: determining a speech frame set in which the target speech frame is located, the speech frame set comprising the target speech frame and the speech frames preceding and succeeding the target speech frame; and determining an expression parameter corresponding to the target speech frame according to linguistics information corresponding to each speech frame in the speech frame set. ### Claim 16 (depends on 15) The computer-readable storage medium according to claim 15, wherein the speech frames preceding and succeeding the target speech frame are consecutive preceding and succeeding speech frames, or the speech frames preceding and succeeding the target speech frame are inconsecutive speech frames. ### Claim 17 (depends on 15) The computer-readable storage medium according to claim 15, wherein the determining an expression parameter corresponding to the target speech frame according to linguistics information corresponding to each speech frame in the speech frame set comprises: determining an undetermined expression parameter corresponding to each speech frame in the speech frame set according to the linguistics information corresponding to each speech frame in the speech frame set; and calculating the expression parameter corresponding to the target speech frame according to undetermined expression parameters of the target speech frame that are respectively determined in different speech frame sets. ### Claim 18 (depends on 14) The computer-readable storage medium according to claim 14, wherein the linguistics information comprises any one or a combination of two or more of a phonetic posterior gram (PPG), a bottleneck feature, and an embedding feature. ### Claim 19 (depends on 14) The computer-readable storage medium according to claim 14, wherein the determining an expression parameter corresponding to the speech frame in the first speech according to the linguistics information comprises: determining the expression parameter corresponding to the speech frame in the first speech according to the linguistics information by using the neural network mapping model, the neural network mapping model comprising a deep neural network (DNN) model, a long short-term memory (LSTM) model, or a bidirectional long short-term memory (BLSTM) model.