Method and apparatus for controlling mouth shape changes of three-dimensional virtual portrait

An apparatus for controlling mouth shape changes of a three-dimensional virtual portrait, comprising:

• at least one processor;
• and
• a memory storing instructions, the instructions when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising:
• acquiring a to-be-played speech;
• sliding a preset time window at a preset step length in the to-be-played speech to obtain at least one speech segment;
• generating, based on the at least one speech segment, a mouth shape control parameter sequence for the to-be-played speech;
• and
• controlling, in response to playing the to-be-played speech, a preset mouth shape of the three-dimensional virtual portrait to change based on the mouth shape control parameter sequence, wherein the generating, based on the at least one speech segment, the mouth shape control parameter sequence for the to-be-played speech comprises:
• generating, for a speech segment of the at least one speech segment, a phoneme information sequence of the speech segment;
• inputting the phoneme information sequence composed of a plurality of pieces of phoneme information into a pre-established mouth shape key point predicting model to obtain a mouth shape key point information sequence composed of a plurality of pieces of mouth shape key point information, wherein the pre-established mouth shape key point predicting model is used to characterize a corresponding relationship between the phoneme information sequence and the mouth shape key point information sequence, wherein the mouth shape key point information indicates position information of a preset number of face key points related to a mouth shape, wherein inputting the phoneme information sequence composed of the plurality of pieces of phoneme information into the pre-established mouth shape key point predicting model to obtain the mouth shape key point information sequence composed of the plurality of pieces of mouth shape key point information comprises:
• outputting, by the pre-established mouth shape key point predicting model, a first piece of mouth shape key point information by using a first piece of phoneme information as a first input, and outputting, by the pre-established mouth shape key point predicting model, a second piece of mouth shape key point information by using a second piece of phoneme information and the first piece of mouth shape key point information as a second input;
• and
• generating, based on the mouth shape key point information sequence, the mouth shape control parameter sequence.

A non-transitory computer readable medium, storing a computer program thereon, wherein the computer program, when executed by a processor, causes the processor to perform operations, the operations comprising:

• acquiring a to-be-played speech;
• sliding a preset time window at a preset step length in the to-be-played speech to obtain at least one speech segment;
• generating, based on the at least one speech segment, a mouth shape control parameter sequence for the to-be-played speech;
• and
• controlling, in response to playing the to-be-played speech, a preset mouth shape of a three-dimensional virtual portrait to change based on the mouth shape control parameter sequence, wherein the generating, based on the at least one speech segment, the mouth shape control parameter sequence for the to-be-played speech comprises:
• generating, for a speech segment of the at least one speech segment, a phoneme information sequence of the speech segment;
• inputting the phoneme information sequence composed of a plurality of pieces of phoneme information into a pre-established mouth shape key point predicting model to obtain a mouth shape key point information sequence composed of a plurality of pieces of mouth shape key point information, wherein the pre-established mouth shape key point predicting model is used to characterize a corresponding relationship between the phoneme information sequence and the mouth shape key point information sequence, wherein the mouth shape key point information indicates position information of a preset number of face key points related to a mouth shape, wherein inputting the phoneme information sequence composed of the plurality of pieces of phoneme information into the pre-established mouth shape key point predicting model to obtain the mouth shape key point information sequence composed of the plurality of pieces of mouth shape key point information comprises:
• outputting, by the pre-established mouth shape key point predicting model, a first piece of mouth shape key point information by using a first piece of phoneme information as a first input, and outputting, by the pre-established mouth shape key point predicting model, a second piece of mouth shape key point information by using a second piece of phoneme information and the first piece of mouth shape key point information as a second input;
• and
• generating, based on the mouth shape key point information sequence, the mouth shape control parameter sequence.