Image processing apparatus, image processing method, program, and data configuration

An image processing method for generating animation data to generate a face image of a different expression on the basis of a face image, the image processing method comprising the steps of:

• detecting eye boundaries between eyeballs and eyelids where eyeballs are exposed and a mouth boundary between lips and the inside of a mouth from a first face image, which is a face image of a first expression;
• detecting eye boundaries and a mouth boundary from a second face image, which is a face image of a second expression different from the first expression;
• generating a first skin model, which is a face model generated by removing only the inside of the eye boundaries and the inside of the mouth boundary from the first face image;
• generating a second skin model, which is a face model generated by removing only the inside of the eye boundaries and the inside of the mouth boundary from the second face image;
• generating an inside-of-mouth model, which is a model of the inside of the mouth boundary;
• generating an eyeball model, which is a model of eyeballs;
• generating a first deformation parameter used to deform the first skin model to the second skin model;
• generating a second deformation parameter used to deform the second skin model to the first skin model;
• and
• outputting the first and second skin models, the first and second deformation parameters, the inside-of-mouth model, and the eyeball model as the animation data.

A program, tangibly embodied in a non-transitory computer-readable storage medium, allowing a computer to execute image processing of generating animation data to generate a face image of a different expression on the basis of a face image, the image processing comprising the steps of:

• detecting eye boundaries between eyeballs and eyelids where eyeballs are exposed and a mouth boundary between lips and the inside of a mouth from a first face image, which is a face image of a first expression;
• detecting eye boundaries and a mouth boundary from a second face image, which is a face image of a second expression different from the first expression;
• generating a first skin model, which is a face model generated by removing only the inside of the eye boundaries and the inside of the mouth boundary from the first face image;
• generating a second skin model, which is a face model generated by removing only the inside of the eye boundaries and the inside of the mouth boundary from the second face image;
• generating an inside-of-mouth model, which is a model of the inside of the mouth boundary;
• generating an eyeball model, which is a model of eyeballs;
• generating a first deformation parameter used to deform the first skin model to the second skin model;
• generating a second deformation parameter used to deform the second skin model to the first skin model;
• and
• outputting the first and second skin models, the first and second deformation parameters, the inside-of-mouth model, and the eyeball model as the animation data.

An image processing apparatus to generate a face image, the image processing apparatus comprising:

• data obtaining means for obtaining animation data including
• a first skin model, which is a face model generated by removing only the inside of eye boundaries between eyeballs and eyelids where eyeballs are exposed and the inside of a mouth boundary between lips and the inside of a mouth from a first face image, which is a face image of a first expression;
• a second skin model, which is a face model generated by removing only the inside of eye boundaries and the inside of a mouth boundary from a second face image, which is a face image of a second expression different from the first expression;
• an inside-of-mouth model, which is a model of the inside of the mouth boundary;
• an eyeball model, which is a model of eyeballs;
• a first deformation parameter used to deform the first skin model to the second skin model;
• and
• a second deformation parameter used to deform the second skin model to the first skin model;
• deformation level setting means for setting a deformation level indicating a level of deformation from one of the first and second expressions to the other;
• first skin model deforming means for deforming the first skin model to a first deformed skin model in accordance with the deformation level by using the first deformation parameter;
• second skin model deforming means for deforming the second skin model to a second deformed skin model in accordance with the deformation level by using the second deformation parameter;
• blending parameter setting means for setting a blending parameter indicating a level of blending the first and second deformed skin models in accordance with the deformation level;
• skin model blending means for blending the first and second deformed skin models in accordance with the blending parameter so as to generate a blended skin model;
• and
• face image generating means for generating a face image of an expression according to the deformation level by blending the blended skin model, the inside-of-mouth model, and the eyeball model.

An image processing method for generating a face image, the image processing method comprising the steps of:

• obtaining animation data including
• a first skin model, which is a face model generated by removing only the inside of eye boundaries between eyeballs and eyelids where eyeballs are exposed and the inside of a mouth boundary between lips and the inside of a mouth from a first face image, which is a face image of a first expression;
• a second skin model, which is a face model generated by removing only the inside of eye boundaries and the inside of a mouth boundary from a second face image, which is a face image of a second expression different from the first expression;
• an inside-of-mouth model, which is a model of the inside of the mouth boundary;
• an eyeball model, which is a model of eyeballs;
• a first deformation parameter used to deform the first skin model to the second skin model;
• and
• a second deformation parameter used to deform the second skin model to the first skin model;
• setting a deformation level indicating a level of deformation from one of the first and second expressions to the other;
• deforming the first skin model to a first deformed skin model in accordance with the deformation level by using the first deformation parameter;
• deforming the second skin model to a second deformed skin model in accordance with the deformation level by using the second deformation parameter;
• setting a blending parameter indicating a level of blending the first and second deformed skin models in accordance with the deformation level;
• blending the first and second deformed skin models in accordance with the blending parameter so as to generate a blended skin model;
• and
• generating a face image of an expression according to the deformation level by blending the blended skin model, the inside-of-mouth model, and the eyeball model.

A program, tangibly embodied in a non-transitory computer-readable storage medium, allowing a computer to execute image processing to generate a face image, the image processing comprising the steps of:

• obtaining animation data including
• a first skin model, which is a face model generated by removing only the inside of eye boundaries between eyeballs and eyelids where eyeballs are exposed and the inside of a mouth boundary between lips and the inside of a mouth from a first face image, which is a face image of a first expression;
• a second skin model, which is a face model generated by removing only the inside of eye boundaries and the inside of a mouth boundary from a second face image, which is a face image of a second expression different from the first expression;
• an inside-of-mouth model, which is a model of the inside of the mouth boundary;
• an eyeball model, which is a model of eyeballs;
• a first deformation parameter used to deform the first skin model to the second skin model;
• and
• a second deformation parameter used to deform the second skin model to the first skin model;
• setting a deformation level indicating a level of deformation from one of the first and second expressions to the other;
• deforming the first skin model to a first deformed skin model in accordance with the deformation level by using the first deformation parameter;
• deforming the second skin model to a second deformed skin model in accordance with the deformation level by using the second deformation parameter;
• setting a blending parameter indicating a level of blending the first and second deformed skin models in accordance with the deformation level;
• blending the first and second deformed skin models in accordance with the blending parameter so as to generate a blended skin model;
• and
• generating a face image of an expression according to the deformation level by blending the blended skin model, the inside-of-mouth model, and the eyeball model.

A tangible, non-transitory computer-readable storage medium product having a data configuration of animation data to generate a face image stored therein, the data configuration comprising:

• a first skin model, which is a face model generated by removing only the inside of eye boundaries between eyeballs and eyelids where eyeballs are exposed and the inside of a mouth boundary between lips and the inside of a mouth from a first face image, which is a face image of a first expression;
• a second skin model, which is a face model generated by removing only the inside of eye boundaries and the inside of a mouth boundary from a second face image, which is a face image of a second expression different from the first expression;
• an inside-of-mouth model, which is a model of the inside of the mouth boundary;
• an eyeball model, which is a model of eyeballs;
• a first deformation parameter used to deform the first skin model to the second skin model;
• and
• a second deformation parameter used to deform the second skin model to the first skin model.

An image processing apparatus generating animation data to generate a face image of a different expression on the basis of a face image, the image processing apparatus comprising:

• a first eyes-and-mouth detecting unit configured to detect eye boundaries between eyeballs and eyelids where eyeballs are exposed and a mouth boundary between lips and the inside of a mouth from a first face image, which is a face image of a first expression;
• a second eyes-and-mouth detecting unit configured to detect eye boundaries and a mouth boundary from a second face image, which is a face image of a second expression different from the first expression;
• a first skin model generating unit configured to generate a first skin model, which is a face model generated by removing only the inside of the eye boundaries and the inside of the mouth boundary from the first face image;
• a second skin model generating unit configured to generate a second skin model, which is a face model generated by removing only the inside of the eye boundaries and the inside of the mouth boundary from the second face image;
• an inside-of-mouth model generating unit configured to generate an inside-of-mouth model, which is a model of the inside of the mouth boundary;
• an eyeball model generating unit configured to generate an eyeball model, which is a model of eyeballs;
• a first deformation parameter generating unit configured to generate a first deformation parameter used to deform the first skin model to the second skin model;
• a second deformation parameter generating unit configured to generate a second deformation parameter used to deform the second skin model to the first skin model;
• and
• an output unit configured to output the first and second skin models, the first and second deformation parameters, the inside-of-mouth model, and the eyeball model as the animation data.

An image processing apparatus to generate a face image, the image processing apparatus comprising:

• a data obtaining unit configured to obtain animation data including
• a first skin model, which is a face model generated by removing only the inside of eye boundaries between eyeballs and eyelids where eyeballs are exposed and the inside of a mouth boundary between lips and the inside of a mouth from a first face image, which is a face image of a first expression;
• a second skin model, which is a face model generated by removing only the inside of eye boundaries and the inside of a mouth boundary from a second face image, which is a face image of a second expression different from the first expression;
• an inside-of-mouth model, which is a model of the inside of the mouth boundary;
• an eyeball model, which is a model of eyeballs;
• a first deformation parameter used to deform the first skin model to the second skin model;
• and
• a second deformation parameter used to deform the second skin model to the first skin model;
• a deformation level setting unit configured to set a deformation level indicating a level of deformation from one of the first and second expressions to the other;
• a first skin model deforming unit configured to deform the first skin model to a first deformed skin model in accordance with the deformation level by using the first deformation parameter;
• a second skin model deforming unit configured to deform the second skin model to a second deformed skin model in accordance with the deformation level by using the second deformation parameter;
• a blending parameter setting unit configured to set a blending parameter indicating a level of blending the first and second deformed skin models in accordance with the deformation level;
• a skin model blending unit configured to blend the first and second deformed skin models in accordance with the blending parameter so as to generate a blended skin model;
• and
• a face image generating unit configured to generate a face image of an expression according to the deformation level by blending the blended skin model, the inside-of-mouth model, and the eyeball model.