Deformable-surface tracking based augmented reality image generation

A method for use by a system including a hardware processor and an augmented reality three-dimensional image generator stored in a system memory, the method comprising:

• receiving, using the hardware processor, image data corresponding to a two-dimensional surface;
• removing, using the hardware processor, colors from the image data by transforming the image data from a red-green-blue (RGB) color space to a hue-saturation-lightness (HSV) color space to provide a gray scale image data;
• detecting, using the hardware processor, a plurality of feature points in the gray scale image data;
• extracting, using the hardware processor, a feature point descriptor for each detected one of the plurality of feature points;
• identifying, using the hardware processor, an image template corresponding to the two-dimensional surface based on highest votes from the extracted feature point descriptors;
• determining, using the hardware processor, a surface deformation of the two dimensional surface, wherein the determining includes representing the two-dimensional surface by a two-dimensional mesh, regularizing the two-dimensional mesh, and performing an outlier rejection based on the regularized two-dimensional mesh;
• generating, using the hardware processor, an augmented reality three-dimensional image including at least one feature of the two-dimensional surface based on the image template and the surface deformation of the two-dimensional surface;
• and
• predicting, using a linear motion model, a three-dimensional pose of the two-dimensional surface in a next video frame, and create an occlusion mask for a surface projection in image data of the next video frame.