# US 10,373,379 B2 · Deformable-surface tracking based augmented reality image generation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,373,379 B2 - Document ID: US10373379B2 - Status: Granted - Kind: B2 - Application number: 14/831,657 - Filed: 2015-08-20 - Published: 2017-02-23 - Granted: 2019-08-06 - Earliest claimed: 2015-08-20 - Assignees: Disney Enterprises, Inc. - Applicants: Disney Enterprises, Inc. - Inventors: Robert Sumner; Dat Tien Ngo; Maurizio Nitti; Alessia Marra; Stephane Magnenat; Mattia Ryffel - Web page: https://patentgrep.com/patent/US10373379B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215) - G06T 7/11: Region-based segmentation - G06T 7/149: involving deformable models, e.g. active contour models - G06V 20/647: {by matching two-dimensional images to three-dimensional objects} - G06V 10/464: {using a plurality of salient features, e.g. bag-of-words [BoW] representations} - G06V 20/20: in augmented reality scenes - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract There are provided systems and methods for performing deformable-surface tracking based augmented reality image generation. In one implementation, such a system includes a hardware processor and a system memory storing an augmented reality three-dimensional image generator. The hardware processor is configured to execute the augmented reality three-dimensional image generator to receive image data corresponding to a two-dimensional surface, and to identify an image template corresponding to the two-dimensional surface based on the image data. In addition, the hardware processor is configured to execute the augmented reality three-dimensional image generator to determine a surface deformation of the two-dimensional surface. The hardware processor is further configured to execute the augmented reality three-dimensional image generator to generate an augmented reality three-dimensional image including at least one feature of the two-dimensional surface, based on the image template and the surface deformation of the two-dimensional surface. ## Claims ### Claim 1 A system comprising: a hardware processor; a system memory having an augmented reality three-dimensional image generator stored therein; wherein the hardware processor is configured to execute the augmented reality three-dimensional image generator to: receive image data from a video frame, the image data corresponding to a two-dimensional surface; remove colors from the image data by transforming the image data from a red-green-blue (RGB) color space to a hue-saturation-lightness (HSV) color space to provide a gray scale image data; detect a plurality of feature points in the gray scale image data; extract a feature point descriptor for each detected one of the plurality of feature points; identify an image template corresponding to the two-dimensional surface based on highest votes from the extracted feature point descriptors; determine a surface deformation of the two-dimensional surface, wherein determining the surface deformation includes representing the two-dimensional surface by a two-dimensional mesh, regularizing the two-dimensional mesh, and performing an outlier rejection based on the regularized two-dimensional mesh; generate an augmented reality three-dimensional image including at least one feature of the two-dimensional surface based on the image template and the surface deformation of the two-dimensional surface; and predict, using a linear motion model, a three-dimensional pose of the two-dimensional surface in a next video frame, and create an occlusion mask for a surface projection in image data of the next video frame. ### Claim 2 (depends on 1) The system of claim 1, wherein the system is part of a handheld mobile communication device. ### Claim 3 (depends on 1) The system of claim 1, wherein the system is further configured to display the augmented reality three-dimensional image to a user of the system. ### Claim 4 (depends on 1) The system of claim 1, wherein the system is further configured to display the augmented reality three-dimensional image to a user of the system as part of a media content in the form of one of entertainment content, game content, and educational content. ### Claim 5 (depends on 1) The system of claim 1, wherein the augmented reality three-dimensional image is an interactive image. ### Claim 6 (depends on 1) The system of claim 1, wherein the image data includes color data corresponding to at least one color appearing on the two-dimensional surface, the hardware processor being further configured to execute the augmented reality three-dimensional image generator to substantially reproduce the at least one color in the augmented reality three-dimensional image. ### Claim 7 A method for use by a system including a hardware processor and an augmented reality three-dimensional image generator stored in a system memory, the method comprising: receiving, using the hardware processor, image data corresponding to a two-dimensional surface; removing, using the hardware processor, colors from the image data by transforming the image data from a red-green-blue (RGB) color space to a hue-saturation-lightness (HSV) color space to provide a gray scale image data; detecting, using the hardware processor, a plurality of feature points in the gray scale image data; extracting, using the hardware processor, a feature point descriptor for each detected one of the plurality of feature points; identifying, using the hardware processor, an image template corresponding to the two-dimensional surface based on highest votes from the extracted feature point descriptors; determining, using the hardware processor, a surface deformation of the two dimensional surface, wherein the determining includes representing the two-dimensional surface by a two-dimensional mesh, regularizing the two-dimensional mesh, and performing an outlier rejection based on the regularized two-dimensional mesh; generating, using the hardware processor, an augmented reality three-dimensional image including at least one feature of the two-dimensional surface based on the image template and the surface deformation of the two-dimensional surface; and predicting, using a linear motion model, a three-dimensional pose of the two-dimensional surface in a next video frame, and create an occlusion mask for a surface projection in image data of the next video frame. ### Claim 8 (depends on 7) The method of claim 7, wherein the system is part of a handheld mobile communication device. ### Claim 9 (depends on 7) The method of claim 7, further comprising displaying, using the hardware processor, the augmented reality three-dimensional image to a user of the system. ### Claim 10 (depends on 7) The method of claim 7, further comprising displaying, using the hardware processor, the augmented reality three-dimensional image to a user of the system as part of a media content in the form of one of entertainment content, game content, and educational content. ### Claim 11 (depends on 7) The method of claim 7, wherein the augmented reality three-dimensional image is an interactive image. ### Claim 12 (depends on 7) The method of claim 7, wherein the image data includes color data corresponding to at least one color appearing on the two-dimensional surface, the method further comprising substantially reproducing, using the hardware processor, the at least one color in the augmented reality three-dimensional image.