# US 10,373,244 B2 · System and method for virtual clothes fitting based on video augmented reality in mobile phone > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,373,244 B2 - Document ID: US10373244B2 - Status: Granted - Kind: B2 - Application number: 14/799,907 - Filed: 2015-07-15 - Published: 2017-01-19 - Granted: 2019-08-06 - Earliest claimed: 2015-07-15 - Assignees: Futurewei Technologies, Inc. - Applicants: Futurewei Technologies, Inc. - Inventors: Jingsong Xu; Ying Cui; Qiang Wu; Jian Zhang; Chen-Xiong Zhang; Haibo Liu; Kui Fang - Web page: https://patentgrep.com/patent/US10373244B2 ## CPC - G06Q 30/0643: {graphically representing goods, e.g. 3D product representation} - G06T 11/00: Two-dimensional [2D] image generation - G06T 11/60: Creating or editing images; Combining images with text - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06T 2210/16: Cloth - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A system and method for virtual clothes fitting based on video augmented reality in a mobile wireless device is disclosed. In an embodiment, a method for virtual cloth-fitting with video augmentation in a mobile wireless device includes receiving a video stream of an upper body of a user; detecting a face in the video stream; detecting a shoulder contour; determining keypoints of the shoulder and neck portions of the video stream of the upper body of the user according to the detected face and the detected shoulder contour; mapping an image of clothes to the video stream of the upper body of the user according to the keypoints; and displaying an augmented video stream of the upper body of the user with the image of the clothes overlaid over a portion of the video stream of the upper body of the user. ## Claims ### Claim 1 A method for virtual cloth-fitting, comprising: prompting, by an indicator on a display of a mobile device, a user of the mobile device to correctly place a video stream of an upper body portion of the user using guidelines presented on the display, the video stream being captured by a camera of the mobile device; detecting, by the mobile device, a location of a face of the user in the video stream from the correct placement of the video stream of the upper body portion of the user using the guidelines; predicting, by the mobile device, a position of a neck of the user using the detected location of the face of the user; detecting, by the mobile device, a location of a shoulder contour of the user in the video stream from the predicted position of the neck; estimating, by the mobile device, keypoints of shoulder and neck portions of the user in the video stream using the detected location of the shoulder contour and the predicted position of the neck, respectively; mapping, by the mobile device, an image of clothes to the video stream of the upper body portion of the user using the estimated keypoints; and displaying, by the display, an augmented video stream of the upper body portion of the user with the mapped image of clothes overlaid over a portion of the video stream of the upper body portion of the user. ### Claim 2 (depends on 1) The method of claim 1, further comprising: tracking movement of the user; remapping, by the mobile device, the image of clothes in accordance with the movement; and displaying, by the display, a remapped augmented video stream of the upper body portion of the user with the remapped image of clothes overlaid over a portion of the remapped augmented video stream of the upper body portion of the user, wherein the remapped image of clothes corresponds to the movement of the user. ### Claim 3 (depends on 2) The method of claim 2, wherein remapping the image of clothes comprises transforming relevant positions on a plane of the image of clothes with corresponding positions on a plane of an image of the upper body portion of the user. ### Claim 4 (depends on 2) The method of claim 2, wherein the remapping the image of clothes comprises a Delaunay triangulation. ### Claim 5 (depends on 1) The method of claim 1, wherein the shoulder contour is automatically determined. ### Claim 6 (depends on 1) The method of claim 1, wherein the face and the shoulder contour are determined according to user input. ### Claim 7 (depends on 6) The method of claim 6, further comprising attempting to detect the face, presenting a candidate face region, and receiving a user input confirming that the candidate face region corresponds to the face of the user. ### Claim 8 (depends on 6) The method of claim 6, further comprising receiving user input moving a shoulder contour outline. ### Claim 9 A mobile wireless device, comprising: a camera; a display; a non-transitory memory storage comprising instructions; and a processor in communication with the camera, the display, and the non-transitory memory storage, wherein the processor executes the instructions to: prompt, by an indicator on the display, a user of the mobile wireless device to correctly place a video stream of an upper body portion of the user using guidelines presented on the display, the video stream being captured by the camera; detect a location of a face of the user in the video stream from the correct placement of the video stream of the upper body portion of the user using the guidelines; predict a position of a neck of the user using the detected location of the face of the user; detect a location of a shoulder contour of the user in the video stream from the predicted position of the neck; estimate keypoints of shoulder and neck portions of the user in the video stream using the detected location of the shoulder contour and the predicted position of the neck, respectively; map an image of clothes to the video stream of the upper body portion of the user using the estimated keypoints; and display, by the display, an augmented video stream of the upper body portion of the user with the mapped image of clothes overlaid over a portion of the video stream of the upper body portion of the user. ### Claim 10 (depends on 9) The mobile wireless device of claim 9, wherein the processor executes the instructions to: track movement of the user; remap the image of clothes in accordance with the movement of the user; and display, by the display, a remapped augmented video stream of the upper body portion of the user with the remapped image of clothes overlaid over a portion of the video stream of the upper body portion of the user, wherein the remapped image of clothes corresponds to the movement of the user. ### Claim 11 (depends on 10) The mobile wireless device of claim 10, wherein the instructions to remap the image of clothes comprises instructions to transform relevant positions on a plane of the image of clothes with corresponding positions on a plane of an image of the upper body portion of the user. ### Claim 12 (depends on 10) The mobile wireless device of claim 10, wherein the instructions to remap the image of clothes comprises instructions to execute a Delaunay triangulation. ### Claim 13 (depends on 9) The mobile wireless device of claim 9, wherein the shoulder contour is automatically determined. ### Claim 14 (depends on 9) The mobile wireless device of claim 9, wherein the face and the shoulder contour are determined according to user input. ### Claim 15 (depends on 14) The mobile wireless device of claim 14, wherein the processor executes the instructions to attempt to detect the face of the user, present a candidate face region, and receive a user input confirming that the candidate face region corresponds to the face of the user. ### Claim 16 A mobile wireless device, comprising: a camera; a live video stream acquisition unit comprising a processor, the live video stream acquisition unit configured to prompt a user of the mobile wireless device to correctly place a live video stream of an upper body portion of the user using guidelines presented to the user, the live video stream being captured by the camera; a system initialization unit comprising a first processor, the system initialization unit configured to detect a location of a face of the user in the live video stream from the correct placement of the video stream of the upper body portion of the user using the guidelines, predict a position of a neck of the user using the detected location of the face of the user, detect a location of a shoulder contour of the user in the live video stream from the predicted position of the neck, and estimate keypoints of the shoulder and neck portions of the user in the live video stream using the detected location of the shoulder contour and the predicted position of the neck, respectively; a virtual-cloth fitting unit comprising a second processor, the virtual-cloth fitting unit configured to map an image of clothes to the live video stream of the upper body portion of the user using the estimated keypoints; and a video display configured to display an augmented video stream of the upper body portion of the user with the mapped image of clothes overlaid over a portion of the live video stream of the upper body portion of the user. ### Claim 17 (depends on 16) The mobile wireless device of claim 16, wherein the virtual-cloth fitting unit is further configured to track movement of the user and to remap the image of clothes in accordance with the movement. ### Claim 18 (depends on 17) The mobile wireless device of claim 17, wherein the virtual-cloth fitting unit is further configured to transform relevant positions on a plane of the image of clothes with corresponding positions on a plane of an image of the upper body portion of the user. ### Claim 19 (depends on 17) The mobile wireless device of claim 17, wherein the virtual-cloth fitting unit is further configured to execute a Delaunay triangulation. ### Claim 20 (depends on 16) The mobile wireless device of claim 16, wherein the video display is further configured to present a candidate face region and to receive a user input confirming that the candidate face region corresponds to the face of the user.