System and method for virtual clothes fitting based on video augmented reality in mobile phone

A mobile wireless device, comprising:

• a camera;
• a display;
• a non-transitory memory storage comprising instructions;
• and
• a processor in communication with the camera, the display, and the non-transitory memory storage, wherein the processor executes the instructions to:
• prompt, by an indicator on the display, a user of the mobile wireless device to correctly place a video stream of an upper body portion of the user using guidelines presented on the display, the video stream being captured by the camera;
• detect a location of a face of the user in the video stream from the correct placement of the video stream of the upper body portion of the user using the guidelines;
• predict a position of a neck of the user using the detected location of the face of the user;
• detect a location of a shoulder contour of the user in the video stream from the predicted position of the neck;
• estimate keypoints of shoulder and neck portions of the user in the video stream using the detected location of the shoulder contour and the predicted position of the neck, respectively;
• map an image of clothes to the video stream of the upper body portion of the user using the estimated keypoints;
• and
• display, by the display, an augmented video stream of the upper body portion of the user with the mapped image of clothes overlaid over a portion of the video stream of the upper body portion of the user.

A mobile wireless device, comprising:

• a camera;
• a live video stream acquisition unit comprising a processor, the live video stream acquisition unit configured to prompt a user of the mobile wireless device to correctly place a live video stream of an upper body portion of the user using guidelines presented to the user, the live video stream being captured by the camera;
• a system initialization unit comprising a first processor, the system initialization unit configured to detect a location of a face of the user in the live video stream from the correct placement of the video stream of the upper body portion of the user using the guidelines, predict a position of a neck of the user using the detected location of the face of the user, detect a location of a shoulder contour of the user in the live video stream from the predicted position of the neck, and estimate keypoints of the shoulder and neck portions of the user in the live video stream using the detected location of the shoulder contour and the predicted position of the neck, respectively;
• a virtual-cloth fitting unit comprising a second processor, the virtual-cloth fitting unit configured to map an image of clothes to the live video stream of the upper body portion of the user using the estimated keypoints;
• and
• a video display configured to display an augmented video stream of the upper body portion of the user with the mapped image of clothes overlaid over a portion of the live video stream of the upper body portion of the user.