# US 12,223,672 B2 · Real-time garment exchange > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,672 B2 - Document ID: US12223672B2 - Status: Granted - Kind: B2 - Application number: 17/557,652 - Filed: 2021-12-21 - Published: 2023-06-22 - Granted: 2025-02-11 - Earliest claimed: 2021-12-21 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Avihay Assouline; Itamar Berger; Nir Malbin; Gal Sasson - Web page: https://patentgrep.com/patent/US12223672B2 ## CPC - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06T 5/77: Retouching; Inpainting; Scratch removal - G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215) - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 11/00: Two-dimensional [2D] image generation - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06V 10/54: relating to texture - G06V 10/56: relating to colour - G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals - G06T 2207/10016: Video; Image sequence - G06T 2207/20081: Training; Learning - G06T 2210/16: Cloth - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) ## Abstract Methods and systems are disclosed for performing operations for transferring garments in a video from one real-world object to another in real time. The operations comprise receiving a first video that includes a depiction of a first person wearing a first garment in a first pose and obtaining a second video that includes a depiction of a second person wearing a second garment in a second pose. The operations comprise modifying a pose of the second person to match the first pose of the first person depicted in the first video. The operations comprise generating a whole-body segmentation of the second garment which the second person is wearing in the second video and changing an appearance of the first person from wearing the first garment to wearing the second garment based on the whole-body segmentation of the second garment which the second person is wearing in the second video. ## Claims ### Claim 1 A method comprising: receiving, by one or more processors, a first video that includes a depiction of a first person wearing a first garment in a first pose; obtaining a second video that includes a depiction of a second person wearing a second garment in a second pose; modifying the second pose of the second person depicted in the second video to match the first pose of the first person depicted in the first video by generating a third video that includes a depiction of the second person having appearance data associated with the second person and motion data associated with the first person; after modifying the second pose of the second person to match the first pose of the first person, generating a whole-body segmentation of the second garment which the second person is wearing in the third video; and changing appearance data representing an appearance of the first person from wearing the first garment to wearing the second garment based on the whole-body segmentation of the second garment which the second person is wearing. ### Claim 2 (depends on 1) The method of claim 1, wherein changing the appearance data associated with the first person comprises: extracting the appearance data and the motion data associated with the first person from the first video; extracting the appearance data and motion data associated with the second person from the second video; modifying the motion data associated with the second person to match the motion data associated with the first person; applying, based on the whole-body segmentation, a mask to the appearance data associated with the second person after modifying the motion data associated with the second person to obtain pixels corresponding to the second garment; replacing a portion of the appearance data associated with the first person based on the pixels corresponding to the second garment in the mask applied to the appearance data associated with the second person; and based on replacing the portion of the appearance data associated with the first person, generating a third fourth video that includes a depiction of the first person wearing the second garment instead of the first garment. ### Claim 3 (depends on 2) The method of claim 2, further comprising: applying the mask to the appearance data associated with the first person to identify a set of target pixels; and copying the pixels corresponding to the second garment to the set of target pixels. ### Claim 4 (depends on 2) The method of claim 2, wherein the appearance data associated with the second person comprises a three-dimensional (3D) pose, a full body segmentation, a set of dense keypoints, texture, color, and a garment segmentation. ### Claim 5 (depends on 2) The method of claim 2, wherein extracting the appearance data and the motion data associated with the first person and the second person comprises applying a first machine learning technique to the first video and the second video, the first machine learning technique being trained to estimate appearance and motion data associated with an input video. ### Claim 6 (depends on 5) The method of claim 5, wherein the first machine learning technique is trained to generate a three-dimensional (3D) pose of a given real-world object depicted in the input video, a full body segmentation of the given real-world object, a set of dense keypoints of the given real-world object, texture, color, and a garment segmentation of the given real-world object simultaneously. ### Claim 7 (depends on 6) The method of claim 6, further comprising: applying the motion data associated with the first person extracted from the first video and the appearance data associated with the second person extracted from the second video to a second machine learning technique, the second machine learning technique being trained to apply motion data associated with a third person to an appearance data associated with a fourth person and to render a new video that depicts the third person wearing a garment of the fourth person. ### Claim 8 (depends on 7) The method of claim 7, further comprising training the first machine learning technique and the second machine learning technique by iterating through a sequence of operations comprising: receiving a first training video that depicts a training person in a first training pose and wearing a first training garment; receiving a second training video that depicts the training person in a second training pose and wearing a second training garment; applying the first machine learning technique to the first training video and the second training video to generate first and second sets of estimated appearance data and motion data associated with the training person; and applying the first and second sets of estimated appearance data and motion data associated with the training person to the second machine learning technique to generate a depiction of the training person wearing the second training garment. ### Claim 9 (depends on 8) The method of claim 8, further comprising: computing a deviation between the generated depiction of the training person wearing the second training garment and the second training video that depicts the training person wearing the second training garment; and updating one or more parameters of the first machine learning technique and the second machine learning technique based on the computed deviation. ### Claim 10 (depends on 8) The method of claim 8, wherein the first machine learning technique comprises an appearance data and motion data extraction module and the second machine learning technique comprises an image generator module. ### Claim 11 (depends on 1) The method of claim 1, wherein modifying the second pose of the second person comprises: extracting the appearance data associated with the second person from the second video; and extracting the motion data of the first person from the first video. ### Claim 12 (depends on 11) The method of claim 11, further comprising: applying a mask to the third video to obtain a set of pixels corresponding to the second garment; and copying pixels within the mask to corresponding pixel positions in the first video to generate a fourth video that includes a depiction of the first person wearing the second garment instead of the first garment. ### Claim 13 (depends on 1) The method of claim 1, further comprising: capturing the second video that depicts the second person using an image capture device of a client device of the first person; and after capturing the second video, capturing the first video that depicts the first person using the image capture device. ### Claim 14 (depends on 13) The method of claim 13, wherein the second video is captured using a rear-facing camera of the client device and the first video is captured using a front-facing camera of the client device. ### Claim 15 (depends on 1) The method of claim 1, wherein the first video and the second video are captured simultaneously using different cameras of a client device. ### Claim 16 (depends on 1) The method of claim 1, wherein a particular video depicting the change in appearance of the first person is generated in real-time as the first video and the second video are being captured. ### Claim 17 A system comprising: at least one processor; and a memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: receiving a first video that includes a depiction of a first person wearing a first garment in a first pose; obtaining a second video that includes a depiction of a second person wearing a second garment in a second pose; modifying the second pose of the second person depicted in the second video to match the first pose of the first person depicted in the first video by generating a third video that includes a depiction of the second person having appearance data associated with the second person and motion data associated with the first person; after modifying the second pose of the second person to match the first pose of the first person, generating a whole-body segmentation of the second garment which the second person is wearing in the third video; and changing appearance data representing an appearance of the first person from wearing the first garment to wearing the second garment based on the whole-body segmentation of the second garment which the second person is wearing. ### Claim 18 (depends on 17) The system of claim 17, wherein changing the appearance data associated with the first person comprises: extracting the appearance data and the motion data associated with the first person from the first video; extracting the appearance data and motion data associated with the second person from the second video; modifying the motion data associated with the second person to match the motion data associated with the first person; applying, based on the whole-body segmentation, a mask to the appearance data associated with the second person after modifying the motion data associated with the second person to obtain pixels corresponding to the second garment; replacing a portion of the appearance data associated with the first person based on the pixels corresponding to the second garment in the mask applied to the appearance data associated with the second person; and based on replacing the portion of the appearance data associated with the first person, generating a fourth video that includes a depiction of the first person wearing the second garment instead of the first garment. ### Claim 19 (depends on 18) The system of claim 18, the operations further comprising: applying the mask to the appearance data associated with the first person to identify a set of target pixels; and copying the pixels corresponding to the second garment to the set of target pixels. ### Claim 20 A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: receiving a first video that includes a depiction of a first person wearing a first garment in a first pose; obtaining a second video that includes a depiction of a second person wearing a second garment in a second pose; modifying the second pose of the second person depicted in the second video to match the first pose of the first person depicted in the first video by generating a third video that includes a depiction of the second person having appearance data associated with the second person and motion data associated with the first person; after modifying the second pose of the second person to match the first pose of the first person, generating a whole-body segmentation of the second garment which the second person is wearing in the third video; and changing appearance data representing an appearance of the first person from wearing the first garment to wearing the second garment based on the whole-body segmentation of the second garment which the second person is wearing.