# US 12,002,175 B2 · Real-time motion transfer for prosthetic limbs > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,175 B2 - Document ID: US12002175B2 - Status: Granted - Kind: B2 - Application number: 18/216,958 - Filed: 2023-06-30 - Published: 2023-10-26 - Granted: 2024-06-04 - Earliest claimed: 2020-11-18 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Avihay Assouline; Itamar Berger; Gal Dudovitch; Matan Zohar - Web page: https://patentgrep.com/patent/US12002175B2 ## CPC - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/251: {involving models} - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/20212: Image combination - G06T 2207/30196: Human being; Person - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for performing operations comprising: receiving a video that depicts a person; identifying a set of skeletal joints corresponding to limbs of the person; tracking 3D movement of the set of skeletal joints corresponding to the limbs of the person in the video; causing display of a 3D virtual object that has a plurality of limbs including one or more extra limbs than the limbs of the person in the video; and moving the one or more extra limbs of the 3D virtual object based on the movement of the set of skeletal joints corresponding to the limbs of the person in the video. ## Claims ### Claim 1 A method comprising: accessing, by one or more processors, a video that depicts an object; identifying a set of skeletal joints corresponding to limbs of the object; tracking three-dimensional (3D) movement of the set of skeletal joints corresponding to the limbs of the object depicted in the video; causing display of a 3D virtual object that has a plurality of limbs including one or more extra limbs than the limbs of the object in the video; computing a 3D position for placement of the 3D virtual object relative to a 3D reference point of the object; causing the 3D virtual object to be displayed within the video at the 3D position; and updating the 3D position of the 3D virtual object in the video based on one or more changes to the 3D reference point of the object. ### Claim 2 (depends on 1) The method of claim 1, further comprising: mapping a first set of the plurality of limbs of the 3D virtual object to a first subset of the set of skeletal joints, movement of the first set of the plurality of limbs tracking movement of the first subset of the set of skeletal joints; and moving the one or more extra limbs of the 3D virtual object that are not included in the first set of the plurality of limbs of the 3D virtual object based on movement of the first subset of the set of skeletal joints. ### Claim 3 (depends on 2) The method of claim 2, wherein the limbs of the object include four limbs, wherein the 3D virtual object includes five limbs, and wherein the 3D virtual object includes four legs and a trunk. ### Claim 4 (depends on 3) The method of claim 3, wherein the 3D virtual object is an elephant. ### Claim 5 (depends on 2) The method of claim 2, wherein the limbs of the object include two arms and two legs, and wherein the 3D virtual object includes four legs and a tail. ### Claim 6 (depends on 1) The method of claim 1, further comprising: physically adapting the one or more extra limbs based on a speed and acceleration of the set of skeletal joints. ### Claim 7 (depends on 1) The method of claim 1, further comprising: mapping a second set of the plurality of limbs of the 3D virtual object to a second subset of the set of skeletal joints, wherein movement of the second set of the plurality of limbs tracks movement of the second subset of the set of skeletal joints. ### Claim 8 (depends on 7) The method of claim 7, further comprising: determining a type of the one or more extra limbs; selecting a subset of skeletal joints between first and second subsets of the set of skeletal joints based on the type of the one or more extra limbs; and moving the one or more extra limbs based on the selected subset of the skeletal joints, wherein movement of the one or more extra limbs comprises copying movement of the selected subset of skeletal joints. ### Claim 9 (depends on 7) The method of claim 7, further comprising: determining a type of the one or more extra limbs; combining movement of first and second subsets of the set of skeletal joints based on the type of the one or more extra limbs; and moving the one or more extra limbs based on the combined movement of the first and second subsets of the skeletal joints. ### Claim 10 (depends on 1) The method of claim 1, wherein moving the one or more extra limbs comprises: moving the one or more extra limbs based on physical movements of the set of skeletal joints in an independent manner. ### Claim 11 (depends on 1) The method of claim 1, further comprising: performing motion retargeting, using inverse kinematics, to move the one or more extra limbs based on physical movements of the set of skeletal joints. ### Claim 12 (depends on 11) The method of claim 11, wherein the 3D virtual object comprises a 3D centaur, the 3D centaur comprising two front legs, two back legs, and two arms, further comprising: mapping the two arms of the 3D centaur to a first subset of the set of skeletal joints corresponding to arms of the object; mapping the two front legs of the 3D centaur to a second subset of the set of skeletal joints corresponding to legs of the object; and retargeting motion of the second subset of the set of skeletal joints corresponding to legs of the object to the back legs of the 3D centaur. ### Claim 13 (depends on 1) The method of claim 1, wherein the 3D virtual object is presented together with the object in the video. ### Claim 14 (depends on 1) The method of claim 1, further comprising: determining that the object has moved from a first 3D position to a second 3D position between a first frame and a second frame of the video; computing a distance and trajectory of the movement of the object from the first 3D position to the second 3D position; and moving the 3D virtual object from a third 3D position to a fourth 3D position based on the distance and trajectory of the movement of the object from the first 3D position to the second 3D position. ### Claim 15 (depends on 1) The method of claim 1, wherein the 3D movement of the set of skeletal joints corresponding to the limbs of the object in the video are tracked using images captured by an RGB camera of a device. ### Claim 16 A system comprising: one or more processors configured to perform operations comprising: accessing a video that depicts an object; identifying a set of skeletal joints corresponding to limbs of the object; tracking three-dimensional (3D) movement of the set of skeletal joints corresponding to the limbs of the object depicted in the video; causing display of a 3D virtual object that has a plurality of limbs including one or more extra limbs than the limbs of the object in the video; computing a 3D position for placement of the 3D virtual object relative to a 3D reference point of the object; causing the 3D virtual object to be displayed within the video at the 3D position; and updating the 3D position of the 3D virtual object in the video based on one or more changes to the 3D reference point of the object. ### Claim 17 (depends on 16) The system of claim 16, wherein the 3D movement of the set of skeletal joints corresponding to the limbs of the object in the video are tracked using images captured by an RGB camera of a device. ### Claim 18 A non-transitory machine-readable storage medium including an augmented reality system that includes instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising: accessing a video that depicts an object; identifying a set of skeletal joints corresponding to limbs of the object; tracking three-dimensional (3D) movement of the set of skeletal joints corresponding to the limbs of the object depicted in the video; causing display of a 3D virtual object that has a plurality of limbs including one or more extra limbs than the limbs of the object in the video; computing a 3D position for placement of the 3D virtual object relative to a 3D reference point of the object; causing the 3D virtual object to be displayed within the video at the 3D position; and updating the 3D position of the 3D virtual object in the video based on one or more changes to the 3D reference point of the object. ### Claim 19 (depends on 18) The non-transitory machine-readable storage medium of claim 18, wherein the limbs of the object include four limbs, and wherein the 3D virtual object includes five limbs. ### Claim 20 (depends on 18) The non-transitory machine-readable storage medium of claim 18, wherein the 3D movement of the set of skeletal joints corresponding to the limbs of the object in the video are tracked using images captured by an RGB camera of a device.