# US 8,571,263 B2 · Predicting joint positions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,571,263 B2 - Document ID: US8571263B2 - Status: Granted - Kind: B2 - Application number: 13/050,858 - Filed: 2011-03-17 - Published: 2012-09-20 - Granted: 2013-10-29 - Earliest claimed: 2011-03-17 - Assignees: Microsoft Corporation - Applicants: Jamie Daniel Joseph Shotton; Pushmeet Kohli; Ross Brook Girshick; Andrew Fitzgibbon; Antonio Criminisi - Inventors: Jamie Daniel Joseph Shotton; Pushmeet Kohli; Ross Brook Girshick; Andrew Fitzgibbon; Antonio Criminisi - Web page: https://patentgrep.com/patent/US8571263B2 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06N 5/025: Extracting rules from data - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Predicting joint positions is described, for example, to find joint positions of humans or animals (or parts thereof) in an image to control a computer game or for other applications. In an embodiment image elements of a depth image make joint position votes so that for example, an image element depicting part of a torso may vote for a position of a neck joint, a left knee joint and a right knee joint. A random decision forest may be trained to enable image elements to vote for the positions of one or more joints and the training process may use training images of bodies with specified joint positions. In an example a joint position vote is expressed as a vector representing a distance and a direction of a joint position from an image element making the vote. The random decision forest may be trained using a mixture of objectives. ## Claims ### Claim 1 A computer-implemented method of predicting joint positions comprising: receiving an input image of a scene comprising at least part of a human or animal body; for each of a plurality of image elements of the input image, making a plurality of votes, each vote being for a position in the input image corresponding to a joint of the human or animal body; the votes being made by comparing each image element with test image elements displaced therefrom by learnt spatial offsets; and aggregating the votes to obtain at least one predicted joint position. ### Claim 2 (depends on 1) A method as claimed in claim 1 comprising assigning a confidence to each predicted joint position. ### Claim 3 (depends on 1) A method as claimed in claim 1 comprising aggregating the votes by taking into account weights that have been learnt during a training process the weights expressing information about uncertainty of the votes. ### Claim 4 (depends on 3) A method as claimed in claim 3 comprising for each vote, adapting the weight according to a depth of an image element which made the vote. ### Claim 5 (depends on 1) A method as claimed in claim 1 comprising expressing each vote using a vector related to the direction and distance from an image element of the input image making the vote to a position in the input image where the joint is voted to be. ### Claim 6 (depends on 1) A method as claimed in claim 1 wherein making the plurality of votes comprises applying each image element of the input image to a random decision forest which has been trained using one or more decision tree node splitting objectives and a set of training images having labeled joint positions or labeled body parts. ### Claim 7 (depends on 6) A method as claimed in claim 6 which comprises using at least two decision tree node splitting objectives which are different from one another. ### Claim 8 (depends on 6) A method as claimed in claim 6 wherein the one or more decision tree node splitting objectives are selected from any of: optimizing information gain of a body part classification task, optimizing reduction in variance of voted joint positions, optimizing reduction in variance of distance to voted joint positions, optimizing reduction in variance of angle to voted joint positions, optimizing a sum of squared distances from a mean voted joint position, optimizing a balance of a random regression tree. ### Claim 9 (depends on 1) A method as claimed in claim 1 wherein aggregating the votes comprises any of: forming a discrete voting space of votes, using a Parzen window density estimator, using a Parzen window density estimator with a mean shift mode detection process, using expectation maximization, using k-means clustering, using agglomerative clustering, calculating a mean vote. ### Claim 10 A method of training a random decision forest to produce votes for positions of joints of a human or animal body in an image comprising: receiving a plurality of training images having labeled joint positions; receiving at least one decision tree node splitting objective; selecting parameters for use at nodes of trees in the random decision forest by using the training images and the at least one objective; at each leaf node of each tree in the random decision forest obtaining a plurality of votes by applying the training images to the random decision forest with the selected parameters; each vote being for a relative position in a training image predicted to correspond to a joint of the human or animal body; aggregating the votes at each leaf node by any of: listing the votes, forming a histogram of votes, calculating a mean of the votes, and fitting a multi-modal distribution to the votes by any of expectation maximization, mean shift mode detection, k-means clustering and agglomerative clustering. ### Claim 11 (depends on 10) A method as claimed in claim 10 comprising filtering the votes using a threshold prior to aggregating the votes. ### Claim 12 (depends on 11) A method as claimed in claim 11 wherein the threshold is learnt on a per joint basis using a validation set of images. ### Claim 13 (depends on 11) A method as claimed in claim 11 wherein the threshold is a distance of a joint position from an image element voting for that joint position. ### Claim 14 (depends on 10) A method as claimed in claim 10 comprising aggregating the votes by using a probability distribution density estimator to find one or more modes and determining a weight for each mode the weight being related to a number of votes that reached that mode. ### Claim 15 (depends on 14) A method as claimed in claim 14 comprising using a mean shift mode detection process to assess the number of votes that reached each mode. ### Claim 16 (depends on 14) A method as claimed in claim 14 comprising setting a parameter of the density estimator on a per joint basis. ### Claim 17 A computer-implemented joint position prediction system comprising: an input arranged to receive an input image of a scene comprising at least part of a human or animal body; a processor arranged, for each of a plurality of image elements of the input image, to make a plurality of votes, each vote being for a position in the input image corresponding to a joint of the human or animal body; the processor being arranged to aggregate the votes to obtain at least one predicted joint position; the processor being arranged to store each vote using a vector related to the direction and distance from an image element of the input image making the vote to a position in the input image where the joint is voted to be. ### Claim 18 (depends on 17) A system as claimed in claim 17 wherein the processor is arranged to aggregate the votes by taking into account weights expressing an uncertainty associated with each vote. ### Claim 19 (depends on 18) A system as claimed in claim 18 wherein the processor is arranged to adapt the weights according to depth values associated with image elements. ### Claim 20 (depends on 17) A computer game system comprising a joint position prediction system as claimed in claim 17.