# US 8,542,910 B2 · Human tracking system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,542,910 B2 - Document ID: US8542910B2 - Status: Granted - Kind: B2 - Application number: 13/365,121 - Filed: 2012-02-02 - Published: 2012-05-24 - Granted: 2013-09-24 - Earliest claimed: 2009-10-07 - Assignees: Microsoft Corporation - Applicants: Tommer Leyvand; Johnny Lee; Szymon Stachniak; Craig Peeper; Shao Liu - Inventors: Tommer Leyvand; Johnny Lee; Szymon Stachniak; Craig Peeper; Shao Liu - Web page: https://patentgrep.com/patent/US8542910B2 ## CPC - A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) - A63F 13/428: involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes - A63F 13/52: involving aspects of the displayed game scene - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06T 7/194: involving foreground-background segmentation - G06T 7/251: {involving models} - G06T 7/75: {involving models} - G06V 10/34: Smoothing or thinning of the pattern; Morphological operations; Skeletonisation - G06V 10/426: Graphical representations - G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06V 40/23: {Recognition of whole body movements, e.g. for sport training} - A63F 13/215: comprising means for detecting acoustic signals, e.g. using a microphone - A63F 13/833: Hand-to-hand fighting, e.g. martial arts competition - A63F 2300/1075: using a touch screen - A63F 2300/5553: user representation in the game field, e.g. avatar - A63F 2300/6045: for mapping control signals received from the input arrangement into game commands - A63F 2300/6607: for animating game characters, e.g. skeleton kinematics - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/30196: Human being; Person - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An image such as a depth image of a scene may be received, observed, or captured by a device. A grid of voxels may then be generated based on the depth image such that the depth image may be downsampled. A background included in the grid of voxels may also be removed to isolate one or more voxels associated with a foreground object such as a human target. A location or position of one or more extremities of the isolated human target may be determined and a model may be adjusted based on the location or position of the one or more extremities. ## Claims ### Claim 1 A method for tracking a user, the method comprising: receiving a depth image; generating a plurality of voxels based on a downsampling of the depth image, including generating a first voxel of the plurality of voxels based on at least one pixel that corresponds to background independent of at least one pixel that corresponds to a human target, the first voxel overlapping another voxel in the plurality of voxels; removing a background included in the plurality of voxels to isolate one or more voxels associated with the human target; determining a location or position of one or more extremities of the isolated human target; and adjusting a model based on the location or position of the one or more extremities. ### Claim 2 (depends on 1) The method of claim 1, wherein the model comprises a skeletal model having joints and bones. ### Claim 3 (depends on 1) The method of claim 1, wherein removing the background of the grid of voxels to isolate one or more voxels associated with a human target further comprises: determining whether an object in the grid is in motion; and discarding the voxels associated with the object when, based on the determination, the object is not in motion. ### Claim 4 (depends on 1) The method of claim 1, further comprising determining dimensions of the one or more extremities, and adjusting the joints and the bones of the model based on the determined dimensions. ### Claim 5 (depends on 1) The method of claim 1, wherein the one or more extremities comprises at least one of the following: a head, a centroid, a shoulder, a hip, a leg, an arm, a hand, an elbow, a knee, and a foot. ### Claim 6 (depends on 1) The method of claim 1, further comprising processing the adjusted model. ### Claim 7 (depends on 1) The method of claim 1, wherein each of the voxels in the grid is generated based on a portion or block of pixels included in the received depth image. ### Claim 8 A computer-readable storage medium having stored thereon computer executable instructions for tracking a user, the computer executable instructions comprising instructions for: receiving a depth image comprising pixels; downsampling the pixels in the received depth image to generate a plurality of voxels, the plurality of voxels including a first voxel based on at least one pixel that corresponds to background independently of at least one pixel that corresponds to a human target, the first voxel overlapping another voxel in the plurality of voxels; isolating one or more voxels of the plurality of voxels associated with a human target; determining a location or position of one or more extremities of the isolated human target; and adjusting a model based on the location or position of the one or more extremities. ### Claim 9 (depends on 8) The computer-readable storage medium of claim 8, wherein adjusting the model based on the one or more determined extremities comprises assigning the location or position of the one or more extremities to the corresponding joints of the model. ### Claim 10 (depends on 8) The computer-readable storage medium of claim 8, further comprising instructions for determining dimensions of the one or more extremities, and adjusting the joints and the bones of the model based on the determined dimensions. ### Claim 11 (depends on 8) The computer-readable storage medium of claim 8, wherein the one or more extremities comprises at least one of the following: a head, a centroid, a shoulder, a hip, a leg, an arm, a hand, an elbow, a knee, and a foot. ### Claim 12 (depends on 8) The computer-readable storage medium of claim 8, further comprising processing the adjusted model. ### Claim 13 (depends on 12) The computer-readable storage medium of claim 12, wherein the one or more voxels in are generated based on a portion or block of pixels included in the received depth image. ### Claim 14 A system for tracking a model associated with a user, the system comprising: a capture device, wherein the capture device comprises a camera component that receives a depth image of a scene; and a computing device in operative communication with the capture device, wherein the computing device comprises a processor that: generates a downsampled depth image based on one or more pixels in the depth image received from the capture device, the downsampled depth image comprising a first voxel of a plurality of voxels based on at least one pixel that corresponds to background independent of at least one pixel that corresponds to a human target, the first voxel overlapping another voxel in the plurality of voxels; removes a background of the plurality of voxels to isolate a human target; determines a location or position of one or more extremities of the isolated human target; and adjusts a model based on the location or position of the one or more extremities. ### Claim 15 (depends on 14) The system of claim 14, wherein the processor generates the downsampled depth image based on the one or more pixels in the depth image received from the capture device by generating a voxel for portions or blocks of the one or more pixels. ### Claim 16 (depends on 14) The system of claim 14, wherein the processor adjust the model based on the one or more determined extremities by assigning the location or position of the one or more extremities to the corresponding joints of the model. ### Claim 17 (depends on 14) The system of claim 14, wherein the one or more extremities comprises at least one of the following: a head, a centroid, a shoulder, a hip, a leg, an arm, a hand, an elbow, a knee, and a foot. ### Claim 18 (depends on 1) The method of claim 1, wherein generating the grid of voxels based on the depth image based on downsampling the depth image comprises: determining that at least one pixel of a plurality of pixels that are downsampled to generate a first voxel of the grid of voxels has a valid depth value; determining that at least one pixel of the plurality of pixels has an invalid depth value; generating the first voxel based on the at least one pixel with the valid depth value without using the at least one pixel with the invalid depth value. ### Claim 19 (depends on 1) The method of claim 1, wherein adjusting the model based on the location or position of the one or more extremities comprises: refining a location of position of the model based on the human target as it appears in the depth image.