# US 8,787,663 B2 · Tracking body parts by combined color image and depth processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,787,663 B2 - Document ID: US8787663B2 - Status: Granted - Kind: B2 - Application number: 13/036,022 - Filed: 2011-02-28 - Published: 2011-09-01 - Granted: 2014-07-22 - Earliest claimed: 2010-03-01 - Assignees: Primesense Ltd. - Applicants: Shai Litvak; Tomer Yanir; Eran Guendelman - Inventors: Shai Litvak; Tomer Yanir; Eran Guendelman - Web page: https://patentgrep.com/patent/US8787663B2 ## CPC - G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light - G06F 18/251: {of input or preprocessed data} - G06T 7/248: {involving reference images or patches} - G06T 7/277: involving stochastic approaches, e.g. using Kalman filters - G06T 7/73: using feature-based methods - G06T 7/90: Determination of colour characteristics - G06V 10/803: {of input or preprocessed data} - G06V 40/107: {Static hand or arm} - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20076: Probabilistic image processing - G06T 2207/30196: Human being; Person - G06T 2207/30241: Trajectory - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method for image processing includes receiving a depth image of a scene containing a human subject and receiving a color image of the scene containing the human subject. A part of a body of the subject is identified in at least one of the images. A quality of both the depth image and the color image is evaluated, and responsively to the quality, one of the images is selected to be dominant in processing of the part of the body in the images. The identified part is localized in the dominant one of the images, while using supporting data from the other one of the images. ## Claims ### Claim 1 A method for image processing, comprising: receiving a depth image of a scene containing a human subject; receiving a color image of the scene containing the human subject; identifying a part of a body of the subject in at least one of the images; evaluating a quality of both the depth image and the color image, and selecting, responsively to the quality, one of the images to be dominant in processing of the part of the body in the images; and localizing the identified part in the dominant one of the images, while using supporting data from the other one of the images. ### Claim 2 (depends on 1) The method according to claim 1, wherein receiving the depth image comprises projecting patterned optical radiation toward the subject, and capturing and processing a two-dimensional image of the patterned optical radiation that is reflected from the subject. ### Claim 3 (depends on 2) The method according to claim 2, wherein projecting the patterned optical radiation comprises projecting a spatial pattern of the optical radiation, and wherein processing the two-dimensional image comprises deriving depth coordinates of the body of the subject based on transverse shifts of the spatial pattern in the two-dimensional image. ### Claim 4 (depends on 2) The method according to claim 2, wherein projecting the patterned optical radiation comprises projecting infrared radiation. ### Claim 5 (depends on 1) The method according to claim 1, wherein the part of the body comprises a hand of the human subject. ### Claim 6 (depends on 5) The method according to claim 5, wherein localizing the identified part comprises identifying a group of pixels belonging to the hand, fitting a parametric model to the group, and finding an orientation of the hand based on a parameter of the model. ### Claim 7 (depends on 6) The method according to claim 6, wherein fitting the parametric model comprises fitting an ellipse to the group, and wherein finding the orientation comprises finding a direction of an axis of the ellipse. ### Claim 8 (depends on 5) The method according to claim 5, wherein localizing the identified part comprises detecting an initial location of the hand by recognizing, in a sequence of the images, a predetermined focal gesture made by the hand, and then tracking the hand beginning from the initial location. ### Claim 9 (depends on 1) The method according to claim 1, wherein localizing the identified part comprises deriving from the depth image and color image respective matrix data structures, having dimensions corresponding to the dimensions of the depth image and color image, and combining at least a part of the respective matrix data structures in order to find a location of the part of the body. ### Claim 10 (depends on 9) The method according to claim 9, wherein deriving the respective matrix data structures comprises deriving respective masks indicating pixels that are candidates to belong to the part of the body, and wherein combining the respective matrix data structures comprises applying a logical operation to the respective masks. ### Claim 11 (depends on 9) The method according to claim 9, wherein at least one of the matrix data structures is indicative of the pixels in at least one of the images in which significant changes occurred over a sequence of the images, thereby representing motion in the images. ### Claim 12 (depends on 9) The method according to claim 9, wherein at least one of the matrix data structures is indicative of morphological features of the part of the body. ### Claim 13 (depends on 9) The method according to claim 9, wherein deriving the respective matrix data structures comprises deriving from the depth image and color image respective maps containing graded values expressing an ascending monotonic function associated with a property of interest at pixels in the images, and wherein combining the respective matrix data structures comprises applying a mathematical operation to the respective maps. ### Claim 14 (depends on 13) The method according to claim 13, wherein the graded values indicate respective probabilities that the pixels to belong to the part of the body, and wherein applying the mathematical operation comprises multiplying the respective maps together in order to find the location of the part of the body. ### Claim 15 (depends on 1) The method according to claim 1, wherein when the color image is selected to be the dominant one, localizing the identified part comprises identifying a range of depths in which the part of the body is located, and filtering the color image using depth coordinates from the depth image to eliminate pixels that are outside the range. ### Claim 16 (depends on 1) The method according to claim 1, wherein when the depth image is selected to be the dominant one, localizing and tracking the identified part comprises identifying a range of colors associated with the part of the body, and filtering the depth image using color information from the color image to eliminate pixels having colors that are outside the range. ### Claim 17 (depends on 1) The method according to claim 1, wherein localizing the identified part comprises applying a location and motion filter to a sequence of the depth and color images in order to combine depth- and color-based information from the images. ### Claim 18 (depends on 1) The method according to claim 1, wherein localizing the identified part comprises tracking the identified part over a sequence of the images. ### Claim 19 (depends on 18) The method according to claim 18, wherein tracking the identified part comprises processing at least some of the images in the sequence using both the supporting data and information from one or more previous images in the sequence. ### Claim 20 Apparatus for image processing, comprising: an input interface coupled to receive a depth image of a scene containing a human subject and to receive a color image of the scene containing the human subject; and a processor, which is configured to identify a part of a body of the subject in at least one of the images, to evaluate a quality of both the depth image and the color image, and to select, responsively to the quality, one of the images to be dominant in processing of the part of the body in the images, and to localize the identified part in the dominant one of the images, while using supporting data from the other one of the images. ### Claim 21 (depends on 20) The apparatus according to claim 20, and comprising a imaging assembly, which is configured to project patterned optical radiation toward the subject, and to capture and process a two-dimensional image of the patterned optical radiation that is reflected from the subject in order to generate the depth image. ### Claim 22 (depends on 21) The apparatus according to claim 21, wherein the patterned optical radiation comprises a spatial pattern, and wherein the processor is configured to derive depth coordinates of the body of the subject based on transverse shifts of the spatial pattern in the two-dimensional image. ### Claim 23 (depends on 21) The apparatus according to claim 21, wherein the patterned optical radiation comprises infrared radiation. ### Claim 24 (depends on 20) The apparatus according to claim 20, wherein the part of the body comprises a hand of the human subject. ### Claim 25 (depends on 24) The apparatus according to claim 24, wherein the processor is configured to identify a group of pixels belonging to the hand, to fit a parametric model to the group, and to find an orientation of the hand based on a parameter of the model. ### Claim 26 (depends on 25) The apparatus according to claim 25, wherein the parametric model comprises an ellipse that is fitted to the group of the pixels, and wherein the orientation of the hand is indicated by a direction of an axis of the ellipse. ### Claim 27 (depends on 24) The apparatus according to claim 24, wherein the processor is configured to detect an initial location of the hand by recognizing, in a sequence of the images, a predetermined focal gesture made by the hand, and then to track the hand beginning from the initial location. ### Claim 28 (depends on 20) The apparatus according to claim 20, wherein the processor is configured to derive from the depth image and color image respective matrix data structures, having dimensions corresponding to the dimensions of the depth image and color image, and to combine at least a part of the respective matrix data structures in order to find a location of the part of the body. ### Claim 29 (depends on 28) The apparatus according to claim 28, wherein the respective matrix data structures comprise respective masks indicating pixels that are candidates to belong to the part of the body, and wherein the processor is configured to combine the respective masks using a logical operation. ### Claim 30 (depends on 28) The apparatus according to claim 28, wherein at least one of the matrix data structures is indicative of the pixels in at least one of the images in which significant changes occurred over a sequence of the images, thereby representing motion in the images. ### Claim 31 (depends on 28) The apparatus according to claim 28, wherein at least one of the matrix data structures is indicative of morphological features of the part of the body. ### Claim 32 (depends on 28) The apparatus according to claim 28, wherein the respective matrix data structures comprise respective maps containing graded values expressing an ascending monotonic function associated with a property of interest at pixels in the images, and wherein the processor is configured to combine the respective maps using a mathematical operation. ### Claim 33 (depends on 32) The apparatus according to claim 32, wherein the graded values indicate, for pixels in the images, respective probabilities that the pixels to belong to the part of the body, and wherein the processor is configured to multiply the respective masks together in order to find the location of the part of the body. ### Claim 34 (depends on 20) The apparatus according to claim 20, wherein the processor is configured, upon selecting the color image to be the dominant one, to identify a range of depths in which the part of the body is located, and to filter the color image using depth coordinates from the depth image to eliminate pixels that are outside the range. ### Claim 35 (depends on 20) The apparatus according to claim 20, wherein the processor is configured, upon selecting the depth image to be the dominant one, to identify a range of colors associated with the part of the body, and to filter the depth image using color information from the color image to eliminate pixels having colors that are outside the range. ### Claim 36 (depends on 20) The apparatus according to claim 20, wherein the processor is configured to apply a location and motion filter to a sequence of the depth and color images in order to combine depth- and color-based information from the images. ### Claim 37 (depends on 20) The apparatus according to claim 20, wherein the processor is configured to track the identified part over a sequence of the images. ### Claim 38 (depends on 37) The apparatus according to claim 37, wherein the processor is configured to track the identified part by processing at least some of the images in the sequence using both the supporting data and information from one or more previous images in the sequence. ### Claim 39 A computer software product, comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a depth image of a scene containing a human subject and to receive a color image of the scene containing the human subject, to identify a part of a body of the subject in at least one of the images, to evaluate a quality of both the depth image and the color image, and to select, responsively to the quality, one of the images to be dominant in processing of the part of the body in the images, and to localize the identified part in the dominant one of the images, while using supporting data from the other one of the images. ### Claim 40 (depends on 39) The product according to claim 39, wherein the depth image is generated by projecting patterned optical radiation toward the subject, and capturing and processing a two-dimensional image of the patterned optical radiation that is reflected from the subject. ### Claim 41 (depends on 40) The product according to claim 40, wherein the patterned optical radiation comprises a spatial pattern, and wherein the instructions cause the computer to derive depth coordinates of the body of the subject based on transverse shifts of the spatial pattern in the two-dimensional image. ### Claim 42 (depends on 40) The product according to claim 40, wherein the patterned optical radiation comprises infrared radiation. ### Claim 43 (depends on 39) The product according to claim 39, wherein the part of the body comprises a hand of the human subject. ### Claim 44 (depends on 43) The product according to claim 43, wherein the instructions cause the computer to identify a group of pixels belonging to the hand, to fit a parametric model to the group, and to find an orientation of the hand based on a parameter of the model. ### Claim 45 (depends on 44) The product according to claim 44, wherein the parametric model comprises an ellipse that is fitted to the group of the pixels, and wherein the orientation of the hand is indicated by a direction of an axis of the ellipse. ### Claim 46 (depends on 43) The product according to claim 43, wherein the instructions cause the computer to detect an initial location of the hand by recognizing, in a sequence of the images, a predetermined focal gesture made by the hand, and then to track the hand beginning from the initial location. ### Claim 47 (depends on 39) The product according to claim 39, wherein the instructions cause the computer to derive from the depth image and color image respective matrix data structures, having dimensions corresponding to the dimensions of the depth image and color image, and to combine at least a part of the respective matrix data structures in order to find a location of the part of the body. ### Claim 48 (depends on 47) The product according to claim 47, wherein the respective matrix data structures comprise respective masks indicating pixels that are candidates to belong to the part of the body, and wherein the instructions cause the computer to combine the respective masks using a logical operation. ### Claim 49 (depends on 47) The product according to claim 47, wherein at least one of the matrix data structures is indicative of the pixels in at least one of the images in which significant changes occurred over a sequence of the images, thereby representing motion in the images. ### Claim 50 (depends on 47) The product according to claim 47, wherein at least one of the matrix data structures is indicative of morphological features of the part of the body. ### Claim 51 (depends on 47) The product according to claim 47, wherein the respective matrix data structures comprise respective maps containing graded values expressing an ascending monotonic function associated with a property of interest at pixels in the images, and wherein the instructions cause the computer to combine the respective maps using a mathematical operation. ### Claim 52 (depends on 51) The product according to claim 51, wherein the graded values indicate, for pixels in the images, respective probabilities that the pixels to belong to the part of the body, and wherein the instructions cause the computer to multiply the respective masks together in order to find the location of the part of the body. ### Claim 53 (depends on 39) The product according to claim 39, wherein the instructions cause the computer, upon selecting the color image to be the dominant one, to identify a range of depths in which the part of the body is located, and to filter the color image using depth coordinates from the depth image to eliminate pixels that are outside the range. ### Claim 54 (depends on 39) The product according to claim 39, wherein the instructions cause the computer, upon selecting the depth image to be the dominant one, to identify a range of colors associated with the part of the body, and to filter the depth image using color information from the color image to eliminate pixels having colors that are outside the range. ### Claim 55 (depends on 39) The product according to claim 39, wherein the instructions cause the computer to apply a location and motion filter to a sequence of the depth and color images in order to combine depth- and color-based information from the images. ### Claim 56 (depends on 39) The product according to claim 39, wherein the instructions cause the computer to track the identified part over a sequence of the images. ### Claim 57 (depends on 56) The product according to claim 56, wherein the instructions cause the computer to track the identified part by processing at least some of the images in the sequence using both the supporting data and information from one or more previous images in the sequence. ### Claim 58 A method for image processing, comprising: receiving a depth image of a scene containing a human subject; receiving a color image of the scene containing the human subject; identifying a part of a body of the subject in at least one of the images; evaluating a quality of both the depth image and the color image, and selecting, responsively to the quality, one of the images to be dominant in processing of the part of the body in the images; and segmenting the identified part in the dominant one of the images, while using supporting data from the other one of the images.