# US 10,372,226 B2 · Visual language for human computer interfaces > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,372,226 B2 - Document ID: US10372226B2 - Status: Granted - Kind: B2 - Application number: 15/346,665 - Filed: 2016-11-08 - Published: 2017-06-01 - Granted: 2019-08-06 - Earliest claimed: 2013-03-08 - Assignees: FastVDO LLC - Applicants: Fastvdo LLC - Inventors: Wei Dai; Madhu Peringassery Krishnan; Pankaj Topiwala - Web page: https://patentgrep.com/patent/US10372226B2 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06T 7/11: Region-based segmentation - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/73: using feature-based methods - G06T 7/90: Determination of colour characteristics - G06T 2207/10024: Color image - G06T 2207/30196: Human being; Person - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Embodiments of the invention recognize human visual gestures, as captured by image and video sensors, to develop a visual language for a variety of human computer interfaces. One embodiment provides a method for recognizing a hand gesture positioned by a user hand. The method includes steps of capturing a digital color image of a user hand against a background, applying a general parametric model to the digital color image of the user hand to generate a specific parametric template of the user hand, receiving a second digital image of the user hand positioned to represent a hand gesture, detecting a hand contour of the hand gesture based at least in part on the specific parametric template of the user hand, and recognizing the hand gesture based at least in part on the detected hand contour. Other embodiments include recognizing hand gestures, facial gestures or body gestures captured in a video. ## Claims ### Claim 1 A computer-implemented method for recognizing hand gestures, the method comprising: presenting a region of interest (ROI) on a display; receiving a digital color image of a user hand against a background, the digital color image captured using a digital image capturing device, the digital color image being represented by pixels in a first color space; selecting a set of pixels of the digital color image in the first color space from the pixels of the digital color image within the ROI, the selected set of pixels of the digital color image in the first color space describing a general parametric model associated with the digital color image in the first color space; obtaining specific parametric templates in additional color spaces, each specific parametric template in a color space comprising a selected set of pixels within the ROI representing the user hand in a corresponding color space of the additional color spaces, the additional color spaces emphasizing chrominance over luminance information; combining the specific parametric templates in the additional color spaces to generate an updated specific parametric template; obtaining a contour of the user hand by applying the updated specific parametric template to subsequent digital images of the user hand; and detecting a hand gesture based on the contour of the user hand. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein obtaining the contour of the user hand comprises: identifying points corresponding to finger tips and convex hulls between the finger tips in the subsequent digital images of the user hand based on the updated specific parametric template; and generating a polygonal contour map of the user hand based on the identified points. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein identifying the points comprises: detecting one or more convexity defects of the convex hulls, a convexity defect comprising a start point, a depth point and an end point; identifying a center of the digital color image of the user hand; and filtering one or more points that do not represent finger tips or finger joints of the user hand. ### Claim 4 (depends on 2) The computer-implemented method of claim 2, wherein detecting the hand gesture comprises analyzing a structure, orientation and motion of the user hand based on the polygonal contour map through a nonlinear classifier to select the hand gesture from a pre-defined vocabulary of gestures. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein a first color space model of the two independent color space models is selected from a hue-saturation-value color space and the first color space model is represented by hue, saturation and value parameters. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein a second color space model of the two independent color space models is selected from a luma-chroma color space and the second color space model is represented by luma and chroma parameters. ### Claim 7 (depends on 4) The computer-implemented method of claim 4, wherein applying the two independent color space models to the digital color image of the user hand comprises: for a color space of a color space model chosen from the color spaces of the two independent color space models: calculating one or more first-order statistics of the color space model in the chosen color space; calculating one or more first-order statistics of the color space model in a red-green-blue color space; determining distribution ranges of each parameter of the chosen color space model in the chosen color space; and determining distribution ranges of each parameter of the chosen color space model in the red-green-blue color space. ### Claim 8 (depends on 4) The computer-implemented method of claim 4, further comprising: generating an updated skin map of the user hand by: performing a logical inclusive OR operation to the two generated skin maps, the logical inclusive OR operation selecting a pixel of the digital color image of the user hand responsive to the pixel being in one of the two generated skin maps; and generating the updated skin map of the user hand based on the logical inclusive OR operation. ### Claim 9 A non-transitory computer readable medium storing instructions for recognizing hand gestures, the instruction when executed by one or more processors cause the one or more processors to: present a region of interest (ROI) on a display; receive a digital color image of a user hand against a background, the digital color image captured using a digital image capturing device, the digital color image being represented by pixels in a first color space; select a set of pixels of the digital color image in the first color space from the pixels of the digital color image within the ROI, the selected set of pixels of the digital color image in the first color space describing a general parametric model associated with the digital color image in the first color space; obtain specific parametric templates in additional color spaces, each specific parametric template in a color space comprising a selected set of pixels within the ROI representing the user hand in a corresponding color space of the additional color spaces, the additional color spaces emphasizing chrominance over luminance information; combine the specific parametric templates in the additional color spaces to generate an updated specific parametric template; obtain a contour of the user hand by applying the updated specific parametric template to subsequent digital images of the user hand; and detect a hand gesture based on the contour of the user hand. ### Claim 10 A computer implemented method for recognizing a visual gesture, the method comprising: receiving a first set of images including a part of a human body within a region of interest (ROI), the part of a human body oriented in a first configuration; registering a flesh tone of the part of the human body within the ROI of the first set of images in a first color space and a second color space; receiving a second set of images including the part of the human body, the part of the human body oriented in different configurations than the first configuration, the part of the human body oriented in the different configurations representing a visual gesture; identifying one or more objects in the second set of images corresponding to the part of the human body based on the registered flesh tone in the first color space and the second color space; obtaining motion vectors of the one or more objects in the second set of images by tracking the one or more objects in the second set of images; and determining the visual gesture represented by the part of the human body oriented in the different configurations according to the identified one or more objects and the motion vectors. ### Claim 11 (depends on 10) The computer implemented method of claim 10, wherein the part of the human body is a hand, a face, or an entire body. ### Claim 12 A computer-implemented method for recognizing hand gestures in video imagery, wherein the hand gesture has a static element, a motion element, or a mixture of the two, the method comprising: pre-registering a user hand, using one or multiple frames of the video; performing an adaptive hand-detection on subsequent video frames, performing an adaptive hand-detection comprising: applying at least one of skin color/tone analysis in color spaces and motion estimation, to segment regions or objects, especially a hand, within the video frames, wherein said skin color/tone analysis in color spaces comprises: using skin tone analysis in one or more color spaces, to obtain skin map(s), merging skin maps from the one or more color spaces, and using an adaptive threshold to segment a region or object by detecting skin pixels and grouping them together, and wherein said motion estimation comprises: obtaining motion vector fields between two successive frames, typically for blocks within video frames, tracking motion vectors on subsequent frames, applying a combination of cluster analysis of motion vectors, and tracking the evolution of motion vectors on spatial regions and their features; and recognizing a hand gesture in part by applying at least one of the following four tool groups: (i) skin color/tone analysis in one or more color spaces, (ii) motion estimation, (iii) morphological operations, and (iv) other image processing tools, on regions or objects, especially a segmented hand, to detect hands, hand contours and their features, as well as any motion of hand parts; wherein said skin color/tone analysis in one or more color spaces comprises: using skin tone analysis in one or more color spaces, to obtain skin map(s), merging skin maps from the one or more color spaces, and using an adaptive threshold to segment a region or object by detecting skin pixels and grouping them together, wherein said motion estimation comprises: obtaining motion vectors between two successive frames, typically for blocks within video frames, tracking motion vectors on subsequent frames, and applying any combination of cluster analysis of motion vectors, and tracking the evolution of motion vectors, defined spatial regions and their features, to detect regions or objects within video frames, wherein said morphological operations include dilations, erosions, opening, and closing operations, and wherein said other image processing tools include measuring distances, angles, extrema points, convexity, and shape on hand contours. ### Claim 13 (depends on 12) The method of claim 12, wherein the analysis of the evolution of tracked motion vectors includes calculating the spatial position of an object from following the time history of motion vectors, in two or three-dimensional space. ### Claim 14 (depends on 12) The method of claim 12, wherein the adaptive hand-detection processes include at least one of the subprocesses: position, motion, and shape based processing. ### Claim 15 (depends on 12) The method of claim 12, wherein the hand gesture recognition processes include a position-based gesture recognition process, wherein the position-based gesture recognition process includes analysis of the evolution of tracked motion vectors up to a given video frame. ### Claim 16 (depends on 12) The method of claim 12, wherein the hand gesture recognition processes include a motion-based gesture recognition process, wherein the motion-based gesture recognition process includes analysis of the evolution of motion vectors over multiple frames. ### Claim 17 (depends on 12) The method of claim 12, wherein the hand gesture recognition processes include a shape-based gesture recognition process, wherein the shape-based gesture recognition process includes any combination of skin tone analysis in color spaces, morphological operations, and applying other image processing tools over the plurality of pixels. ### Claim 18 (depends on 12) The method of claim 12, wherein the hand gesture recognition processes include at least two of the subprocesses: position, motion, and shape based processing. ### Claim 19 (depends on 9) The non-transitory computer readable medium of claim 9, wherein the instructions when executed by the one or more processors cause the one or more processors to obtain the contour of the user hand by: identifying points corresponding to finger tips and convex hulls between the finger tips in the subsequent digital images of the user hand based on the updated specific parametric template; and generating a polygonal contour map of the user hand based on the identified points. ### Claim 20 (depends on 19) The non-transitory computer readable medium of claim 19, wherein the instructions when executed by the one or more processors cause the one or more processors to identify the points by: detecting one or more convexity defects of the convex hulls, a convexity defect comprising a start point, a depth point and an end point; identifying a center of the digital color image of the user hand; and filtering one or more points that do not represent finger tips or finger joints of the user hand. ### Claim 21 (depends on 19) The non-transitory computer readable medium of claim 19, wherein the instructions when executed by the one or more processors cause the one or more processors to detect the hand gesture by analyzing a structure, orientation and motion of the user hand based on the polygonal contour map through a nonlinear classifier to select the hand gesture from a pre-defined vocabulary of gestures. ### Claim 22 (depends on 21) The non-transitory computer readable medium of claim 21, wherein a first color space model of the two independent color space models is selected from a hue-saturation-value color space and the first color space model is represented by hue, saturation and value parameters. ### Claim 23 (depends on 22) The non-transitory computer readable medium of claim 22, wherein a second color space model of the two independent color space models is selected from a luma-chroma color space and the second color space model is represented by luma and chroma parameters. ### Claim 24 (depends on 21) The non-transitory computer readable medium of claim 21, wherein applying the two independent color space models to the digital color image of the user hand comprises: for a color space of a color space model chosen from the color spaces of the two independent color space models: calculating one or more first-order statistics of the color space model in the chosen color space; calculating one or more first-order statistics of the color space model in a red-green-blue color space; determining distribution ranges of each parameter of the chosen color space model in the chosen color space; and determining distribution ranges of each parameter of the chosen color space model in the red-green-blue color space. ### Claim 25 (depends on 21) The non-transitory computer readable medium of claim 21, wherein the instructions when executed by the one or more processors cause the one or more processors to: generate an updated skin map of the user hand by: perform a logical inclusive OR operation to the two generated skin maps, the logical inclusive OR operation selecting a pixel of the digital color image of the user hand responsive to the pixel being in one of the two generated skin maps; and generate the updated skin map of the user hand based on the logical inclusive OR operation. ### Claim 26 An apparatus for recognizing hand gestures, the apparatus comprising: a computer processor for performing steps comprising: presenting a region of interest (ROI) on a display; receiving a digital color image of a user hand against a background, the digital color image captured using a digital image capturing device, the digital color image being represented by pixels in a first color space; selecting a set of pixels of the digital color image in the first color space from the pixels of the digital color image within the ROI, the selected set of pixels of the digital color image in the first color space describing a general parametric model associated with the digital color image in the first color space; obtaining specific parametric templates in additional color spaces, each specific parametric template in a color space comprising a selected set of pixels within the ROI representing the user hand in a corresponding color space of the additional color spaces, the additional color spaces emphasizing chrominance over luminance information; combining the specific parametric templates in the additional color spaces to generate an updated specific parametric template; obtaining a contour of the user hand by applying the updated specific parametric template to subsequent digital images of the user hand; and detecting a hand gesture based on the contour of the user hand. ### Claim 27 (depends on 26) The apparatus of claim 26, wherein obtaining the contour of the user hand comprises: identifying points corresponding to finger tips and convex hulls between the finger tips in the subsequent digital images of the user hand based on the updated specific parametric template; and generating a polygonal contour map of the user hand based on the identified points. ### Claim 28 (depends on 27) The apparatus of claim 27, wherein identifying the points comprises: detecting one or more convexity defects of the convex hulls, a convexity defect comprising a start point, a depth point and an end point; identifying a center of the digital color image of the user hand; and filtering one or more points that do not represent finger tips or finger joints of the user hand. ### Claim 29 (depends on 27) The apparatus of claim 27, wherein detecting the hand gesture comprises analyzing a structure, orientation and motion of the user hand based on the polygonal contour map through a nonlinear classifier to select the hand gesture from a pre-defined vocabulary of gestures. ### Claim 30 (depends on 29) The apparatus of claim 29, wherein a first color space model of the two independent color space models is selected from a hue-saturation-value color space and the first color space model is represented by hue, saturation and value parameters. ### Claim 31 (depends on 30) The apparatus of claim 30, wherein a second color space model of the two independent color space models is selected from a luma-chroma color space and the second color space model is represented by luma and chroma parameters. ### Claim 32 (depends on 29) The apparatus of claim 29, wherein applying the two independent color space models to the digital color image of the user hand comprises: for a color space of a color space model chosen from the color spaces of the two independent color space models: calculating one or more first-order statistics of the color space model in the chosen color space; calculating one or more first-order statistics of the color space model in a red-green-blue color space; determining distribution ranges of each parameter of the chosen color space model in the chosen color space; and determining distribution ranges of each parameter of the chosen color space model in the red-green-blue color space. ### Claim 33 (depends on 29) The apparatus of claim 29, further comprising: generating an updated skin map of the user hand by: performing a logical inclusive OR operation to the two generated skin maps, the logical inclusive OR operation selecting a pixel of the digital color image of the user hand responsive to the pixel being in one of the two generated skin maps; and generating the updated skin map of the user hand based on the logical inclusive OR operation. ### Claim 34 A non-transitory computer readable medium storing instructions for recognizing a visual gesture, the instruction when executed by one or more processors cause the one or more processors to: receive a first set of images including a part of a human body within a region of interest (ROI), the part of a human body oriented in a first configuration; register a flesh tone of the part of the human body within the ROI of the first set of images in a first color space and a second color space; receive a second set of images including the part of the human body, the part of the human body oriented in different configurations than the first configuration, the part of the human body oriented in the different configurations representing a visual gesture; identify one or more objects in the second set of images corresponding to the part of the human body based on the registered flesh tone in the first color space and the second color space; obtain motion vectors of the one or more objects in the second set of images by tracking the one or more objects in the second set of images; and determine the visual gesture represented by the part of the human body oriented in the different configurations according to the identified one or more objects and the motion vectors. ### Claim 35 (depends on 34) The non-transitory computer readable medium of claim 34, wherein the part of the human body is a hand, a face, or an entire body. ### Claim 36 An apparatus for recognizing a visual gesture, the apparatus comprising: a computer processor for performing steps comprising: receive a first set of images including a part of a human body within a region of interest (ROI), the part of a human body oriented in a first configuration; register a flesh tone of the part of the human body within the ROI of the first set of images in a first color space and a second color space; receive a second set of images including the part of the human body, the part of the human body oriented in different configurations than the first configuration, the part of the human body oriented in the different configurations representing a visual gesture; identify one or more objects in the second set of images corresponding to the part of the human body based on the registered flesh tone in the first color space and the second color space; obtain motion vectors of the one or more objects in the second set of images by tracking the one or more objects in the second set of images; and determine the visual gesture represented by the part of the human body oriented in the different configurations according to the identified one or more objects and the motion vectors. ### Claim 37 (depends on 36) The apparatus of claim 36, wherein the part of the human body is a hand, a face, or an entire body. ### Claim 38 A non-transitory computer readable medium storing instructions for recognizing hand gestures in video imagery, wherein the hand gesture has a static element, a motion element, or a mixture of the two, the instruction when executed by one or more processors cause the one or more processors to: pre-register a user hand, using one or multiple frames of the video; perform an adaptive hand-detection on subsequent video frames, comprising: applying at least one of skin color/tone analysis in color spaces and motion estimation, to segment regions or objects, especially a hand, within the video frames, wherein said skin color/tone analysis in color spaces comprises: using skin tone analysis in one or more color spaces, to obtain skin map(s), merging skin maps from the one or more color spaces, and using an adaptive threshold to segment a region or object by detecting skin pixels and grouping them together, and wherein said motion estimation comprises: obtaining motion vector fields between two successive frames, typically for blocks within video frames, tracking motion vectors on subsequent frames, applying a combination of cluster analysis of motion vectors, and tracking the evolution of motion vectors on spatial regions and their features; and recognize a hand gesture in part by applying at least one of the following four tool groups: (i) skin color/tone analysis in one or more color spaces, (ii) motion estimation, (iii) morphological operations, and (iv) other image processing tools, on regions or objects, especially a segmented hand, to detect hands, hand contours and their features, as well as any motion of hand parts; wherein said skin color/tone analysis in one or more color spaces comprises: using skin tone analysis in one or more color spaces, to obtain skin map(s), merging skin maps from the one or more color spaces, and using an adaptive threshold to segment a region or object by detecting skin pixels and grouping them together, wherein said motion estimation comprises: obtaining motion vectors between two successive frames, typically for blocks within video frames, tracking motion vectors on subsequent frames, and applying any combination of cluster analysis of motion vectors, and tracking the evolution of motion vectors, defined spatial regions and their features, to detect regions or objects within video frames, wherein said morphological operations include dilations, erosions, opening, and closing operations, and wherein said other image processing tools include measuring distances, angles, extrema points, convexity, and shape on hand contours. ### Claim 39 (depends on 38) The non-transitory computer readable medium of claim 38, wherein the analysis of the evolution of tracked motion vectors includes calculating the spatial position of an object from following the time history of motion vectors, in two or three-dimensional space. ### Claim 40 (depends on 38) The non-transitory computer readable medium of claim 38, wherein the adaptive hand-detection processes include at least one of the subprocesses: position, motion, and shape based processing. ### Claim 41 (depends on 38) The non-transitory computer readable medium of claim 38, wherein the hand gesture recognition processes include a position-based gesture recognition process, wherein the position-based gesture recognition process includes analysis of the evolution of tracked motion vectors up to a given video frame. ### Claim 42 (depends on 38) The non-transitory computer readable medium of claim 38, wherein the hand gesture recognition processes include a motion-based gesture recognition process, wherein the motion-based gesture recognition process includes analysis of the evolution of motion vectors over multiple frames. ### Claim 43 (depends on 38) The non-transitory computer readable medium of claim 38, wherein the hand gesture recognition processes include a shape-based gesture recognition process, wherein the shape-based gesture recognition process includes any combination of skin tone analysis in color spaces, morphological operations, and applying other image processing tools over the plurality of pixels. ### Claim 44 (depends on 38) The non-transitory computer readable medium of claim 38, wherein the hand gesture recognition processes include at least two of the subprocesses: position, motion, and shape based processing. ### Claim 45 An apparatus for recognizing hand gestures in video imagery, wherein the hand gesture has a static element, a motion element, or a mixture of the two, the apparatus comprising: a computer processor for performing steps comprising: pre-registering a user hand, using one or multiple frames of the video; performing an adaptive hand-detection on subsequent video frames, performing an adaptive hand-detection comprising: applying at least one of skin color/tone analysis in color spaces and motion estimation, to segment regions or objects, especially a hand, within the video frames, wherein said skin color/tone analysis in color spaces comprises: using skin tone analysis in one or more color spaces, to obtain skin map(s), merging skin maps from the one or more color spaces, and using an adaptive threshold to segment a region or object by detecting skin pixels and grouping them together, and wherein said motion estimation comprises: obtaining motion vector fields between two successive frames, typically for blocks within video frames, tracking motion vectors on subsequent frames, applying a combination of cluster analysis of motion vectors, and tracking the evolution of motion vectors on spatial regions and their features; and recognizing a hand gesture in part by applying at least one of the following four tool groups: (i) skin color/tone analysis in one or more color spaces, (ii) motion estimation, (iii) morphological operations, and (iv) other image processing tools, on regions or objects, especially a segmented hand, to detect hands, hand contours and their features, as well as any motion of hand parts; wherein said skin color/tone analysis in one or more color spaces comprises: using skin tone analysis in one or more color spaces, to obtain skin map(s), merging skin maps from the one or more color spaces, and using an adaptive threshold to segment a region or object by detecting skin pixels and grouping them together, wherein said motion estimation comprises: obtaining motion vectors between two successive frames, typically for blocks within video frames, tracking motion vectors on subsequent frames, and applying any combination of cluster analysis of motion vectors, and tracking the evolution of motion vectors, defined spatial regions and their features, to detect regions or objects within video frames, wherein said morphological operations include dilations, erosions, opening, and closing operations, and wherein said other image processing tools include measuring distances, angles, extrema points, convexity, and shape on hand contours. ### Claim 46 (depends on 45) The apparatus of claim 45, wherein the analysis of the evolution of tracked motion vectors includes calculating the spatial position of an object from following the time history of motion vectors, in two or three-dimensional space. ### Claim 47 (depends on 45) The apparatus of claim 45, wherein the adaptive hand-detection processes include at least one of the subprocesses: position, motion, and shape based processing. ### Claim 48 (depends on 45) The apparatus of claim 45, wherein the hand gesture recognition processes include a position-based gesture recognition process, wherein the position-based gesture recognition process includes analysis of the evolution of tracked motion vectors up to a given video frame. ### Claim 49 (depends on 45) The apparatus of claim 45, wherein the hand gesture recognition processes include a motion-based gesture recognition process, wherein the motion-based gesture recognition process includes analysis of the evolution of motion vectors over multiple frames. ### Claim 50 (depends on 45) The apparatus of claim 45, wherein the hand gesture recognition processes include a shape-based gesture recognition process, wherein the shape-based gesture recognition process includes any combination of skin tone analysis in color spaces, morphological operations, and applying other image processing tools over the plurality of pixels. ### Claim 51 (depends on 45) The apparatus of claim 45, wherein the hand gesture recognition processes include at least two of the subprocesses: position, motion, and shape based processing.