# US 11,994,377 B2 · Systems and methods of locating a control object appendage in three dimensional (3D) space > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,994,377 B2 - Document ID: US11994377B2 - Status: Granted - Kind: B2 - Application number: 17/010,531 - Filed: 2020-09-02 - Published: 2020-12-24 - Granted: 2024-05-28 - Earliest claimed: 2012-01-17 - Assignees: Ultrahaptics IP Two Limited - Applicants: Ultrahaptics IP Two Limited - Inventors: David S. Holz - Web page: https://patentgrep.com/patent/US11994377B2 ## CPC - G01B 11/2433: {for measuring outlines by shadow casting} - G06T 7/292: Multi-camera tracking - G06T 7/507: from shading (G06T7/586 takes precedence) - G06T 7/586: from multiple light sources, e.g. photometric stereo - G06T 7/593: from stereo images - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06V 10/145: Illumination specially adapted for pattern recognition, e.g. using gratings - G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) - G06V 20/64: Three-dimensional [3D] objects - G06V 40/107: {Static hand or arm} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - A61B 2017/00207: {with hand gesture control or hand gesture recognition} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06T 17/10: Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes - G06T 2207/10152: Varying illumination - G06T 2207/30196: Human being; Person - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06V 40/11: {Hand-related biometrics; Hand pose recognition} - G06V 40/113: {Recognition of static hand signs} - G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods and systems for capturing motion and/or determining the shapes and positions of one or more objects in 3D space utilize cross-sections thereof. In various embodiments, images of the cross-sections are captured using a camera based on reflections therefrom or shadows cast thereby. ## Claims ### Claim 1 A system for recognizing gestures from a control object moving in three dimensional (3D) space, the system including: one or more processors coupled to a memory, the memory loaded with computer instructions that, when executed by the one or more processors, implement actions including: receiving a result of analyzing a gesture; interpreting the result of analyzing a gesture as an instruction to perform an operation, wherein the result of analyzing a gesture is obtained from another program, which is executed on a processor, to analyze differences in positions of fit closed curves fit to observed edges of a control object thereby tracking motion of the control object while making a gesture in a 3D space by repeatedly: obtaining captured images of a control object moving in 3D space captured from at least two geometrically distinct predetermined vantages; calculating observed edges of the control object from the captured images; fitting closed curves to the observed edges of the control object as captured in the captured images by selecting a closed curve from a family of similar closed curves that fit the observed edges of the control object as captured using an assumed parameter; whenever the control object is a complex control object that includes a palm and multiple fingers, applying actions including obtaining captured images, calculating observed edges, and fitting closed curves to construct multiple fingers of control object appendages; and fitting cross sections of a palm to observed edges of a palm as captured in the captured images. ### Claim 2 (depends on 1) The system of claim 1, wherein one or more fitted parameters of a first fitted closed curve are substituted for one or more of the observed edges and the assumed parameter when fitting an adjacent second closed curve. ### Claim 3 (depends on 1) The system of claim 1, wherein a first fitted closed curve is used to filter fits of additional closed curves. ### Claim 4 (depends on 1) The system of claim 1, wherein tracking motion of the control object while making a gesture in a 3D space further includes: repeatedly applying actions including obtaining captured images, calculating observed edges, and fitting closed curves actions of claim 1 over time; and calculating motion of the control object over time based on differences between modeled locations of the control object. ### Claim 5 (depends on 3) The system of claim 3, wherein tracking motion of the control object while making a gesture in a 3D space further includes: repeatedly applying actions including obtaining captured images, calculating observed edges, fitting closed curves and using a first fitted closed curve to filter fits of additional closed curves actions of claim 3 over time; and calculating motion of a complex control object over time based on differences between modeled locations of a complex control object. ### Claim 6 (depends on 1) The system of claim 1, wherein tracking motion of the control object while making a gesture in a 3D space further includes: determining and fitting a circle selected from among closed curves for a plurality of portions of the control object from the captured images, including: calculating three co planar tangents to observed edges of the control object from the captured images; and fitting a circle to the control object using at least the three co planar tangents. ### Claim 7 (depends on 6) The system of claim 6, wherein tracking motion of the control object while making a gesture in a 3D space further includes: for a complex control object model that includes a palm and multiple fingers, applying the determining and fitting a circle actions of claim 6 to construct multiple fingers of control object appendages; and fitting cross sections of a palm to observed edges of the palm as captured in the captured images. ### Claim 8 (depends on 6) The system of claim 6, wherein tracking motion of the control object while making a gesture in a 3D space further includes: repeatedly applying the determining and fitting a circle actions of claim 6 over time; and calculating motion of the control object over time based on differences between modeled locations of the control object. ### Claim 9 (depends on 7) The system of claim 7, wherein tracking motion of the control object while making a gesture in a 3D space further includes: repeatedly applying the determining and fitting a circle and fitting cross sections actions of claim 7 over time; and calculating motion of a complex control object over time based on differences between modeled locations of the complex control object. ### Claim 10 A non-transitory computer readable medium storing a plurality of instructions for programming one or more processors, which instructions, when executed by one or more processors, implement actions including: receiving a result of analyzing a gesture; interpreting as an instruction the result of analyzing a gesture; wherein the result of analyzing a gesture is obtained from another program executing on a processor from analysis of differences in positions of fit closed curves fit to observed edges of a control object thereby tracking motion of the control object while making a gesture in a 3D space including repeatedly: obtaining captured images of a control object moving in 3D space captured from at least two geometrically distinct predetermined vantages; calculating observed edges of the control object from the captured images; fitting closed curves to the observed edges of the control object as captured in the captured images by selecting a closed curve from a family of similar closed curves that fit the observed edges of the control object as captured using an assumed parameter; and whenever the control object is a complex control object that includes a palm and multiple fingers, applying actions including obtaining captured images, calculating observed edges, and fitting closed curves actions to construct multiple fingers of control object appendages; and fitting cross sections of a palm to observed edges of a palm as captured in the captured images. ### Claim 11 (depends on 10) The non-transitory computer readable medium of claim 10, wherein one or more fitted parameters of a first fitted closed curve are substituted for one or more of the observed edges and the assumed parameter when fitting an adjacent second closed curve. ### Claim 12 (depends on 10) The non-transitory computer readable medium of claim 10, wherein a first fitted closed curve is used to filter fits of additional closed curves to contiguous cross-sections. ### Claim 13 (depends on 10) The non-transitory computer readable medium of claim 10, wherein tracking motion of the control object while making a gesture in a 3D space further includes: repeatedly applying actions including obtaining captured images, calculating observed edges, and fitting closed curves actions of claim 10 overtime; and calculating motion of the control object over time based on differences between modeled locations of the control object. ### Claim 14 (depends on 12) The non-transitory computer readable medium of claim 12, wherein tracking motion of the control object while making a gesture in a 3D space further includes: repeatedly applying actions including obtaining captured images, calculating observed edges, fitting closed curves and using a first fitted closed curve to filter fits of additional closed curves actions of claim 12 overtime; and calculating motion of a complex control object over time based on differences between modeled locations of a complex control object. ### Claim 15 (depends on 10) The non-transitory computer readable medium of claim 10, wherein tracking motion of the control object while making a gesture in a 3D space further includes: determining and fitting a circle selected from among closed curves in the family of similar closed curves for a plurality of portions of the control object from the captured images, including: calculating three co-planar tangents to observed edges of the control object from the captured images; and fitting a circle to the control object using at least the three co-planar tangents. ### Claim 16 (depends on 15) The non-transitory computer readable medium of claim 15, wherein tracking motion of the control object while making a gesture in a 3D space further includes: for a complex control object model that includes a palm and multiple fingers, applying the determining and fitting a circle actions of claim 15 to construct multiple fingers of control object appendages; and fitting cross sections of a palm to observed edges of the palm as captured in the images. ### Claim 17 (depends on 15) The non-transitory computer readable medium of claim 15, wherein tracking motion of the control object while making a gesture in a 3D space further includes: repeatedly applying the determining and fitting a circle actions of claim 15 over time; and calculating motion of the control object over time based on differences between modeled locations of the control object. ### Claim 18 (depends on 16) The non-transitory computer readable medium of claim 16, wherein tracking motion of the control object while making a gesture in a 3D space further includes: repeatedly applying the determining and fitting a circle and fitting cross sections actions of claim 16 overtime; and calculating motion of a complex control object over time based on differences between modeled locations of a complex control object. ### Claim 19 A method for executing a gesture command, the method including: receiving a result of analyzing a gesture; interpreting as an instruction the result of analyzing a gesture; wherein the result of analyzing a gesture is obtained from another program executing on a processor from analysis of differences in positions of fit closed curves fit to observed edges of a control object to track motion of the control object while making a gesture in a 3D space including repeatedly: obtaining captured images of a control object moving in 3D space captured from at least two geometrically distinct predetermined vantages; calculating observed edges of the control object from the captured images; fitting closed curves to the observed edges of the control object as captured in the captured images by selecting a closed curve from a family of similar closed curves that fit the observed edges of the control object as captured using an assumed parameter; and whenever the control object is a complex control object that includes a palm and multiple fingers, applying actions including obtaining captured images, calculating observed edges, and fitting closed curves to construct multiple fingers of control object appendages; and fitting cross sections of a palm to observed edges of a palm as captured in the captured images.