# US 9,436,998 B2 · Systems and methods of constructing three-dimensional (3D) model of an object using image cross-sections

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,436,998 B2
- Document ID: US9436998B2
- Status: Granted
- Kind: B2
- Application number: 14/710,512
- Filed: 2015-05-12
- Published: 2015-08-27
- Granted: 2016-09-06
- Earliest claimed: 2012-01-17
- Assignees: Leap Motion, Inc.
- Applicants: Leap Motion, Inc.
- Inventors: David S. Holz
- Web page: https://patentgrep.com/patent/US9436998B2

## CPC

- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G02B 27/0172: {characterised by optical features}
- G06F 3/005: {Input arrangements through a video camera}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06T 7/292: Multi-camera tracking
- G06T 7/344: {involving models}
- G06T 7/586: from multiple light sources, e.g. photometric stereo
- G06T 7/593: from stereo images
- G06T 7/60: Analysis of geometric attributes
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/75: {involving models}
- G06T 15/04: Texture mapping
- G06T 15/205: {Image-based rendering}
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/64: Three-dimensional [3D] objects
- G06V 40/107: {Static hand or arm}
- G06V 40/11: {Hand-related biometrics; Hand pose recognition}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- H04N 7/183: {for receiving images from a single remote source}
- H04N 13/207: using a single two-dimensional [2D] image sensor
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/0141: {characterised by the informative content of the display}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06T 2200/04: involving 3D image data
- G06T 2200/08: involving all processing steps from image acquisition to 3D model generation
- G06T 2207/10012: Stereo images
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30196: Human being; Person
- G06V 40/117: {Biometrics derived from hands}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Methods and systems for capturing motion and/or determining the shapes and positions of one or more objects in 3D space utilize cross-sections thereof. In various embodiments, images of the cross-sections are captured using a camera based on edge points thereof.

## Claims

### Claim 1

A method of identifying a position and shape of at least a portion of an object or object portion in three-dimensional (3D) space, the method comprising:
capturing, with a plurality of cameras, each camera having a particular vantage point, two or more images generated by casting an output from at least one source onto a portion of an object;
analyzing the two or more images captured by the cameras from the particular vantage points to computationally represent the object portion, as captured, as one or more mathematically represented three-dimensional (3D) surfaces, each 3D surface corresponding to a cross-section of the object portion, based at least in part on a plurality of edge points of the object portion in the image, tangent lines extending from the cameras to at least two of the plurality of edge points and a centerline corresponding to the tangent lines; and
reconstructing the position of and a shape fitting at least a portion of the object portion in 3D space based at least in part on the plurality of edge points and the centerline.

### Claim 2 (depends on 1)

The method of claim 1, further comprising analyzing the one or more surfaces to produce a 3D model.

### Claim 3 (depends on 1)

The method of claim 1, further comprising refining a 3D model by correlating a cross-section of an object portion represented by a first surface with a cross-section of at least one different object portion represented by at least one different 3D surface.

### Claim 4 (depends on 1)

The method of claim 1, further comprising refining a 3D model based on at least one of an object type of an object comprising the object portion and characteristics of the object type.

### Claim 5 (depends on 1)

The method of claim 1, further comprising refining a 3D model by informing results of the analyzing with results of such analyzing as conducted on images obtained at a different time and correspondingly modifying the model.

### Claim 6 (depends on 1)

The method of claim 1, further comprising refining a position of an object portion based upon informing results of the analyzing with results of such analyzing as conducted on images obtained at a different time.

### Claim 7 (depends on 1)

The method of claim 1, further comprising determining constraints on at least one of a number and position of one or more object portions according an object type.

### Claim 8 (depends on 1)

The method of claim 1, further comprising resolving an at least partially occluded object portion by conducting spatial correlations across surfaces.

### Claim 9 (depends on 1)

The method of claim 1, further comprising capturing one or more additional image sets of two or more images; and resolving an at least partially occluded object portion by conducting temporal correlations across image sets.

### Claim 10 (depends on 1)

The method of claim 1, further comprising resolving an at least partially occluded object portion by applying physical constraints of an object type.

### Claim 11 (depends on 1)

The method of claim 1, further comprising refining a 3D model formed by the reconstructing.

### Claim 12 (depends on 11)

The method of claim 11, wherein the model represents a generic hand, the refining further comprising texture mapping details from image capture onto a generic hand model.

### Claim 13 (depends on 11)

The method of claim 11, further comprising morphing the model according to detected object orientation and motion.

### Claim 14 (depends on 11)

The method of claim 11, wherein the refining includes refining a 3D model based at least in part upon one of continuity in motion and continuity in deformation.

### Claim 15 (depends on 1)

The method of claim 1, further comprising refining a 3D model formed by the reconstructing by comparing a reconstructed model of the object portion to object types in a library of object types and characteristic parameters for objects in one or more possible poses; and assigning matching object types to the model when a match based on characteristic parameters is found.

### Claim 16 (depends on 15)

The method of claim 15, wherein the object type includes a hand, the pose includes fingers extended and palm facing the cameras, and the characteristic parameters include five fingers.

### Claim 17 (depends on 15)

The method of claim 15, wherein refining the model further comprises applying a constraint to the model based on characteristics of the object type.

### Claim 18 (depends on 17)

The method of claim 17, wherein the object type includes a hand, and the constraint includes positions and angles of fingers relative to each other and to a palm portion of a hand.

### Claim 19

A head mounted device, comprising:
an optical assembly to display a displayed environment including at least one of a surrounding environment and a virtual environment, the surrounding environment including at least a portion of an object or object portion; and
a motion-capture system facilitating interactive control of the displayed environment, the system including:
a plurality of cameras, each camera having a particular vantage point, to capture two or more images of the object portion;
at least one source to direct illumination onto the object portion; and
an image analyzer coupled to the cameras and the at least one source and configured to:
analyze the two or more images captured by the cameras from the particular vantage points to computationally represent the object portion, as captured, as one or more mathematically represented 3 dimensional (3D) surfaces, each 3D surface corresponding to a cross-section of the object portion, based at least in part on a plurality of edge points of the object portion in the image, tangent lines extending from the cameras to at least two of the plurality of edge points and a centerline corresponding to the tangent lines; and
reconstruct a position of and a shape fitting at least a portion of the object portion in 3D space based at least in part on the plurality of edge points and the centerline.

### Claim 20 (depends on 19)

The head mounted device of claim 19, wherein the motion-capture system further comprises a time-of-flight camera, and wherein a plurality of points on at least one object surface are determined from time-of-flight data.

### Claim 21 (depends on 20)

The head mounted device of claim 20, wherein the motion-capture system further combines time-of-flight data with tangent-line information to provide a more detailed model of an object's shape.

### Claim 22 (depends on 19)

The head mounted device of claim 19, wherein the virtual environment comprises one or more virtual objects manipulable using hand gestures tracked by the motion-capture system.

### Claim 23 (depends on 19)

The head mounted device of claim 19, wherein the object portion comprises at least a portion of a hand; and wherein the motion-capture system detects a position and shape of the hand.

### Claim 24 (depends on 23)

The head mounted device of claim 23, wherein the optical assembly projects gestures made by the hand into the displayed environment enabling interactive control of objects in the displayed environment.

### Claim 25 (depends on 24)

The head mounted device of claim 24, wherein the displayed environment comprises at least one selected from a gaming application and an internet browsing application.