# US 8,437,535 B2 · System and method of determining object pose

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 8,437,535 B2
- Document ID: US8437535B2
- Status: Granted
- Kind: B2
- Application number: 11/857,680
- Filed: 2007-09-19
- Published: 2008-03-20
- Granted: 2013-05-07
- Earliest claimed: 2006-09-19
- Assignees: RoboticVISIONTech LLC
- Applicants: Remus F. Boca; Simona Liliana Pescaru; Jeffrey Scott Beis; Babak Habibi
- Inventors: Remus F. Boca; Simona Liliana Pescaru; Jeffrey Scott Beis; Babak Habibi
- Web page: https://patentgrep.com/patent/US8437535B2

## CPC

- G01B 11/25: by projecting a pattern, e.g. {one or more lines,} moiré fringes on the object (G01B11/255 takes precedence {; image analysis for depth or shape recovery G06T7/50})
- G01S 17/875: for determining attitude
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES

## Abstract

Briefly described, one embodiment determines pose of an object of interest at a run time by capturing a first image of a first structured light pattern projected onto a first local surface of the object of interest; determining a first run-time data set from the captured first image, wherein the first run-time data set corresponds to information determined from the first structured light pattern projected onto the first local surface; comparing the determined first run-time data set and a corresponding first reference data set, the first reference data set corresponding to an ideal pose of the first local surface on an ideally posed reference object; and determining at least one first degree of constraint that defines a first partial pose of the first local surface, the at least one first degree of constraint based upon the comparison of the first run-time data set with the corresponding first reference data set.

## Claims

### Claim 1

A method for determining a pose of an object of interest at a run time, the method comprising:
capturing a first image of a first structured light pattern projected onto a first local surface of the object of interest, the first structured light pattern including a plurality of lines of structured light;
wherein the first local surface is a portion of a surface on the object of interest characterized by the absence of a discernible feature that, when captured in the first image, would otherwise provide sufficient information to determine a complete pose of the object of interest; determining a first run-time data set from the captured first image, wherein the first run-time data set corresponds to information determined from the first structured light pattern projected onto the first local surface and corresponds to surface contours of the first local surface determined from the plurality of lines of structured light of the first structured light pattern;
comparing the determined first run-time data set and a corresponding first reference data set, the first reference data set corresponding to an ideal pose of the first local surface on an ideally posed reference object; and
determining at least one first degree of constraint that defines a first partial pose of the first local surface, the at least one first degree of constraint based upon the comparison of the first run-time data set with the corresponding first reference data set.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
capturing a second image of a second structured light pattern projected onto a second local surface of the object of interest;
determining a second run-time data set from the captured second image, wherein the second run-time data set corresponds to information determined from the second structured light pattern projected onto the second local surface;
comparing the determined second run-time data set and a corresponding second reference data set, the second reference data set corresponding to the ideal pose of the second local surface on the ideally posed reference object; and
determining at least one second degree of constraint that defines a second partial pose of the second local surface, the at least one second degree of constraint based upon the comparison of the second run-time data set and the corresponding second reference data set.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
determining the pose of the object of interest based upon the at least one first degree of constraint that defines the first partial pose of the first local surface and the at least one second degree of constraint that defines the second partial pose of the second local surface.

### Claim 4 (depends on 2)

The method of claim 2, further comprising:
combining the at least one first degree of constraint that defines the first partial pose of the first local surface with the at least one second degree of constraint that defines the second partial pose of the second local surface; and
determining the pose of the object of interest based upon the combined at least one first degree of constraint and the at least one second degree of constraint.

### Claim 5 (depends on 2)

The method of claim 2 wherein comparing the determined first run-time data set and the corresponding first reference data set, determining the at least one first degree of constraint, comparing the determined second run-time data set and the corresponding second reference data set, and determining the at least one second degree of constraint is replaced by the method comprising:
combining the first run-time data set with the second run-time data set to determine a global data set;
comparing the global data set and a corresponding reference global data set, the reference global data set data set corresponding to the ideal pose of at least the first local surface and the second local surface on the ideally posed reference object; and
determining a plurality of degrees of constraint that define at least partial pose of the first local surface and the second local surface.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
determining the pose of the object of interest based upon the determined plurality of degrees of constraint.

### Claim 7 (depends on 2)

The method of claim 2, further comprising:
moving an image capture device after capturing the first image so that the second image is captured with a different orientation.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
determining a transformation that defines a spatial relationship between the first run-time data set and the corresponding first reference data set.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
projecting the first structured light pattern onto the first local surface of the object of interest.

### Claim 10 (depends on 9)

The method of claim 9 wherein projecting the first structured light pattern comprises:
projecting a first structured laser light pattern.

### Claim 11 (depends on 9)

The method of claim 9 wherein projecting the first structured light pattern comprises:
projecting a first moiré fringe pattern.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
capturing a second image of the first structured light pattern projected onto the first local surface of the object of interest; and
determining a second run-time data set from the captured second image, wherein the second run-time data set corresponds to information determined from the first structured light pattern projected onto the first local surface.

### Claim 13 (depends on 12)

The method of claim 12, further comprising:
comparing the determined first run-time data set and the second run-time data set to determine an aggregate run-time data set;
comparing the determined aggregate run-time data set and a corresponding reference data set, the reference data set corresponding to an ideal pose of the first local surface on an ideally posed reference object; and
determining at least one first degree of constraint that defines a first partial pose of the first local surface, the at least one first degree of constraint based upon the comparison of the aggregate run-time data set with the corresponding first reference data set.

### Claim 14

A method for determining a pose of an object of interest at a run time, the method comprising:
capturing a first image of a first structured light pattern projected onto a first local surface of the object of interest;
determining a first run-time data set from the captured first image, wherein the first run-time data set corresponds to information determined from the first structured light pattern projected onto the first local surface;
comparing the determined first run-time data set and a corresponding first reference data set, the first reference data set corresponding to an ideal pose of the first local surface on an ideally posed reference object;
determining at least one first degree of constraint that defines a first partial pose of the first local surface, the at least one first degree of constraint based upon the comparison of the first run-time data set with the corresponding first reference data set;
comparing the determined first run-time data set and the second run-time data set to determine an aggregate run-time data set;
comparing the determined aggregate run-time data set and a corresponding reference data set, the reference data set corresponding to an ideal pose of the first local surface on an ideally posed reference object;
determining at least one first degree of constraint that defines a first partial pose of the first local surface, the at least one first degree of constraint based upon the comparison of the aggregate run-time data set with the corresponding first reference data set
comparing the determined second run-time data set and a corresponding reference data set, the reference data set corresponding to an ideal pose of the first local surface on an ideally posed reference object;
determining at least one second degree of constraint that defines the first partial pose of the first local surface, the at least one second degree of constraint based upon the comparison of the second run-time data set with the corresponding first reference data set;
comparing the determined first degree of constraint and the second degree of constraint to determine an aggregate degree of constraint; and
determining at least partial pose of the first local surface based upon the comparison of the a first degree of constraint and the second degree of constraint.

### Claim 15 (depends on 1)

The method of claim 1, further comprising:
determining a second degree of constraint that further defines the first partial pose of the first local surface, the at least one second degree of constraint based upon the comparison of the first run-time data set with the corresponding first reference data set.

### Claim 16 (depends on 9)

The method of claim 9 wherein projecting the first structured light pattern comprises:
projecting the first structured light pattern using an optical element operable to project the first structured light pattern at an orientation of interest onto the first local surface.

### Claim 17

A system that determines a pose of an object of interest at a run time, comprising:
an image capture system operable to capture a first image of a first structured light pattern projected onto a first local surface of the object of interest and operable to capture a second image of a second structured light pattern projected onto a second local surface of the object of interest, wherein the first structured light pattern includes a first plurality of lines of structured light and wherein the second structured light pattern includes a second plurality of lines of structured light;
a structured light source system operable to project the first structured light pattern onto the first local surface and operable to project the second structured light pattern onto the second local surface; and
a processor communicatively coupled to the image capture system and the structured light source system, and operable to:
determine a first run-time data set from the captured first image, wherein the first run-time data set corresponds to information determined from the first structured light pattern projected onto the first local surface and corresponds to surface contours of the first local surface determined from the first plurality of lines of structured light of the first structured light pattern;
determine a second run-time data set from the captured second image, wherein the second run-time data set corresponds to information determined from the second structured light pattern projected onto the second local surface and corresponds to surface contours of the second local surface determined from the second plurality of lines of structured light of the second structured light pattern;
compare the determined first run-time data set and a corresponding first reference data set, the first reference data set corresponding to an ideal pose of the first local surface on an ideally posed reference object, to determine at least a first degree of constraint corresponding to the pose of the first local surface; and
compare the determined second run-time data set and a corresponding second reference data set, the second reference data set corresponding to the ideal pose of the second local surface on the ideally posed reference object, to determine at least a second degree of constraint corresponding to the pose of the second local surface.

### Claim 18 (depends on 17)

The system of claim 17, wherein the image capture system comprises:
a robot device operable to maneuver at least one robot device member through a workspace;
a single image capture device coupled to the at least one robot device member such that the robot device moves the single image capture device to a first position and a first orientation so that the single image capture device captures the first image, and such that the robot device moves the single image capture device to a second position and a second orientation so that the single image capture device captures the second image.

### Claim 19 (depends on 17)

The system of claim 17, wherein the single image capture system comprises:
a first image capture device at a first position and a first orientation so that the first image capture device captures the first image; and
a second image capture device at a second position and a second orientation so that the second image capture device captures the second image.

### Claim 20 (depends on 17)

The system of claim 17, wherein the structured light source system comprises:
a robot device operable to maneuver at least one robot device member through a workspace;
a single structured light source coupled to the at least one robot device member such that the robot device moves the structured light source system to a first position and a first orientation so that the structured light source system projects the first structured light pattern onto the first local surface, and such that the robot device moves the structured light source system to a second position and a second orientation so that the structured light source system projects the second structured light pattern onto the second local surface.

### Claim 21 (depends on 17)

The system of claim 17, wherein the structured light source system comprises:
a first structured light source at a first position and a first orientation so that the first structured light source projects the first structured light pattern onto the first local surface; and
a second structured light source at a second position and a second orientation so that the second structured light source projects the second structured light pattern onto the second local surface.

### Claim 22 (depends on 17)

The system of claim 17, wherein the structured light source system comprises:
a laser light source, such that the laser light source at least projects a first structured laser light pattern onto the first local surface.

### Claim 23

A method for determining a pose of an object of interest, the method comprising:
capturing at least one reference image of each of a plurality of reference local surfaces on a reference object oriented in a reference pose, wherein each of the plurality of reference local surfaces of the reference object has a structured light pattern projected thereon, the structured light pattern including a plurality of lines of structured light;
determining a plurality of reference data sets, wherein one reference data set is determined for each one of the plurality of reference local surfaces, and wherein the plurality of reference data sets corresponds to information determined from the structured light pattern projected onto its respective reference local surface and corresponds to surface contours of the respective local surface determined from the plurality of lines of structured light; and
capturing at least one image of each of a plurality of local surfaces on the object of interest in an unknown pose at a run time, wherein each of the local surfaces of the object of interest in the unknown pose has the structured light pattern projected thereon, and wherein each one of the local surfaces of the object of interest in the unknown pose correspond to one of the reference local surfaces;
determining a plurality of run-time data sets, wherein one run-time data set is determined for each one of the plurality of local surfaces, and wherein the run-time data sets correspond to information determined from the structured light pattern projected onto its respective local surface; and
comparing the determined run-time data sets and the corresponding reference data sets.

### Claim 24 (depends on 23)

The method of claim 23, further comprising:
determining at least one reference degree of constraint from each of the plurality of reference data sets; and
determining a partial reference pose for each of the reference local surfaces based upon the determined at least one reference degree of constraint.

### Claim 25

A method for determining a pose of an object of interest, the method comprising:
capturing at least one reference image of each of a plurality of reference local surfaces on a reference object oriented in a reference pose, wherein each of the plurality of reference local surfaces of the reference object has a structured light pattern projected thereon;
determining a plurality of reference data sets, wherein one reference data set is determined for each one of the plurality of reference local surfaces, and wherein the plurality of reference data sets corresponds to information determined from the structured light pattern projected onto its respective reference local surface;
capturing at least one image of each of a plurality of local surfaces on the object of interest in an unknown pose at a run time, wherein each of the local surfaces of the object of interest in the unknown pose has the structured light pattern projected thereon, and wherein each one of the local surfaces of the object of interest in the unknown pose correspond to one of the reference local surfaces;
determining a plurality of run-time data sets, wherein one run-time data set is determined for each one of the plurality of local surfaces, and wherein the run-time data sets correspond to information determined from the structured light pattern projected onto the respective local surface;
comparing the determined run-time data sets and the corresponding reference data sets;
determining at least one reference degree of constraint from each of the reference data sets; and
determining at least one run-time degree of constraint from each of the run-time data sets,
wherein comparing the run-time data sets and the reference data sets comprises:
comparing the determined at least one run-time degree of constraint and the determined respective reference degree of constraint; and
determining the pose of the object of interest in the unknown pose based upon the comparison of the determined at least one run-time degree of constraint and the determined respective reference degree of constraint.

### Claim 26 (depends on 25)

The method of claim 25, further comprising:
determining a reference partial pose for each of the reference local surfaces based upon the determined at least one reference degree of constraint; and
determining a run-time partial pose for each of the local surfaces based upon the determined at least one run-time degree of constraint,
wherein comparing the run-time data sets with the reference data sets comprises:
comparing the determined run-time partial pose and the determined respective reference partial pose; and
determining the pose of the object of interest in the unknown pose based upon the comparison of the determined run-time partial pose with the determined respective reference partial pose.

### Claim 27

A system that determines a pose of an object of interest at a run time, comprising:
means for capturing at least one image of each of a plurality of local surfaces on the object of interest in an unknown pose at the run time, wherein each of the local surfaces of the object of interest in the unknown pose has a structured light pattern projected thereon, wherein each structured light pattern includes a plurality of lines of structured light and wherein each one of the local surfaces of the object of interest in the unknown pose corresponds to one of a plurality of reference local surfaces;
means for determining a plurality of run-time data sets, each data set including a plurality of points, wherein one run-time data set is determined for each one of the plurality of local surfaces, and wherein the plurality of run-time data sets corresponds to information determined from the structured light pattern projected onto its respective local surface and corresponds to surface contours of the respective local surface determined from the plurality of lines of structured light; and
means for comparing the determined run-time data sets and a plurality of corresponding reference data sets
means for capturing at least one reference image from a fixed location with respect to each of a plurality of reference local surfaces on a reference object oriented in a reference pose, wherein each of the plurality of reference local surfaces of the reference object has a structured light pattern projected thereon; and
means for determining the plurality of reference data sets, wherein one reference data set is determined for each one of the plurality of reference local surfaces, and wherein the plurality of reference data sets corresponds to information determined from the structured light pattern projected onto its respective reference local surface.