# US 12,109,703 B2 · Method for autonomously scanning a workpiece

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,109,703 B2
- Document ID: US12109703B2
- Status: Granted
- Kind: B2
- Application number: 18/389,166
- Filed: 2023-11-13
- Published: 2024-05-09
- Granted: 2024-10-08
- Earliest claimed: 2022-05-27
- Assignees: GrayMatter Robotics Inc.
- Applicants: GrayMatter Robotics Inc.
- Inventors: Avadhoot L. Ahire; Rishav Guha; Satyandra K. Gupta; Ariyan M. Kabir; Sagarkumar J. Panchal; Brual C. Shah
- Web page: https://patentgrep.com/patent/US12109703B2

## CPC

- B24B 27/0038: {with the grinding tool mounted at the end of a set of bars}
- B25J 9/1653: {parameters identification, estimation, stiffness, accuracy, error analysis}
- B24B 49/12: involving optical means
- B24B 49/16: taking regard of the load
- B24B 51/00: Arrangements for automatic control of a series of individual steps in grinding a workpiece (if applicable to other machine tools, G05B takes precedence)
- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1633: {compliant, force, torque control, e.g. combined with position control}
- B25J 9/1664: {characterised by motion, path, trajectory planning}
- B25J 9/1679: {characterised by the tasks executed}
- B25J 9/1697: {Vision controlled systems}
- B25J 11/0065: {Polishing or grinding}
- B25J 13/085: {Force or torque sensors (B25J13/082, B25J13/084 take precedence)}
- B25J 19/021: {Optical sensing devices}
- G06T 1/0014: {Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023)}
- G06T 7/13: Edge detection
- G06T 7/60: Analysis of geometric attributes
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G05B 2219/40586: 6-DOF force sensor
- G05B 2219/40607: Fixed camera to observe workspace, object, workpiece, global
- G05B 2219/45082: Sanding robot, to clean surfaces
- G06T 2207/10024: Color image
- G06T 2219/004: Annotating, labelling
- G06T 2219/2021: Shape modification
- B24B: MACHINES, DEVICES, OR PROCESSES FOR GRINDING OR POLISHING (grinding of gear teeth B23F, of screw-threads B23G1/36; by electro-erosion B23H; abrasive or related blasting B24C; tools for grinding, buffing or sharpening B24D; polishing compositions C09G1/00; abrasives C09K3/14; electrolytic etching or polishing C25F3/00; grinding arrangements for use on assembled railway tracks E01B31/17); DRESSING OR CONDITIONING OF ABRADING SURFACES; FEEDING OF GRINDING, POLISHING, OR LAPPING AGENTS
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A method includes, traversing a laser line scanning sensor over a workpiece to generate a series of scan data according to a first set of scan parameters; assembling the series of scan data into a virtual model; detecting a first hole, defining absence of scan data, in a first region of the virtual model; responsive to the first hole defining a dimension less than a threshold dimension, assigning the first set of scan parameters to the first region; detecting a second hole, in a second region of the virtual model; responsive to the second hole defining a dimension greater than the threshold dimension, defining a second set of scan parameters associated with an increased resolution and assigning the second set of scan parameters to the second workpiece region; and compiling the first and second set of scan parameters into a scan protocol defining a minimum scan cycle duration.

## Claims

### Claim 1

A method comprising:
during a first time interval:
during a first scan of a first workpiece approximating a workpiece geometry, driving a set of actuators to traverse a laser line scanning sensor along a scan axis over the first workpiece loaded into a work zone;
accessing a first series of scan data captured by the laser line scanning sensor, according to a first set of scan parameters, during the first scan;
assembling the first series of scan data into a first virtual model characterized by a first resolution;
accessing a baseline virtual model, of the first workpiece, characterized by a baseline resolution greater than the first resolution;
detecting a first virtual hole, defining absence of scan data, in a first region of the first virtual model;
in response to the first virtual hole defining a first dimension less than a threshold hole dimension, assigning the first set of scan parameters to a first workpiece region of the first workpiece;
detecting a second virtual hole, defining absence of scan data, in a second region of the first virtual model;
confirming absence of a real hole in a second workpiece region of the first workpiece, corresponding to the second virtual hole, based on the baseline virtual model;
in response to the second virtual hole defining a second dimension greater than the threshold hole dimension:
defining a second set of scan parameters associated with a second resolution greater than the first resolution and less than the baseline resolution; and
assigning the second set of scan parameters to the second workpiece region of the first workpiece; and
compiling the first set of scan parameters for the first workpiece region and the second set of scan parameters for the second workpiece region into a scan protocol;
during a second time interval shorter than the first time interval:
during a second scan of a second workpiece approximating the first workpiece geometry, driving the set of actuators to traverse the laser line scanning sensor along the scan axis over the second workpiece loaded into the work zone according to the scan protocol;
accessing a second series of scan data captured by the laser line scanning sensor, according to the first set of scan parameters while traversing the first workpiece region of the second workpiece;
accessing a third series of scan data captured by the laser line scanning sensor, according to the second set of scan parameters while traversing the second workpiece region of the second workpiece;
assembling the second series of scan data and the third series of scan data into a second virtual model of the second workpiece; and
generating a first toolpath representing a trajectory of a sanding head across the second workpiece based on the second virtual model.

### Claim 2 (depends on 1)

The method of claim 1:
wherein accessing the first series of scan data captured by the laser line scanning sensor according to the first set of scan parameters comprises, during the first scan:
accessing the first series of scan data captured by the laser line scanning sensor, according to a baseline set of scan parameters comprising:
a first orientation of the laser line scanning sensor along the scan axis; and
a first distance between the laser line scanning sensor and the first workpiece defining a first field of view area; and
wherein accessing the baseline virtual model, of the first workpiece, characterized by the baseline resolution greater than the first resolution comprises, during a baseline scan:
driving the set of actuators to traverse the laser line scanning sensor along the scan axis over the first workpiece loaded into the work zone;
accessing a baseline series of scan data captured by the laser line scanning sensor, according to a baseline set of scan parameters comprising:
the first orientation of the laser line scanning sensor along the scan axis; and
a baseline distance between the laser line scanning sensor and the first workpiece, less than the first distance, defining a baseline field of view area, less than the first field of view area; and
assembling the baseline series of scan data into the baseline virtual model.

### Claim 3 (depends on 1)

The method of claim 1:
wherein detecting the first virtual hole, defining absence of scan data, in the first region of the first virtual model comprises:
aligning the first virtual model and the baseline virtual model; and
detecting a difference of the first virtual model from the baseline virtual model defining an absence of data in the first virtual model, the data present in the baseline virtual model; and
wherein assigning the first set of scan parameters to the first workpiece region of the first workpiece comprises, in response to the first virtual hole defining the first dimension less than the threshold hole dimension:
filling the first virtual hole in the first virtual model with the data present in baseline virtual model data.

### Claim 4 (depends on 1)

The method of claim 1, wherein accessing the baseline virtual model of the first workpiece comprises:
accessing a computer-aided-design model defining a three-dimensional virtual representation of the first workpiece by:
accessing an identifier of the first workpiece;
accessing a database of a set of computer-aided-design models representing a set of workpieces; and
based on the identifier, accessing the computer-aided-design model of the set of computer-aided-design models corresponding to the first workpiece within the database.

### Claim 5 (depends on 1)

The method of claim 1:
further comprising:
segmenting the first virtual model into a set of virtual regions of the first virtual model, each virtual region of the set of virtual regions corresponding to a region of the first workpiece; and
for each virtual region in the set of virtual regions:
identifying a region characteristic exhibited by the region of the first workpiece corresponding to the virtual region; and
wherein detecting the first virtual hole in the first virtual region comprises:
aligning a first virtual region of the first virtual model with a corresponding virtual region of the baseline virtual model; and
detecting a difference between the first virtual region of the first virtual model and the corresponding virtual region of the baseline virtual model, the difference defining an absence of data in the first virtual model, the data present in the baseline virtual model.

### Claim 6 (depends on 5)

The method of claim 5:
wherein driving the set of actuators to traverse the laser line scanning sensor over the first workpiece comprises:
driving the set of actuators to traverse the laser line scanning sensor over an aircraft wing; and
wherein segmenting the first virtual model into the set of virtual regions comprises:
identifying a first region of the first workpiece characterized by a matte surface finish;
accessing a first subset of the first series of scan data corresponding to the first region of the first workpiece;
assembling the first subset of the first series of scan data into the first virtual region;
identifying a second region of the first workpiece characterized by a reflective surface;
accessing a second subset of the first series of scan data corresponding to the second region of the first workpiece;
assembling the second subset of the first series of scan data into a second virtual region;
identifying a third region of the first workpiece comprising an array of fastener bores;
accessing a third subset of the first series of scan data corresponding to the third region of the first workpiece; and
assembling the third subset of the first series of scan data into a third virtual region defining a virtual representation of the array of fastener bores.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
generating the third series of scan data of the first workpiece according to a third set of scan parameters, the third series of scan data interpolated from the first series of scan data and the second series of scan data; and
in response to the third series of scan data above a threshold error, compiling the third series of scan data into the scan protocol.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
detecting a first error metric, defining variability of the first series of scan data, in a third region of the first virtual model;
in response to the first error metric of the third region less than an error threshold:
assigning the first set of scan parameters to a third workpiece region of the first workpiece;
detecting a second error metric, defining variability of the first series of scan data, in a fourth region of the first virtual model; and
in response to the second error metric of the fourth region greater than the error threshold:
assigning the second set of scan parameters to a fourth workpiece region of the first workpiece.

### Claim 9 (depends on 1)

The method of claim 1:
wherein driving the set of actuators to traverse the laser line scanning sensor along the scan axis over the first workpiece loaded into the work zone during the first scan comprises:
driving the set of actuators to traverse the laser line scanning sensor along the scan axis through an environment including the work zone and defining external lighting characteristics;
wherein accessing the baseline virtual model comprises accessing a computer-aided-design model defining a three-dimensional virtual representation of the first workpiece; and
further comprising:
assembling the first series of scan data into a virtual environment approximating the external lighting characteristics of the environment including the work zone, based on the baseline virtual model;
accessing the second series of scan data captured by the laser line scanning sensor along the scan axis over the first workpiece, according to the second set of scan parameters;
simulating a virtual scan of the first workpiece defining the second set of scan parameters within the virtual environment to derive a first series of virtual scan data;
in response to the first series of virtual scan data different from the second series of scan data:
modifying the virtual environment; and
in response to the first series of virtual scan data approximating the second series of scan data:
verifying the virtual environment; and
simulating a virtual scan of the first workpiece in virtual environment according to a third set of test parameters.

### Claim 10 (depends on 1)

The method of claim 1:
further comprising, during the second scan of the second workpiece:
executing the scan protocol by:
driving the set of actuators to traverse the first workpiece region of the second workpiece according to the first set of scan parameters to generate a first minimal set of data representing the first workpiece region; and
driving the set of actuators to traverse the second workpiece region of the second workpiece according to the second set of scan parameters to generate a second minimal set of data representing the second workpiece region; and
wherein compiling the first set of scan parameters for the first workpiece region and the second set of scan parameters for the second workpiece region into the scan protocol comprises:
compiling the first minimal set of data representing the first workpiece region into a first virtual region of a second virtual model; and
compiling the second minimal set of data representing the second workpiece region into a second virtual region of the second virtual model.

### Claim 11 (depends on 1)

The method of claim 1, further comprising, during a processing cycle:
accessing a nominal target force assigned to the first toolpath;
accessing a first sequence of force values output by a force sensor coupled to the sanding head; and
via the set of actuators:
navigating the sanding head across the second workpiece according to the first toolpath; and
based on the first sequence of force values, deviating the sanding head from the first toolpath to maintain forces, applied by the sanding head to the first workpiece, proximal the nominal target force.

### Claim 12 (depends on 11)

The method of claim 11:
wherein generating the first toolpath comprises:
defining the first toolpath comprising a first ordered sequence of keypoints located on the third virtual model; and
for each keypoint in the first ordered sequence of keypoints:
calculating a vector normal to the third virtual model at a location of the keypoint on the third virtual model; and
storing the vector in the keypoint; and
wherein navigating the sanding head across the first workpiece region according to the first toolpath and deviating the sanding head from the first toolpath comprises:
for a first keypoint in the first ordered sequence of keypoints, via the set of actuators:
locating the sanding head at a first position intersecting the first keypoint; and
aligning an axis of the sanding head to a first vector contained in the first keypoint; and
driving the sanding head, coaxial with the first vector, toward the first workpiece to match force values, in the first sequence of force values read from the force sensor to the nominal target force.

### Claim 13 (depends on 11)

The method of claim 11:
wherein generating the first toolpath comprises:
identifying the first workpiece region, defining a convex surface profile, in the geometry of the first workpiece; and
generating the first toolpath defining a first continuous path across the first workpiece region of the first workpiece; and
further comprising, during the processing cycle, detecting a first sequence of positions of the sanding head traversing the first workpiece region by:
detecting the first sequence of positions of a reference point while the sanding head traverses the first workpiece region, the reference point located on a sanding pad mounted to the sanding head and coaxial with an axis of rotation of the sanding head.

### Claim 14 (depends on 11)

The method of claim 11, further comprising:
during the processing cycle:
accessing a first sequence of force values output by a force sensor coupled to the sanding head;
via the set of actuators:
navigating the sanding head across the first workpiece region according to the first toolpath; and
based on the first sequence of force values, deviating the sanding head from the first toolpath to maintain forces of the sanding head on the first workpiece region proximal the first target force; and
detecting a first sequence of positions of the sanding head traversing the first workpiece region;
interpreting a first surface contour in the first workpiece region based on the first sequence of positions;
accessing a first dimensional tolerance assigned to the first target surface defined in the target model;
detecting a first difference between the first surface contour and a first target surface, corresponding to the first surface contour, defined in the target model;
in response to the first difference exceeding the first dimensional tolerance, generating a second toolpath for the first workpiece region; and
during a second processing cycle:
via the set of actuators:
navigating the sanding head across the first workpiece region according to the second toolpath.

### Claim 15 (depends on 11)

The method of claim 11:
wherein navigating the sanding head across the first workpiece region comprises navigating the sanding head across the first workpiece region via the set of actuators comprising:
a robotic arm supporting an end effector and the sanding head, the sanding head comprising an orbital sander; and
wherein accessing the first sequence of force values comprises accessing the first sequence of force values output by the force sensor arranged between the end effector and the sanding head.

### Claim 16

A method comprising:
during a setup period:
during a first scan of a first workpiece approximating a workpiece geometry, driving a set of actuators to traverse a laser line scanning sensor along a scan axis over the first workpiece loaded into a work zone;
accessing a first series of scan data captured by the laser line scanning sensor, according to a first set of scan parameters, during the first scan;
assembling the first series of scan data into a first virtual model characterized by a first resolution;
accessing a baseline virtual model, of the first workpiece, characterized by a baseline resolution greater than the first resolution;
detecting a first error metric, defining variability of the first series of scan data, in a first region of the first virtual model;
in response to the first error metric of the first region less than an error threshold, assigning the first set of scan parameters to a first workpiece region of the first workpiece;
detecting a second error metric, defining variability of the first series of scan data, in a second region of the first virtual model;
in response to the second error metric of the second region greater than the error threshold:
defining a second set of scan parameters associated with a second resolution greater than the first resolution and less than the baseline resolution; and
assigning the second set of scan parameters to a second workpiece region of the first workpiece;
compiling the first set of scan parameters for the first workpiece region and the second set of scan parameters for the second workpiece region into a scan protocol;
during a second scan of a second workpiece approximating the first workpiece geometry, driving the set of actuators to traverse the laser line scanning sensor along the scan axis over the second workpiece loaded into the work zone according to the scan protocol;
accessing a second series of scan data captured by the laser line scanning sensor, according to the first set of scan parameters while traversing the first workpiece region of the second workpiece;
accessing a third series of scan data captured by the laser line scanning sensor, according to the second set of scan parameters while traversing the second workpiece region of the second workpiece;
assembling the second series of scan data and the third series of scan data into a second virtual model of the second workpiece; and
generating a first toolpath for the second workpiece based on the second virtual model.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
detecting a first virtual hole, defining absence of scan data, in a third region of the first virtual model;
in response to the first virtual hole defining a first dimension less than a threshold hole dimension:
assigning the first set of scan parameters to a third workpiece region of the first workpiece represented by the third region of the first virtual model;
detecting a second virtual hole, defining absence of scan data, in a fourth region of the first virtual model;
confirming absence of a real hole in a fourth workpiece region of the first workpiece, corresponding to the second virtual hole, based on the baseline virtual model; and
in response to the second virtual hole defining a second dimension greater than the threshold hole dimension:
defining a third set of scan parameters associated with a second resolution greater than the first resolution and less than the baseline resolution; and
assigning the third set of scan parameters to a fourth workpiece region of the first workpiece represented by the third region of the first virtual model.

### Claim 18 (depends on 16)

The method of claim 16, further comprising:
detecting a first virtual hole, defining absence of scan data, in a third region of the first virtual model; and
in response to the first virtual hole defining a first dimension less than a threshold hole dimension:
interpolating scan data from the first series of scan data to generate a simulated depth datum within the first virtual hole.

### Claim 19 (depends on 16)

The method of claim 16:
further comprising, during a processing cycle:
accessing a nominal target force assigned to the first toolpath;
accessing a first sequence of force values output by a force sensor coupled to a sanding head; and
via the set of actuators:
navigating the sanding head across the second workpiece according to the first toolpath; and
based on the first sequence of force values, deviating the sanding head from the first toolpath to maintain forces, applied by the sanding head to the first workpiece, proximal the nominal target force.

### Claim 20

A method comprising:
during a setup period:
during a first scan of a first workpiece approximating a workpiece geometry, driving a set of actuators to traverse a laser line scanning sensor along a scan axis over the first workpiece loaded into a work zone;
accessing a first series of scan data captured by the laser line scanning sensor according to a first set of scan parameters, during the first scan;
assembling the first series of scan data into a first virtual model characterized by a first resolution;
accessing a baseline virtual model, of the first workpiece, characterized by a baseline resolution greater than the first resolution;
detecting a first virtual hole, defining absence of scan data, in a first virtual region of the first virtual model;
in response to the first virtual hole defining a first dimension less than a threshold hole dimension, assigning the first set of scan parameters to a first workpiece region of the first workpiece;
detecting a second virtual hole, defining absence of scan data, in a second virtual region of the first virtual model;
confirming absence of a real hole in a second workpiece region of the first workpiece, corresponding to the second virtual hole, based on the baseline virtual model;
in response to the second virtual hole defining a second dimension greater than the threshold hole dimension:
defining a second set of scan parameters associated with a second resolution greater than the first resolution and less than the baseline resolution; and
assigning the second set of scan parameters to the second workpiece region of the first workpiece;
compiling the first set of scan parameters for the first workpiece region and the second set of scan parameters for the second workpiece region into a scan protocol;
during a second scan of a second workpiece approximating the first workpiece geometry, driving the set of actuators to traverse the laser line scanning sensor along the scan axis over the second workpiece loaded into the work zone according to the scan protocol;
accessing a second series of scan data captured by the laser line scanning sensor, according to the first set of scan parameters while traversing the first workpiece region of the second workpiece;
accessing a third series of scan data captured by the laser line scanning sensor, according to the second set of scan parameters while traversing the second workpiece region of the second workpiece;
assembling the second series of scan data and the third series of scan data into a second virtual model of the second workpiece; and
generating a first toolpath for the second workpiece based on the second virtual model; and
during a processing cycle:
accessing a nominal target force assigned to the first toolpath;
accessing a first sequence of force values output by a force sensor coupled to a sanding head; and
via the set of actuators:
navigating the sanding head across the second workpiece according to the first toolpath; and
based on the first sequence of force values, deviating the sanding head from the first toolpath to maintain forces, applied by the sanding head to the first workpiece, proximal the nominal target force.