# US 12,304,080 B2 · Method for autonomously repairing surface defects in a workpiece through surface modifications

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,304,080 B2
- Document ID: US12304080B2
- Status: Granted
- Kind: B2
- Application number: 18/379,130
- Filed: 2023-10-11
- Published: 2024-02-01
- Granted: 2025-05-20
- Earliest claimed: 2020-07-31
- Assignees: GrayMatter Robotics Inc.
- Applicants: GrayMatter Robotics Inc.
- Inventors: Avadhoot Ahire; YiWei Chen; Rishav Guha; Satyandra K. Gupta; Ariyan M. Kabir; Ashish Kulkarni; Ceasar Navarro; Sagar Panchal; Brual C. Shah
- Web page: https://patentgrep.com/patent/US12304080B2

## CPC

- B25J 9/1653: {parameters identification, estimation, stiffness, accuracy, error analysis}
- B24B 51/00: Arrangements for automatic control of a series of individual steps in grinding a workpiece (if applicable to other machine tools, G05B takes precedence)
- B25J 5/005: {mounted on endless tracks or belts}
- B25J 9/162: {Mobile manipulator, movable base with manipulator arm mounted on it}
- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1664: {characterised by motion, path, trajectory planning}
- B25J 9/1679: {characterised by the tasks executed}
- B25J 9/1684: {Tracking a line or surface by means of sensors}
- B25J 9/1697: {Vision controlled systems}
- B25J 11/0065: {Polishing or grinding}
- B25J 13/085: {Force or torque sensors (B25J13/082, B25J13/084 take precedence)}
- B25J 19/023: {including video camera means}
- G06T 1/0014: {Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023)}
- G06T 7/13: Edge detection
- G06T 7/60: Analysis of geometric attributes
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G05B 2219/45082: Sanding robot, to clean surfaces
- G06T 2207/10024: Color image
- G06T 2219/004: Annotating, labelling
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- B24B: MACHINES, DEVICES, OR PROCESSES FOR GRINDING OR POLISHING (grinding of gear teeth B23F, of screw-threads B23G1/36; by electro-erosion B23H; abrasive or related blasting B24C; tools for grinding, buffing or sharpening B24D; polishing compositions C09G1/00; abrasives C09K3/14; electrolytic etching or polishing C25F3/00; grinding arrangements for use on assembled railway tracks E01B31/17); DRESSING OR CONDITIONING OF ABRADING SURFACES; FEEDING OF GRINDING, POLISHING, OR LAPPING AGENTS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A method includes: compiling lower-resolution images, captured during a global scan cycle executed over a workpiece, into a virtual model; defining a nominal toolpath and a nominal target force for the workpiece based on a the virtual model; detecting a defect indicator on the workpiece based on the lower-resolution images; accessing a higher-resolution image captured during a local scan cycle over the defect indicator; characterizing the defect indicator as a defect reparable via material removal based on the higher-resolution image; defining a repair toolpath for the defect based on the virtual model; navigating a sanding head over the workpiece according to the repair toolpath to repair the defect; and, during a processing cycle: navigating the sanding head across the workpiece according to the nominal toolpath and deviating the sanding head from the nominal toolpath to maintain forces of the sanding head on the workpiece proximal the nominal target force.

## Claims

### Claim 1

A method comprising:
accessing a virtual model of a workpiece;
accessing a nominal toolpath for the workpiece;
accessing a nominal target force assigned to the nominal toolpath;
accessing a first set of images depicting the workpiece;
based on the first set of images, detecting a first defect in a first workpiece region of the workpiece;
in response to characterizing the first defect as repairable via material removal from the workpiece, generating a repair toolpath for the first workpiece region based on a first geometry of the first workpiece region represented in the virtual model;
during a repair cycle, via a set of actuators coupled to a sanding head, navigating the sanding head across the first workpiece region according to the repair toolpath; and
during a processing cycle:
accessing a first sequence of force values output by a force sensor coupled to the sanding head; and
via the set of actuators:
navigating the sanding head across the workpiece according to the nominal toolpath; and
based on the first sequence of force values, deviating the sanding head from the nominal toolpath to maintain forces, applied by the sanding head to the workpiece, proximal the nominal target force.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
based on the first defect, assigning a target repair force, greater than the nominal target force, to the repair toolpath; and
during the repair cycle:
accessing a second sequence of force values output by the force sensor; and
via the set of actuators:
based on the second sequence of force values, deviating the sanding head from the repair toolpath to maintain forces, applied by the sanding head to the workpiece, proximal the target repair force.

### Claim 3 (depends on 2)

The method of claim 2:
wherein navigating the sanding head across the first workpiece region according to the repair toolpath during the repair cycle comprises navigating the sanding head across the first workpiece region according to the repair toolpath during a first time period; and
wherein navigating the sanding head across the workpiece according to the nominal toolpath during the processing cycle comprises navigating the sanding head across the workpiece according to the nominal toolpath during a second time period succeeding the first time period.

### Claim 4 (depends on 1)

The method of claim 1:
wherein navigating the sanding head across the workpiece during the processing cycle comprises:
navigating the sanding head within a second workpiece region of the workpiece and toward the first workpiece region according to the nominal toolpath during a first time period; and
wherein accessing the first set of images comprises:
in response to the sanding head reaching the first workpiece region during the first time period:
pausing the processing cycle; and
navigating an end effector, supporting the sanding head and an optical sensor, over the first workpiece region to capture the first set of images via the optical sensor; and
further comprising:
in response to pausing the processing cycle:
initiating the repair cycle within the first workpiece region during a second time period succeeding the first time period; and
in response to completion of the repair cycle within the first workpiece region during the second time period:
resuming the processing cycle during a third time period succeeding the second time period.

### Claim 5 (depends on 1)

The method of claim 1:
wherein generating the repair toolpath comprises generating the repair toolpath defining a boustrophedonic geometry contained within the first workpiece region and spanning the first defect;
wherein navigating the sanding head across the first workpiece region during the repair cycle comprises navigating the sanding head within the first workpiece region;
wherein accessing the nominal toolpath for the workpiece comprises accessing the nominal toolpath for the workpiece spanning the first workpiece region and a second workpiece region of the workpiece; and
wherein navigating the sanding head across the workpiece during the processing cycle comprises navigating the sanding head across the first workpiece region and the second workpiece region.

### Claim 6 (depends on 1)

The method of claim 1:
further comprising:
accessing a nominal feed rate assigned to the nominal toolpath; and
accessing a repair feed rate, less than nominal feed rate, specified for defects;
wherein navigating the sanding head across the first workpiece region during the repair cycle comprises:
navigating the sanding head along the repair toolpath at the nominal feed rate; and
in response to the sanding head approaching the first defect, navigating the sanding head along the repair toolpath at the repair feed rate; and
wherein navigating the sanding head across the workpiece during the processing cycle comprises navigating the sanding head across the workpiece at the nominal feed rate.

### Claim 7 (depends on 1)

The method of claim 1:
further comprising navigating an optical sensor over and offset from the workpiece during a global scan cycle;
wherein accessing the first set of images depicting the workpiece comprises accessing the first set of images, characterized by a first resolution, captured by the optical sensor during the global scan cycle;
wherein accessing the virtual model of the workpiece comprises generating the virtual model of the workpiece based on the first set of images;
wherein accessing the nominal toolpath for the workpiece comprises generating the nominal toolpath based on a geometry of the workpiece represented in the virtual model; and
further comprising:
in response to detecting the first defect in the first workpiece region of the workpiece based on the first set of images, navigating the optical sensor over and offset from the first workpiece region of the workpiece during a local scan cycle;
accessing a second set of images, characterized by a second resolution greater than the first resolution, captured by the optical sensor during the local scan cycle; and
based on the second set of images, characterizing the first defect as repairable via material removal from the workpiece.

### Claim 8 (depends on 7)

The method of claim 7:
further comprising:
based on the first set of images, detecting a second defect indicator in a second workpiece region of the workpiece;
in response to detecting the second defect in the second workpiece region of the workpiece, navigating the optical sensor over and offset from the second workpiece region of the workpiece during a second local scan cycle;
accessing a third set of images, characterized by the second resolution, captured by the optical sensor during the second local scan cycle; and
based on the third set of images, characterizing the second defect as irreparable via material removal from the workpiece; and
in response to characterizing the second defect as irreparable via material removal from the workpiece:
generating a notification to manually repair the second defect in the second workpiece region; and
serving the notification to an operator; and
wherein generating the nominal toolpath comprises generating the nominal toolpath excluding the second workpiece region.

### Claim 9 (depends on 7)

The method of claim 7, further comprising, in response to characterizing the second defect as irreparable via material removal from the workpiece:
annotating the virtual model with a location of the second defect on the workpiece; and
rendering the virtual model on a display.

### Claim 10 (depends on 8)

The method of claim 8:
wherein characterizing the first defect as repairable via material removal from the workpiece comprises:
extracting a first set of features from a first image, in the second set of images, depicting the first workpiece region of the workpiece; and
characterizing the first defect as a paint sag based on the first set of features; and
wherein characterizing the second defect as irreparable via material removal from the workpiece comprises:
extracting a second set of features from a second image, in the third set of images, depicting the second workpiece region of the workpiece; and
characterizing the second defect as a gouge based on the second set of features.

### Claim 11 (depends on 1)

The method of claim 1:
wherein accessing the first set of images comprises accessing the first set of images comprising depth maps captured by a depth sensor navigated over the workpiece; and
wherein detecting the first defect comprises detecting the first defect based on a three-dimensional surface discontinuity, present in the first workpiece region of the workpiece, based on the first set of images.

### Claim 12 (depends on 1)

The method of claim 1:
wherein accessing the first set of images comprises accessing the first set of images comprising photographic images captured by a color camera sensor navigated over the workpiece; and
wherein detecting the first defect comprises detecting the first defect based on a two-dimensional color discontinuity detected in a first photographic image, in the first set of images, depicting the first workpiece region of the workpiece.

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
characterizing the first defect indicator as the first defect of a first defect type based on features detected in the second set of images;
annotating the virtual model with:
the first defect type at a location of the first defect; and
characteristics of the repair toolpath;
further comprising navigating an optical sensor over and offset from the workpiece during a scan cycle;
wherein accessing the first set of images depicting the workpiece comprises accessing the first set of images comprising photographic images captured by the optical sensor during the global scan cycle;
wherein detecting the first defect in the first workpiece region of the workpiece based on the first set of images comprises:
extracting a first set of features depicting the first workpiece region of the workpiece from the first set of images;
detecting the first defect based on the first set of features;
characterizing a first size of the first defect based on the first set of features; and
characterizing a first severity of the first defect proportional to the first size;
further comprising:
extracting a second set of features depicting a second workpiece region of the workpiece from the first set of images;
detecting a second defect in the second workpiece region of the workpiece based on the second set of features; and
characterizing a second size, less than the first size, of the second defect based on the second set of features; and
characterizing a second severity, less than the first severity, of the second defect proportional to the second size;
wherein generating the repair toolpath for the first workpiece region comprises generating the repair toolpath for the first workpiece region further in response to the first severity exceeding a threshold severity; and
wherein accessing the nominal toolpath for the workpiece comprises, in response to the second severity score falling below the threshold severity, generating the nominal toolpath, spanning the second workpiece region and a third workpiece region of the workpiece, based on a geometry of the workpiece represented in the virtual model.

### Claim 14

A method comprising:
accessing a virtual model of a workpiece;
accessing a first image depicting a first workpiece region of the workpiece;
based on the first image, detecting a first defect in the first workpiece region of the workpiece;
in response to characterizing the first defect as repairable via material removal from the workpiece, generating a first toolpath for the first workpiece region based on a first geometry of the first workpiece region represented in the virtual model;
accessing a second toolpath for a second workpiece region of the workpiece;
accessing a nominal target force assigned to the workpiece;
during a repair cycle, via a set of actuators coupled to a sanding head, navigating the sanding head across the first workpiece region according to the first toolpath; and
during a processing cycle:
accessing a sequence of force values output by a force sensor coupled to the sanding head; and
via the set of actuators:
navigating the sanding head across the second workpiece region of the workpiece according to the second toolpath; and
based on the sequence of force values, deviating the sanding head from the second toolpath to maintain forces, applied by the sanding head to the second workpiece region of the workpiece, proximal the nominal target force.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
based on the first defect, assigning a target repair force, greater than the nominal target force, to the repair toolpath; and
during the repair cycle:
accessing a second sequence of force values output by the force sensor; and
via the set of actuators:
based on the second sequence of force values, deviating the sanding head from the repair toolpath to maintain forces, applied by the sanding head to the workpiece, proximal the target repair force.

### Claim 16 (depends on 14)

The method of claim 14:
wherein navigating the sanding head across the second workpiece region during the processing cycle comprises:
navigating the sanding head within the second workpiece region of the workpiece and toward the first workpiece region according to the nominal toolpath during a first time period;
wherein accessing the first set of images comprises:
in response to the sanding head reaching the first workpiece region during the first time period:
pausing the processing cycle; and
navigating an end effector, supporting the sanding head and an optical sensor, over the first workpiece region to capture the first set of images via the optical sensor; and
further comprising:
in response to pausing the processing cycle:
initiating the repair cycle within the first workpiece region during a second time period succeeding the first time period; and
in response to completion of the repair cycle within the first workpiece region during the second time period:
resuming the processing cycle during a third time period succeeding the second time period.

### Claim 17 (depends on 14)

The method of claim 14:
further comprising navigating an optical sensor over and offset from the workpiece during a global scan cycle;
wherein accessing the first set of images depicting the workpiece comprises accessing the first set of images, characterized by a first resolution, captured by the optical sensor during the global scan cycle;
wherein accessing the virtual model of the workpiece comprises generating the virtual model of the workpiece based on the first set of images;
wherein accessing the nominal toolpath for the workpiece comprises generating the nominal toolpath based on a geometry of the workpiece represented in the virtual model; and
further comprising:
in response to detecting the first defect in the first workpiece region of the workpiece based on the first set of images, navigating the optical sensor over and offset from the first workpiece region of the workpiece during a local scan cycle;
accessing a second set of images, characterized by a second resolution greater than the first resolution, captured by the optical sensor during the local scan cycle; and
based on the second set of images, characterizing the first defect as repairable via material removal from the workpiece.

### Claim 18

A method comprising:
accessing a virtual model of a workpiece;
accessing a first image depicting a first workpiece region of the workpiece;
based on the first image, detecting a first defect in the first workpiece region of the workpiece;
in response to characterizing the first defect as repairable via material removal from the workpiece:
generating a first toolpath for the first workpiece region based on a first geometry of the first workpiece region represented in the virtual model; and
assigning a target repair force to the first toolpath;
accessing a second toolpath for a second workpiece region of the workpiece; and
during a processing cycle:
accessing a sequence of force values output by a force sensor coupled to a sanding head; and
via a set of actuators coupled to the sanding head:
navigating the sanding head across the first workpiece region of the workpiece according to the first toolpath; and
based on the sequence of force values, deviating the sanding head from the first toolpath to maintain forces, applied by the sanding head to the first workpiece region of the workpiece, proximal the target repair force; and
via the set of actuators, navigating the sanding head across the second workpiece region according to the second toolpath.

### Claim 19 (depends on 18)

The method of claim 18:
further comprising navigating an optical sensor over and offset from the workpiece during a global scan cycle;
wherein accessing the first set of images depicting the workpiece comprises accessing the first set of images, characterized by a first resolution, captured by the optical sensor during the global scan cycle; and
wherein accessing the virtual model of the workpiece comprises generating the virtual model of the workpiece based on the first set of images.

### Claim 20 (depends on 18)

The method of claim 18, wherein accessing the sequence of force values comprises accessing the sequence of force values, output by the force sensor, representing forces applied by the sanding head to the workpiece parallel to an axis of rotation of the sanding head.