# US 12,007,220 B2 · Optical structural health monitoring

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,007,220 B2
- Document ID: US12007220B2
- Status: Granted
- Kind: B2
- Application number: 16/853,591
- Filed: 2020-04-20
- Published: 2020-10-29
- Granted: 2024-06-11
- Earliest claimed: 2017-12-29
- Applicants: John Tyson, II
- Inventors: John Tyson, II
- Web page: https://patentgrep.com/patent/US12007220B2

## CPC

- B64F 5/60: Testing or inspecting aircraft components or systems
- G01B 11/16: for measuring the deformation in a solid, e.g. optical strain gauge
- B25J 15/0019: {End effectors other than grippers}
- B25J 19/023: {including video camera means}
- G01B 11/165: {by means of a grating deformed by the object}
- G01J 5/00: Radiation pyrometry, e.g. infrared or optical thermometry
- G01M 5/0016: {of aircraft wings or blades}
- G01M 5/0033: {by determining damage, crack or wear}
- G01M 5/0041: {by determining deflection or stress}
- G01M 5/0091: {by using electromagnetic excitation or detection}
- G06K 19/06037: {multi-dimensional coding}
- G06T 7/001: {using an image reference approach}
- B64U 80/25: for recharging batteries; for refuelling
- B64U 2101/10: for generating power to be supplied to a remote station, e.g. UAVs with solar panels
- B64U 2101/30: for imaging, photography or videography
- B64U 2201/10: autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
- B64U 2201/202: using tethers for connecting to ground station
- G01J 5/48: Thermography; Techniques using wholly visual means
- G01J 2005/0077: {Imaging}
- G06T 2207/10064: Fluorescence image
- G06T 2207/30108: Industrial image inspection
- G06T 2207/30156: Vehicle coating
- G06T 2207/30204: Marker
- B64F: GROUND OR AIRCRAFT-CARRIER-DECK INSTALLATIONS SPECIALLY ADAPTED FOR USE IN CONNECTION WITH AIRCRAFT; DESIGNING, MANUFACTURING, ASSEMBLING, CLEANING, MAINTAINING OR REPAIRING AIRCRAFT, NOT OTHERWISE PROVIDED FOR; HANDLING, TRANSPORTING, TESTING OR INSPECTING AIRCRAFT COMPONENTS, NOT OTHERWISE PROVIDED FOR
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G01J: MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY (light sources F21, H01J, H01K, H05B; investigating properties of materials by optical means G01N)
- G01M: TESTING STATIC OR DYNAMIC BALANCE OF MACHINES OR STRUCTURES; TESTING OF STRUCTURES OR APPARATUS, NOT OTHERWISE PROVIDED FOR
- G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR

## Abstract

Methods, systems and devices for optical structural health monitoring of a subject, including the utilization of pointillism to provide a design or painting of the surface of the structure to be monitored, which also serves as a mechanism for conducting SHM, implementing digital image correlation (DIC) by applying a pattern comprising a random dot pattern and/or codes. The subject is imaged using imaging equipment to capture images of the pattern. For some applications, the captured images of the pattern and codes are stored in a CAD file that represents the subject or portion thereof to which the pattern and codes are applied, and includes the locations of the pattern and codes. Indicia applied to a structure may be applied using a paintjet or inkjet, or robotic mechanism, while some applications implement pre-patterning of a composite sheet that is used to form the structure.

## Claims

### Claim 1

A method for conducting structural health monitoring of a substrate, comprising:
a) designating a substrate or portion thereof to be monitored;
b) applying a pattern on the substrate or portion thereof that comprises one or more of a design element or background provided on one or more locations on the surface of the substrate or portion thereof;
c) imaging the pattern with equipment that captures the image of the pattern;
d) storing the image captured in step c);
e) wherein the pattern applied comprises a pattern of discrete elements which is applied to form the one or more of a design element or background by pointillism, and wherein the substrate or portion thereof is imaged with said imaging equipment to identify strains and deformations of the substrate at the one or more locations of the substrate or portion thereof where the said one or more pointillized design element or background is applied;
f) including determining from the imaging the presence of or the extent of strain or deformation of the substrate or an area of the substrate or portion thereof at the one or more locations of the substrate or portion thereof where the pointillized design element or background is applied, and
g) wherein determining the presence of or the extent of strain or deformation is carried out on a substrate that is or comprises an in service object.

### Claim 2 (depends on 1)

The method of claim 1, including determining from the imaging the presence of or the extent of strain or deformation of the substrate or an area of the substrate or portion thereof at the one or more locations of the substrate or portion thereof where the pointillized design element or background is applied, wherein said image captured of said pattern imaged is aligned to correspond with at least one or more locations of the structure, and wherein the method includes aligning a prior image of the substrate to correspond with said one or more substrate locations, and wherein said determining the presence of or the extent of strain or deformation of said substrate or said area of said substrate comprises comparing the imaged pattern with said prior imaged pattern.

### Claim 3 (depends on 2)

The method of claim 2, including storing the image captured in step c) in a database;
wherein steps c) and d) are repeated over time intervals, and wherein monitoring of one or more conditions of the substrate or portion thereof is carried out;
wherein said database comprises imaging results that include historical imaging results for said monitoring of said one or more conditions; and
monitoring one or more conditions by conducting Digital Image Correlation (DIC) or thermography NDT to compare one or more historical imaging results to a current imaging result; and
determining whether slight changes of the substrate or portion thereof, or a structure formed from said substrate have taken place;
wherein the historical imaging results for the substrate or portion thereof to be monitored are stored in a database, and wherein determining changes includes identifying slight changes of said captured image from the historical measurements of historical imaging results stored in said database, wherein said changes are determined by Digital Image Correlation (DIC) or thermography NDT.

### Claim 4 (depends on 3)

The method of claim 3, conditions by conducting Digital Image Correlation (DIC) or thermography NDT.

### Claim 5 (depends on 4)

The method of claim 4, using an existing pattern in the material or structure, such as the natural patterns in asphalt or concrete for measurement with DIC.

### Claim 6 (depends on 4)

The method of claim 4, using the natural thermal changes in a material structure for thermography NDT such as the solar warming of a bridge deck.

### Claim 7 (depends on 2)

The method of claim 2, wherein said structure comprises a composite structure, and wherein said pattern is applied to the composite structure during the construction process of said structure.

### Claim 8 (depends on 7)

The method of claim 7, wherein said composite structure is formed by a plurality of layers and wherein at least one layer includes said pattern.

### Claim 9 (depends on 8)

The method of claim 8, including pre-applying said pattern to said at least one layer that includes said pattern, and constructing said composite structure by attaching said at least one layer that includes the pattern thereon to at least one other layer, wherein said at least one layer that includes the pattern comprises an outermost layer, and wherein said at least one other layer includes a pattern thereon.

### Claim 10 (depends on 9)

The method of claim 9, wherein said at least one additional layer comprises an innermost layer of the composite structure.

### Claim 11 (depends on 8)

The method of claim 8, including pre-applying said pattern to said at least one layer that includes said pattern, and constructing said composite structure by attaching said at least one layer that includes the pattern thereon to at least one other layer, wherein said at least one layer that includes the pattern comprises an outermost layer or an innermost layer of the composite structure.

### Claim 12 (depends on 11)

The method of claim 11 wherein attaching said at least one layer that includes the pattern thereon to at least one other layer comprises autoclaving the layers together so that the said outermost or innermost layer having the pattern thereon comprises an outer surface of the composite structure.

### Claim 13 (depends on 12)

The method of claim 12, wherein said at least one layer that includes the pattern thereon comprises a thermoset epoxy layer.

### Claim 14 (depends on 13)

The method of claim 13, wherein the thermoset epoxy layer comprises urethane, and wherein the pattern is preprinted on a urethane layer as the first or last layer of a composite layup, such as graphite epoxy or fiberglass.

### Claim 15 (depends on 14)

The method of claim 14, wherein said epoxy layer comprising urethane is a film having the pattern pre-printed thereon.

### Claim 16 (depends on 1)

The method of claim 1, wherein the pattern is applied by applying a urethane composite film having the pattern pre-printed thereon.

### Claim 17 (depends on 1)

The method of claim 1, wherein the pattern comprises one or more of a pattern of random dots and one or more codes, wherein said one or more codes include unique QR codes to locate where on the structure the data is being taken, wherein said code comprises at least one QR code defining a QR code area of the pattern on the substrate on which the QR code is located, and wherein structural health monitoring uses said at least one QR code to identify a structural health condition within the area of the structure that is within the QR code area, and wherein historical data is recalled for a baseline comparison of said image of said pattern.

### Claim 18 (depends on 17)

The method of claim 17, wherein said pattern comprises an invisible pattern.

### Claim 19 (depends on 1)

The method of claim 1, wherein said pattern comprises an invisible pattern.

### Claim 20 (depends on 19)

The method of claim 19, wherein the invisible pattern comprises a pattern of a UV coating.

### Claim 21 (depends on 20)

The method of claim 20, wherein the UV coating is a UV paint, ink or dye.

### Claim 22 (depends on 20)

The method of claim 20, wherein the pattern is formed in the pattern areas by applying a UV paint.

### Claim 23 (depends on 1)

The method of claim 1, wherein said pattern applied to the substrate comprises a design, and wherein said design is formed by a plurality of spaced apart discrete points, dots or elements.

### Claim 24 (depends on 23)

The method of claim 23, wherein said plurality of spaced apart discrete points, dots or elements are applied on a field, wherein said plurality of spaced apart discrete points, dots or elements exhibit contrast from said field.

### Claim 25 (depends on 24)

The method of claim 24, wherein imaging includes filtering the image capture of said plurality of spaced apart discrete points, dots or elements applied on said field to increase contrast, wherein filtering includes placing an optical filter of a color that increases contrast between the discrete points, dots or elements applied and said field.

### Claim 26 (depends on 1)

The method of claim 1, wherein said pattern is provided on a sheet, and wherein said sheet is used to form said structure.

### Claim 27 (depends on 1)

The method of claim 1, wherein imaging the pattern with equipment that captures the image of the pattern is carried out using a robot.

### Claim 28 (depends on 27)

The method of claim 27, wherein imaging the pattern with equipment that captures the image of the pattern is carried out using a mobile robot to measure ever part of a structure or vehicle.

### Claim 29 (depends on 1)

The method of claim 1, wherein imaging the pattern with equipment that captures the image of the pattern is carried out using an inspection drone, the inspection drone being controllable or pre-programmed to image the substrate being monitored or a portion of the substrate, and implementing DIC to determine a structural health condition of the substrate, and wherein the method includes operating an inspection drone to image the pattern on the substrate.

### Claim 30 (depends on 1)

The method of claim 1, wherein imaging the pattern with equipment that captures the image of the pattern is carried out using a space-based inspection drone, the inspection drone being controllable or pre-programmed to image the substrate being monitored or a portion of the substrate, and implementing DIC to determine a structural health condition of the substrate, and wherein the method includes operating an inspection drone to image the pattern on the substrate.

### Claim 31 (depends on 1)

The method of claim 1, wherein applying the pattern to said substrate comprises placing a vinyl sheet or vinyl wrap as a template on said substrate, wherein said vinyl sheet or vinyl wrap contains openings corresponding to the pattern to be applied, and applying the coating over the vinyl sheet or wrap and through the openings.