# US 11,451,192 B2 · Automated photovoltaic plant inspection system and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,451,192 B2
- Document ID: US11451192B2
- Status: Granted
- Kind: B2
- Application number: 16/772,930
- Filed: 2017-12-14
- Published: 2021-06-03
- Granted: 2022-09-20
- Earliest claimed: 2017-12-14
- Assignees: Acciona Energia, S.A.
- Applicants: Acciona Energia S.A.
- Inventors: Eugenio Guelbenzu Michelena; Asun Padros Razquin; Miguel De La Parra Abad
- Web page: https://patentgrep.com/patent/US11451192B2

## CPC

- H02S 50/10: Testing of PV devices, e.g. of PV modules or single PV cells (testing of semiconductor devices during manufacturing {H10P74/00})
- H02S 50/15: using optical means, e.g. using electroluminescence
- G01N 25/72: Investigating presence of flaws
- G06T 7/0004: {Industrial image inspection}
- G06T 7/13: Edge detection
- G06T 2207/10024: Color image
- G06T 2207/10048: Infrared image
- G06T 2207/30108: Industrial image inspection
- Y02E 10/50: Photovoltaic [PV] energy
- H02S: GENERATION OF ELECTRIC POWER BY CONVERSION OF INFRARED RADIATION, VISIBLE LIGHT OR ULTRAVIOLET LIGHT, e.g. USING PHOTOVOLTAIC [PV] MODULES (obtaining electrical energy from radioactive sources G21H1/12; light sensitive inorganic semiconductor devices H10F; light sensitive organic semiconductor devices H10K30/00; thermoelectric devices H10N10/00; pyroelectric devices H10N15/00)
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- Y02E: REDUCTION OF GREENHOUSE GAS [GHG] EMISSIONS, RELATED TO ENERGY GENERATION, TRANSMISSION OR DISTRIBUTION

## Abstract

Systems and methods are provided for automatically inspecting photovoltaic (PV) installations. The system utilizes drones flying preprogrammed routes to conduct aerial thermography of PV modules for inspection (i.e., gathering IR images of the PV modules). The system conducts analytics on the gathered images to determine defects in the PV modules of the PV installation, classifies detected defects, issues reports automatically, and retains technical data of every module included in the database of a PV installation for trend analysis and preventative and predictive maintenance analysis.

## Claims

### Claim 1

A method of inspecting a photovoltaic (PV) installation of a facility for generating electricity, said method comprising:
receiving thermographic images of the PV installation at a processor, said PV installation comprising at least one array formed of a plurality of PV modules, wherein each module comprises a plurality of PV cells;
receiving, at the processor, a database having identification data for each module of the PV installation;
identifying, via the processor, each PV module in the acquired thermographic images of the PV installation, wherein identifying each PV module comprises:
processing each image, wherein processing each image comprises:
executing, via the processor, at least one accommodation algorithm on the image, wherein said executing the at least one accommodation algorithm on the image comprises:
executing the accommodation algorithm with each of a plurality of accommodation algorithm parameters to generate a set of accommodated images corresponding to the image; and
converting each accommodated image to a binary image to produce a set of binary images;
executing an edge detection algorithm on each binary image of the set of binary images to generate a set of edges associated with each of the plurality of accommodation algorithm parameters;
filtering non-trapezoidal shapes from the converted binary image;
calculating, via the processor, an average temperature of the at least one array; and
determining, via the processor a fault condition of at least one cell as a function of a difference between a temperature of the at least one cell and the average temperature of the array.

### Claim 2 (depends on 1)

The method of claim 1, wherein identifying each PV module further comprises:
processing each image, wherein processing each image further comprises:
identifying trapezoids within the sets of edges generated from all of the binary images of the set of binary images;
identifying at least one cluster of trapezoids in the identified trapezoids;
filtering the trapezoids in each identified cluster as a function of the surface area, orientation, and aspect ratio of each trapezoid to remove duplicate trapezoids and trapezoids exceeding a predetermined variance threshold for any of the surface area, orientation, and aspect ratio averages of the cluster to generate a tracker candidate cluster.

### Claim 3 (depends on 2)

The method of claim 2, wherein identifying each PV module further comprises:
processing each image, wherein processing each image further comprises:
determining whether each tracker candidate cluster is complete by identifying any trapezoids within any of the binary images of the set of binary images have trapezoids within the predetermined variance threshold for all of the surface area, orientation, and aspect ratio averages of the tracker candidate cluster; and
adding any identified trapezoids to the tracker candidate cluster.

### Claim 4 (depends on 3)

The method of claim 3, wherein identifying each PV module further comprises:
processing each image, wherein identifying the at least one cluster of trapezoids in the identified trapezoids includes identifying a plurality of clusters of trapezoids and filtering the trapezoids in each identified cluster results in generating a plurality of tracker candidate clusters, and wherein processing each image further comprises:
generating a set of generally horizontal lines by connecting the center of each trapezoid in each cluster to each horizontally adjacent trapezoid in the cluster, wherein each generally horizontal line is more horizontal than vertical with respect to the image;
generating a set of generally vertical lines by connecting the center of each trapezoid in each cluster to each vertically adjacent trapezoid in the cluster wherein each generally vertical line is more vertical than horizontal with respect to the image;
eliminating every cluster that corresponds to generally horizontal lines or generally vertical lines that do not match a plurality of generally horizontal lines and generally vertical lines generated for the image to identify a plurality of candidate modules, each corresponding to an intersection of the remaining generally horizontal lines and generally vertical lines; and
storing the trapezoid corresponding to each of the identified plurality of candidate modules.

### Claim 5 (depends on 4)

The method of claim 4, wherein identifying each PV module further comprises:
processing each image, wherein processing each image further comprises:
identifying the corners of each of the stored trapezoids corresponding to each of the identified plurality of candidate modules;
determining a homographic matrix to match the identified corners of each of the stored trapezoids to an ideal grid of predetermined proportions corresponding to the modules represented in the image;
translating, via the homographic matrix, the identified corners of each of the stored trapezoids to the ideal grid;
applying a compensation algorithm to the translated identified corners and the ideal grid to determine a position of the ideal grid with respect to the translated identified corners; and
translating, via an inverse of the homographic matrix the ideal grid in the determined position of the ideal grid to the image.

### Claim 6 (depends on 5)

The method of claim 5, wherein the compensation algorithm is one of a random sample consensus (RANSAC) algorithm and a least squares algorithm.

### Claim 7 (depends on 5)

The method of claim 5, wherein identifying each PV module further comprises:
processing each image, wherein processing each image further comprises:
optimizing the position of the translated ideal grid with respect to the image by iteratively;
moving the translated ideal grid one pixel in each direction up to a quantity of pixels equal to two times a quantity of the identified corners of each of the stored trapezoids corresponding to each of the identified plurality of candidate modules; and
applying a gradient algorithm to the moved grid to determine fit; and
selecting the position of the translated ideal grid with respect the image having the highest fit according to the applied gradient algorithm.

### Claim 8 (depends on 7)

The method of claim 7, wherein each image of the plurality of received thermographic images comprises image attributes comprising: latitude, longitude, altitude, pitch, roll, and direction.

### Claim 9 (depends on 8)

The method of claim 8, wherein identifying each PV module further comprises:
processing each image, wherein the grid identifies candidate modules in the database of modules, and wherein processing each image further comprises:
determining a tracker hypothesis as a function of the image attributes associated with the image and the database comprising identification data for each module of the PV installation, wherein said tracker hypothesis comprises a quantity of modules in rows and columns of a tracker in the image;
applying at least one of: row and column extension, row and column elimination, shadow limits, and distance limits, to the candidate modules identified by the grid to add or remove modules from the candidate modules identified by the grid and expand or reduce the grid as a function of the added or removed candidate modules; and
associating identification data from the database with each module identified by the reduced or expanded grid.

### Claim 10 (depends on 7)

The method of claim 7, wherein each image of the plurality of received thermographic images comprises image attributes comprising: a date and time, a corresponding Red Greed Blue (RGB) image, latitude, longitude, altitude, pitch, roll, and direction.

### Claim 11 (depends on 1)

The method of claim 1, wherein the average temperature of the array is determined, via the processor, by averaging a temperature value of each pixel of the thermographic images corresponding to the array.

### Claim 12 (depends on 11)

The method of claim 11, wherein:
the processor disregards any images in excess of one per panel; and
includes in the average temperature value only pixels corresponding to panels of the array.

### Claim 13 (depends on 1)

The method of claim 1, wherein:
the array comprises a plurality of rows of modules, each row of the array having a different vertical height than the other rows of the array;
the average temperature of the array is determined, via the processor, by averaging a temperature value of each pixel in the thermographic images corresponding to a given row of modules in the array; and
determining a fault condition comprises comparing the temperature of each cell in the array to the average temperature of the row to which the cell corresponds.

### Claim 14 (depends on 1)

The method of claim 1, wherein:
the array comprises a plurality of modules;
the average temperature of the array is determined, via the processor, by averaging a temperature value of each pixel in the thermographic images corresponding to a selected module in the array; and
determining a fault condition comprises comparing the temperature of each pixel in the image corresponding to the module to the average temperature; and
determining a hot spot in the selected module when a pixel exceeds the average temperature of the module by a predetermined minimum temperature threshold.

### Claim 15 (depends on 14)

The method of claim 14, wherein:
adjacent pixels determined as a hot spot are grouped together into a single hot spot; and
disregarding the single hot spot when a size of the single hot spot does not exceed a predetermined minimum size threshold.

### Claim 16 (depends on 14)

The method of claim 14, wherein:
processing the image comprises translating the pixels of the image of the module via a homographic matrix to an ideal grid; and
determining a fault condition comprises:
determining a faulty column when the translated pixels corresponding to each module in each column of the ideal grid exceed the average temperature by the predetermined threshold, wherein a top and bottom portion of the pixels in each column of the ideal grid are ignored; and
determining a defective band when a quantity of adjacent determined faulty columns exceeds a predetermined threshold width of the module.

### Claim 17 (depends on 16)

The method of claim 16 wherein:
the portion of the pixels in each column is 5% and the predetermined threshold width of the module is 18%.

### Claim 18

A system for automatically inspecting photovoltaic installations of a facility for generating electricity comprising:
a processor for receiving thermographic images of the photovoltaic installation, said photovoltaic installation comprising at least one array formed of a plurality of photovoltaic modules, wherein each module comprises a plurality of photovoltaic cells;
a database comprising identification data for each module of the photovoltaic installation;
wherein the processor is configured for:
identifying each photovoltaic module in the acquired thermographic images of the photovoltaic installation
processing each image of the acquired thermographic images, wherein processing each image comprises:
executing at least one accommodation algorithm on the image, wherein said accommodation algorithm comprises:
executing the accommodation algorithm with each of a plurality of accommodation algorithm parameters to generate a set of accommodated images corresponding to the image; and
converting each accommodated image to a binary image to produce a set of binary images;
wherein processing each image further comprises:
executing an edge detection algorithm on each binary image of the set of binary images to generate a set of edges associated with each of the plurality of accommodation algorithm parameters;
filtering non-trapezoidal shapes from the converted binary image;
calculating an average temperature of the at least one array; and
determining a fault condition of at least one cell as a function of a difference between a temperature of the at least one cell and the average temperature of the array.