# US 2024/0051146 A1 · Autonomous solar installation using artificial intelligence > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2024/0051146 A1 - Document ID: US20240051146A1 - Status: Published application - Kind: A1 - Application number: 18/232,965 - Filed: 2023-08-11 - Published: 2024-02-15 - Earliest claimed: 2022-08-11 - Assignees: The AES Corporation - Applicants: The AES Corporation - Inventors: Scott T. LUAN; Michael Jason PIPER; Bardh RUSHITI; Deise Yumi ASAMI; Alexander AVERY; Jacob KIGGINS; John Christopher SHELTON - Web page: https://patentgrep.com/patent/US20240051146A1 ## CPC - B25J 9/1697: {Vision controlled systems} - B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor} - B25J 9/163: {learning, adaptive, model based, rule based expert control} - B25J 9/1679: {characterised by the tasks executed} - B25J 9/1687: {Assembly, peg and hole, palletising, straight line, weaving pattern movement} - B25J 11/008: {Manipulators for service tasks} - B25J 15/0052: {multiple gripper units or multiple end effectors} - B25J 15/0616: {with vacuum} - G05B 19/4155: characterised by program execution, i.e. part program or machine function execution, e.g. selection of a program - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06T 5/10: using non-spatial domain filtering - G06T 7/12: Edge-based segmentation - G06T 7/194: involving foreground-background segmentation - G06T 7/50: Depth or shape recovery - G06T 7/73: using feature-based methods - G06T 7/74: {involving reference images or patches} - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06V 10/82: using neural networks - G06V 20/50: Context or environment of the image - G06V 20/60: Type of objects - H02S 20/10: Supporting structures directly fixed to the ground (H02S20/30 takes precedence) - H02S 20/32: specially adapted for solar tracking - H02S 50/00: Monitoring or testing of PV systems, e.g. load balancing or fault identification - H02S 50/15: using optical means, e.g. using electroluminescence - G05B 2219/36439: Guide arm in path by slaving arm to projected path, beam riding - G05B 2219/39109: Dual arm, multiarm manipulation, object handled in cooperation - G05B 2219/40607: Fixed camera to observe workspace, object, workpiece, global - G06N 3/045: Combinations of networks - G06N 3/09: Supervised learning - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06T 2207/20061: Hough transform - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30184: Infrastructure - Y02E 10/50: Photovoltaic [PV] energy - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H02S: GENERATION OF ELECTRIC POWER BY CONVERSION OF INFRARED RADIATION, VISIBLE LIGHT OR ULTRAVIOLET LIGHT, e.g. USING PHOTOVOLTAIC [PV] MODULES (obtaining electrical energy from radioactive sources G21H1/12; light sensitive inorganic semiconductor devices H10F; light sensitive organic semiconductor devices H10K30/00; thermoelectric devices H10N10/00; pyroelectric devices H10N15/00) - Y02E: REDUCTION OF GREENHOUSE GAS [GHG] EMISSIONS, RELATED TO ENERGY GENERATION, TRANSMISSION OR DISTRIBUTION ## Abstract A system and method for installing solar panels are provided. The method includes obtaining images of solar panels an installation structure during installation. The method also includes pre-processing the images by compensating for camera intrinsics or distortions, rectifying the images, and/or determining depth information. The method also includes detecting the solar panels by inputting the images into neural networks. The method also includes a first post-processing to compute a first panel pose based on an output of the neural networks. The method also includes generating control signals, based on the first panel pose, for operating a robotic controller for installing the solar panels. In some embodiments, the method also includes homography transforms to obtain a second panel pose, based on the first panel pose and visual patterns or fiducials on a solar panel, and generating the control signals further based on the second panel pose. ## Claims ### Claim 1 A method for autonomous solar panel installation, the method comprising: obtaining one or more images during installation, wherein the one or more images comprises an image of one or more solar panels and an installation structure; pre-processing the one or more images including one or more of compensating for camera intrinsics or distortions, rectifying the images, and determining depth information; detecting the one or more solar panels by inputting the one or more images into one or more neural networks that are trained to detect solar panels; a first post-processing to compute a first panel pose based on an output of the one or more neural networks; and generating control signals, based on the first panel pose, for operating a robotic controller for installing the one or more solar panels. ### Claim 2 (depends on 1) The method of claim 1, further comprising: a second post-processing comprising one or more homography transforms to obtain a second panel pose for the one or more solar panels, based on the first panel pose, wherein the second post-processing compensates or corrects for inaccuracies in the first panel pose based on visual patterns or fiducials on a solar panel, and wherein the control signals for operating the robotic controller for installing the one or more solar panels is further based on the second panel pose. ### Claim 3 (depends on 2) The method of claim 2, wherein the visual patterns comprise a grid pattern on the solar panel. ### Claim 4 (depends on 2) The method of claim 2, wherein the output of the one or more neural networks comprises panel segmentation, and wherein the second post-processing comprises: filtering out background in an image using the panel segmentation as a mask, to obtain a masked panel; identifying grid intersections in the masked panel as corners using a corner finding algorithm; computing a homography matrix H1 using the corners; determining locations of grid intersections in millimeter-space, based on H1; computing a homography matrix H2 to transform the grid intersections in image-space to their locations in the millimeter space; and back-projecting corners in the millimeter space to image-space, using inverse of H2. ### Claim 5 (depends on 4) The method of claim 4, wherein determining locations of the grid intersections comprises an association in H1-space based on Euclidean distance. ### Claim 6 (depends on 2) The method of claim 2, wherein the output of the one or more neural networks comprises panel segmentation, and wherein the second post-processing comprises: filtering out background in an image using the panel segmentation as a mask, to obtain a masked panel; identifying grid intersections in the masked panel in millimeter space using one or more artificial intelligence techniques; computing a homography matrix H1 based on the grid intersections; and back-projecting corners in the millimeter space to image-space, using inverse of H1. ### Claim 7 (depends on 2) The method of claim 2, wherein the second post-processing comprises: estimating four corners of a solar panel in an image-space ki; computing a homography matrix H1 that maps ki to a millimeter-space; identifying (i) pixel locations pi of panel features in image-space, and (ii) corresponding locations qi for the pixel locations pi in millimeter-space, based on H1; computing a homography matrix H2 that maps pi to qi; and back-projecting corners (0, 0), (H, 0), (H, W), (0, W) from millimeter-space to image-space based on H2−1. ### Claim 8 (depends on 1) The method of claim 1, wherein the one or more neural networks is trained to output bounding boxes, segmentation, keypoints, depth and/or a 6DoF pose. ### Claim 9 (depends on 1) The method of claim 1, wherein the pre-processing comprises compensating for a camera distortion, rectifying the image, and/or determining depth information based on a single-baseline stereo camera, a multi-baseline stereo camera, a time-of-flight sensor, or a LiDAR sensor. ### Claim 10 (depends on 1) The method of claim 1, wherein the first post-processing comprises one or more computer vision algorithms for processing the output of the one or more neural networks based on invariant structures in the images to determine locations of panel keypoints. ### Claim 11 (depends on 10) The method of claim 10, wherein the first post-processing further comprises solving for Perspective-n-Point based on panel dimensions and panel keypoints. ### Claim 12 (depends on 11) The method of claim 11, wherein the panel keypoints are four corners of the panel frame. ### Claim 13 (depends on 1) The method of claim 1, wherein the installation structure includes a torque tube and a clamp, and wherein the method further comprises a third post-processing comprising processing one or more images of the torque tube and/or clamp. ### Claim 14 (depends on 13) The method of claim 13, wherein the one or more images of the torque tube and/or clamp is obtained with a high-resolution camera and structured lighting. ### Claim 15 (depends on 14) The method of claim 14, wherein the structured lighting is a laser line that is approximately orthogonal or parallel with respect to the torque tube. ### Claim 16 (depends on 13) The method of claim 13, wherein the processing of the one or more images of the torque tube and/or clamp is performed by one or more neural networks and/or a computer vision pipeline. ### Claim 17 (depends on 13) The method of claim 13, further comprising locating a nut associated with the clamp by using high-intensity illumination and computer vision algorithms. ### Claim 18 (depends on 17) The method of claim 17, wherein high-intensity illumination is a ring light. ### Claim 19 A system for installing solar panels, the system comprising: a camera system for obtaining one or more images during installation, wherein the one or more images comprises an image of one or more solar panels and an installation structure; one or more devices for (i) pre-processing the one or more images including one or more of compensating for camera intrinsics or distortions, rectifying the images, and determining depth information, estimating panel poses for the one or more solar panels, based on the solar panel segments; (ii) detecting the one or more solar panels based on the one or more images; and (iii) a first post-processing to compute a first panel pose based on an output of the one or more neural networks; and a controller for generating control signals, based on the first panel pose, for operating a robotic controller for installing the one or more solar panels. ### Claim 20 (depends on 19) The system of claim 19, further comprising: the one or more devices for a second post-processing comprising one or more homography transforms to obtain a second panel pose for the one or more solar panels, based on the first panel pose, wherein the second post-processing compensates or corrects for inaccuracies in the first panel pose based on visual patterns or fiducials on a solar panel, and wherein the control signals for operating the robotic controller for installing the one or more solar panels is further based on the second panel pose.