# US 2024/0051152 A1 · Autonomous solar installation using artificial intelligence > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2024/0051152 A1 - Document ID: US20240051152A1 - Status: Published application - Kind: A1 - Application number: 18/232,981 - Filed: 2023-08-11 - Published: 2024-02-15 - Earliest claimed: 2022-08-11 - Assignees: The AES Corporation - Applicants: The AES Corporation - Inventors: Srikanth TADEPALLI; Sean OTTO; Diego BOTELHO; Deise Yumi ASAMI; Alexander AVERY; Jacob KIGGINS; John Christopher SHELTON - Web page: https://patentgrep.com/patent/US20240051152A1 ## CPC - B25J 9/1697: {Vision controlled systems} - B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor} - B25J 9/163: {learning, adaptive, model based, rule based expert control} - B25J 9/1679: {characterised by the tasks executed} - B25J 9/1687: {Assembly, peg and hole, palletising, straight line, weaving pattern movement} - B25J 11/008: {Manipulators for service tasks} - B25J 15/0052: {multiple gripper units or multiple end effectors} - B25J 15/0616: {with vacuum} - G05B 19/4155: characterised by program execution, i.e. part program or machine function execution, e.g. selection of a program - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06T 5/10: using non-spatial domain filtering - G06T 7/12: Edge-based segmentation - G06T 7/194: involving foreground-background segmentation - G06T 7/50: Depth or shape recovery - G06T 7/73: using feature-based methods - G06T 7/74: {involving reference images or patches} - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06V 10/82: using neural networks - G06V 20/50: Context or environment of the image - G06V 20/60: Type of objects - H02S 20/10: Supporting structures directly fixed to the ground (H02S20/30 takes precedence) - H02S 20/32: specially adapted for solar tracking - H02S 50/00: Monitoring or testing of PV systems, e.g. load balancing or fault identification - H02S 50/15: using optical means, e.g. using electroluminescence - G05B 2219/36439: Guide arm in path by slaving arm to projected path, beam riding - G05B 2219/39109: Dual arm, multiarm manipulation, object handled in cooperation - G05B 2219/40607: Fixed camera to observe workspace, object, workpiece, global - G06N 3/045: Combinations of networks - G06N 3/09: Supervised learning - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06T 2207/20061: Hough transform - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30184: Infrastructure - Y02E 10/50: Photovoltaic [PV] energy - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H02S: GENERATION OF ELECTRIC POWER BY CONVERSION OF INFRARED RADIATION, VISIBLE LIGHT OR ULTRAVIOLET LIGHT, e.g. USING PHOTOVOLTAIC [PV] MODULES (obtaining electrical energy from radioactive sources G21H1/12; light sensitive inorganic semiconductor devices H10F; light sensitive organic semiconductor devices H10K30/00; thermoelectric devices H10N10/00; pyroelectric devices H10N15/00) - Y02E: REDUCTION OF GREENHOUSE GAS [GHG] EMISSIONS, RELATED TO ENERGY GENERATION, TRANSMISSION OR DISTRIBUTION ## Abstract A system and method for installing solar panels are provided. The method obtains an image of a solar panel during an in-progress solar installation and estimates features of the solar panel based on a first image using distance simulation, geometric correction, and/or angular adjustment. The method also generates control signals, based on the estimated features, for operating a robotic controller for picking the solar panel. The method also obtains a second image of the solar panel when the solar panel is in a perspective view and detects placement of the solar panel based on the image by determining if the solar panel is co-planar with and at a predetermined offset from a fixed solar panel. Based on the detected placement, control signals are generated for operating a second robotic controller for aligning the solar panel with the fixed solar panel. ## Claims ### Claim 1 A method for autonomous solar installation, the method comprising: obtaining a first image of a solar panel during an in-progress solar installation; estimating a plurality of features of the solar panel based on the first image using distance simulation, geometric correction and angular adjustment; generating a first set of control signals, based on the estimated features, for operating a first robotic controller for picking the solar panel; obtaining a second image of the solar panel when the solar panel is in a perspective view; detecting placement of the solar panel based on the second image by determining if the solar panel is co-planar with and at a predetermined offset from a fixed solar panel; and generating a second set of control signals, based on the detected placement, for operating a second robotic controller for aligning the solar panel with the fixed solar panel. ### Claim 2 (depends on 1) The method of claim 1, wherein the plurality of features comprises center and a plurality of corners of the solar panel. ### Claim 3 (depends on 1) The method of claim 1, estimating the plurality of features of the solar panel further comprises: in accordance with a determination that an identifying label is visible in the first image: detecting a location of the identifying label; and using the location to estimate the plurality of features of the solar panel. ### Claim 4 (depends on 1) The method of claim 1, wherein estimating the plurality of features of the solar panel comprises: identifying and plotting, based on the first image, a longest visible horizontal line and a longest visible vertical line signifying edges of the solar panel; for each point in the first image that is internal to the solar panel in the first image, calculating horizontal and vertical distance from the respective point to the longest visible horizontal line and the longest visible vertical line, respectively; and identifying, based on the calculated horizontal and vertical distances, a subset of points for which: (i) horizontal distance is greater than a first predetermined percentage of width of the solar panel, and (ii) vertical distance is greater than a second predetermined percentage of height of the solar panel; calculating horizontal angles of intersection and vertical angles of intersection from each point in the subset of points with the longest visible horizontal line and the longest visible vertical line, respectively; correcting the calculated horizontal and vertical distances based on the calculated horizontal angles of intersection and vertical angles of intersection, to obtain corrected horizontal and vertical distances; identifying, based on the corrected horizontal and vertical distances, a candidate set of points for which: (i) horizontal distance is greater than a first predetermined percentage of width of the solar panel, and (ii) vertical distance is greater than a second predetermined percentage of height of the solar panel; computing a centroid of the candidate set of points to obtain a center of the solar panel; estimating candidate corners of the solar panel using Pythagorean theorem based on the centroid and the candidate set of points; and correcting a distance for the candidate corners based on the calculated horizontal and vertical angles of intersection to obtain final corners of the solar panel. ### Claim 5 (depends on 4) The method of claim 4, wherein the identifying and plotting are performed using Hough transform. ### Claim 6 (depends on 4) The method of claim 4, further comprising: determining if a point is internal to the solar panel by determining if a ray that originates from the point intersects the longest visible horizontal line and the longest visible vertical line in odd number of points. ### Claim 7 (depends on 1) The method of claim 1, wherein detecting placement of the solar panel comprises: obtaining a third image of the fixed solar panel when the fixed solar panel is in a perspective view; identifying two centers including (i) a center of the solar panel when the solar panel is floating based on the second image and (ii) a center of the fixed solar panel based on the third image; drawing a line between the two centers as a function of time; establishing Euler angles and radius vector at each point in time; generating one or more control signals, for operating the first robotic controller, to move the solar panel such that 5 degrees of freedom approach a numerical value of 0; and confirming coplanarity of points and Hough lines of the solar panel and the fixed solar panel, using computer vision. ### Claim 8 (depends on 7) The method of claim 7, wherein generating the one or more control signals comprises: generating control signals to slide the solar panel when the solar panel is floating such that distance between the two centers is approximately equal to a predetermined safe offset distance plus widths of each panel. ### Claim 9 (depends on 1) The method of claim 1, further comprising: obtaining a fourth image of the solar panel in a second perspective view or in a top view; converting the fourth image to grayscale to obtain a grayscale image; denoising and applying a bilateral filter to the grayscale image to obtain a processed image; eroding and thresholding the processed image to obtain a candidate image; drawing convex hulls for the candidate image; determining external contours above a predetermined length based on the convex hulls; dilating edges for the candidate image; blurring and applying Canny edge detection for the candidate image; testing for convex hull in orthogonal direction; and in accordance with a determination that multiple convex hulls are created and orthogonality is observed, detecting a supporting mechanical equipment. ### Claim 10 (depends on 9) The method of claim 9, wherein the supporting mechanical equipment includes a clamp or a fan gear. ### Claim 11 (depends on 9) The method of claim 9, wherein the supporting mechanical equipment is a clamp, and wherein the method further comprises: detecting a location of a clamp; and assisting an operator with movement of the clamp, based on the detected placement of the solar panel, in order to protect the solar panel from impact with the clamp. ### Claim 12 (depends on 9) The method of claim 9, wherein the supporting equipment is a fan gear, and wherein the method further comprises: detecting a location of a fan gear; and assisting an operator with safe placement of the solar panel, based on the detected placement of the solar panel, in order to guard the solar panel from collision. ### Claim 13 (depends on 1) The method of claim 1, wherein the solar panel and the fixed solar panel are substantially rectangular in shape. ### Claim 14 (depends on 1) The method of claim 1, wherein the solar panel and the fixed solar panel are substantially similar in shape. ### Claim 15 (depends on 1) The method of claim 1, wherein the solar panel includes a reflective material that causes the solar panel to reflect sunlight, and the first image or the second image includes dark white spots. ### Claim 16 (depends on 1) The method of claim 1, wherein the first image includes a view of the solar panel and background. ### Claim 17 (depends on 1) The method of claim 1, further comprising: prior to estimating the plurality of areas/features of the solar panel: extracting sun's relative position in the sky using solar physics, sun's elevation, and azimuth; estimating a glare based on the estimation of the sun's position; and using high fidelity noise cancellation and image correction algorithm to false color the glare. ### Claim 18 (depends on 1) The method of claim 1, further comprising: prior to estimating the plurality of areas/features of the solar panel: using image masking and segmentation or filtering or image correction techniques to remove sun's glare on the solar panel. ### Claim 19 (depends on 1) The method of claim 1, wherein the first set of control signals causes the first robotic controller to move the picked solar panel towards the fixed solar panel. ### Claim 20 (depends on 1) The method of claim 1, wherein determining if the solar panel is co-planar with and at a predetermined offset from the fixed solar panel comprises: determining if (i) the solar panel is on a same plane as the fixed solar panel and (ii) the solar panel and the fixed panel are near flush in a lateral direction. ### Claim 21 (depends on 6) The method of claim 6, wherein detecting placement of the solar panel comprises: obtaining a third image of the fixed solar panel when the fixed solar panel is in a perspective view; identifying two centers including (i) a center of the solar panel when the solar panel is floating based on the second image and (ii) a center of the fixed solar panel based on the third image; drawing a line between the two centers as a function of time; establishing Euler angles and radius vector at each point in time; generating one or more control signals, for operating the first robotic controller, to move the solar panel such that 5 degrees of freedom approach a numerical value of 0; and confirming coplanarity of points and Hough lines of the solar panel and the fixed solar panel, using computer vision, wherein generating the one or more control signals comprises: generating control signals to slide the solar panel when the solar panel is floating such that distance between the two centers is approximately equal to a predetermined safe offset distance plus widths of each panel, wherein the method further comprises: obtaining a fourth image of the solar panel in a second perspective view or in a top view; converting the fourth image to grayscale to obtain a grayscale image; denoising and applying a bilateral filter to the grayscale image to obtain a processed image; eroding and thresholding the processed image to obtain a candidate image; drawing convex hulls for the candidate image; determining external contours above a predetermined length based on the convex hulls; dilating edges for the candidate image; blurring and applying Canny edge detection for the candidate image; testing for convex hull in orthogonal direction; and in accordance with a determination that multiple convex hulls are created and orthogonality is observed, detecting a supporting mechanical equipment, and wherein the method further comprises: prior to estimating the plurality of areas/features of the solar panel: extracting sun's relative position in the sky using solar physics, sun's elevation, and azimuth; estimating a glare based on the estimation of the sun's position; and using high fidelity noise cancellation and image correction algorithm to false color the glare. ### Claim 22 A method for autonomous solar installation, the method comprising: obtaining a first image of a solar panel in a staging area using a viewing camera; estimating a plurality of areas/features of the solar panel based on the first image using at least one or distance simulation, geometric correction and angular adjustment; generating a first set of control signals for operating a first robotic controller for picking the solar panel, wherein the first set of control signals are based on one or more of the estimated plurality of areas/features; obtaining a second image of the solar panel when the solar panel is picked and is in a perspective orientation relative to the viewing camera; detecting an orientation in space of the picked solar panel based on the second image; and generating a second set of control signals, based on the detected orientation, for operating a second robotic controller for moving the picked the solar panel to an installation position, wherein the installation position aligns the picked solar panel to a previously installed solar panel by determining if the picked solar panel is co-planar with and at a predetermined offset from the previously installed solar panel.