# US 11,861,276 B2 · Locating a center point of a lug nut for an automated vehicle wheel removal system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,861,276 B2 - Document ID: US11861276B2 - Status: Granted - Kind: B2 - Application number: 17/975,910 - Filed: 2022-10-28 - Published: 2023-05-04 - Granted: 2024-01-02 - Earliest claimed: 2021-10-28 - Assignees: RoboTire, Inc. - Applicants: RoboTire, Inc. - Inventors: Bradley Vargo; Keegan Elliott; William Mapes; Scott Coburn; Riyadir Alalami; Anoopkumar Sandip Sonar - Web page: https://patentgrep.com/patent/US11861276B2 ## CPC - B25J 9/1687: {Assembly, peg and hole, palletising, straight line, weaving pattern movement} - G06F 30/27: using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model - B60B 7/068: {to the wheel bolts or wheel nuts} - G05B 15/02: electric - B60B 3/16: by bolts or the like - B60B 29/00: Apparatus or tools for mounting or dismounting wheels {(mounting of wheels at assembly lines B62D65/12)} - G05B 2219/40298: Manipulator on vehicle, wheels, mobile - G05B 2219/45021: Wheel mounting - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - B60B: VEHICLE WHEELS (making wheels or wheel parts by rolling B21H1/00, by forging, hammering or pressing B21K1/28); CASTORS; AXLES FOR WHEELS OR CASTORS; INCREASING WHEEL ADHESION - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract Described is a system (and method) for locating a center point of a lug nut for an automated vehicle wheel removal system. To improve the accuracy of the center point, the system may perform machine learning inferences using two-dimensional (2D) and three-dimensional (3D) image data. The system may process a 2D image to infer an initial center point, and potentially improve the accuracy by leveraging a 3D image. More particularly, the system may process a 3D image to infer a location of one or more edges (or edge points) around the perimeter of the lug nut and measure a set of distances between the initial center point and the located set of edges. The system may then refine (or adjust) the center point based on such measurements. ## Claims ### Claim 1 A system for locating a center of a lug nut comprising: one or more processors; and a non-transitory computer-readable medium storing a plurality of instructions, which when executed, cause the one or more processors to: obtain a two-dimensional image of a vehicle wheel, the vehicle wheel including a set of lug nuts; determine an initial center point of at least a first lug nut by processing the two-dimensional image using a first machine learning model; obtain a three-dimensional image of at least a portion of the vehicle wheel including the first lug nut; locate a set of edges at various points around a perimeter of the first lug nut by processing the three-dimensional image using a second machine learning model; determine a set of distances between a point within the three-dimensional image that corresponds to the initial center point and the located set of edges; update the initial center point of the first lug nut based on the determined set of distances; and map a set of two-dimensional coordinates representing the initial center point to a set of three-dimensional coordinates representing the point within the three-dimensional image that corresponds to the initial center point. ### Claim 2 (depends on 1) The system of claim 1, wherein updating the initial center point of the first lug nut based on the determined set of distances includes: determining an average distance of the determined set of distances; and updating the center point such that a distance from the updated center point to each of the set of edges corresponds to the determined average distance. ### Claim 3 (depends on 1) The system of claim 1, wherein locating the set of edges includes: determining a confidence score for each edge of the set of edges; and retaining only those edges satisfying a predetermined confidence score as part of the located set of edges. ### Claim 4 (depends on 1) The system of claim 1, wherein locating the set of edges includes: determining a change of depth between a plurality of points proximate to the various points around the perimeter of the first lug nut; and locating the set of edges at the various points around the perimeter of the first lug nut based on the determined change of depth. ### Claim 5 (depends on 1) The system of claim 1, wherein the plurality of instructions, when executed, further cause the one or more processors to: direct a robotic apparatus to maneuver to a position based on the updated center point of the first lug nut as part of a sequence of operations to remove the first lug nut from the vehicle wheel. ### Claim 6 (depends on 1) The system of claim 1, wherein the plurality of instructions, when executed, further cause the one or more processors to: determine one or more dimensions of the first lug nut based on the updated center point and the determined set of distances. ### Claim 7 (depends on 6) The system of claim 6, wherein the plurality of instructions, when executed, further cause the one or more processors to: determine a lug nut size based on the determined one or more dimensions of the first lug nut; and direct a robotic apparatus to select a socket corresponding the determined lug nut size as part of a sequence of operations to remove the first lug nut from the vehicle wheel. ### Claim 8 A computer-implemented method for locating a center of a lug nut comprising: obtaining a two-dimensional image of a vehicle wheel, the vehicle wheel including a set of lug nuts; determining an initial center point of at least a first lug nut by processing the two-dimensional image using a first machine learning model; obtaining a three-dimensional image of at least a portion of the vehicle wheel including the first lug nut; locating a set of edges at various points around a perimeter of the first lug nut by processing the three-dimensional image using a second machine learning model; determining a set of distances between a point within the three-dimensional image that corresponds to the initial center point and the located set of edges; updating the initial center point of the first lug nut based on the determined set of distances; and mapping a set of two-dimensional coordinates representing the initial center point to a set of three-dimensional coordinates representing the point within the three-dimensional image that corresponds to the initial center point. ### Claim 9 (depends on 8) The method of claim 8, wherein updating the initial center point of the first lug nut based on the determined set of distances includes: determining an average distance of the determined set of distances; and updating the center point such that a distance from the updated center point to each of the set of edges corresponds to the determined average distance. ### Claim 10 (depends on 8) The method of claim 8, wherein locating the set of edges includes: determining a confidence score for each edge of the set of edges; and retaining only those edges satisfying a predetermined confidence score as part of the located set of edges. ### Claim 11 (depends on 8) The method of claim 8, wherein locating the set of edges includes: determining a change of depth between a plurality of points proximate to the various points around the perimeter of the first lug nut; and locating the set of edges at the various points around the perimeter of the first lug nut based on the determined change of depth. ### Claim 12 (depends on 8) The method of claim 8, further comprising: directing a robotic apparatus to maneuver to a position based on the updated center point of the first lug nut as part of a sequence of operations to remove the first lug nut from the vehicle wheel. ### Claim 13 A non-transitory computer-readable medium storing instructions which, when executed by one or more processors of a system, cause the system to perform operations comprising: obtaining a two-dimensional image of a vehicle wheel, the vehicle wheel including a set of lug nuts; determining an initial center point of at least a first lug nut by processing the two-dimensional image using a first machine learning model; obtaining a three-dimensional image of at least a portion of the vehicle wheel including the first lug nut; locating a set of edges at various points around a perimeter of the first lug nut by processing the three-dimensional image using a second machine learning model; determining a set of distances between a point within the three-dimensional image that corresponds to the initial center point and the located set of edges; and updating the initial center point of the first lug nut based on the determined set of distances and mapping a set of two-dimensional coordinates representing the initial center point to a set of three-dimensional coordinates representing the point within the three-dimensional image that corresponds to the initial center point. ### Claim 14 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein updating the initial center point of the first lug nut based on the determined set of distances includes: determining an average distance of the determined set of distances; and updating the center point such that a distance from the updated center point to each of the set of edges corresponds to the determined average distance. ### Claim 15 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein locating the set of edges includes: determining a confidence score for each edge of the set of edges; and retaining only those edges satisfying a predetermined confidence score as part of the located set of edges. ### Claim 16 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein locating the set of edges includes: determining a change of depth between a plurality of points proximate to the various points around the perimeter of the first lug nut; and locating the set of edges at the various points around the perimeter of the first lug nut based on the determined change of depth. ### Claim 17 (depends on 13) The non-transitory computer-readable medium of claim 13, storing further instructions which, when executed by the one or more processors of the system, cause the system to perform further operations comprising: directing a robotic apparatus to maneuver to a position based on the updated center point of the first lug nut as part of a sequence of operations to remove the first lug nut from the vehicle wheel.