# US 12,222,704 B2 · Robotic assisted tool and die making systems and methods > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,222,704 B2 - Document ID: US12222704B2 - Status: Granted - Kind: B2 - Application number: 17/874,775 - Filed: 2022-07-27 - Published: 2024-02-01 - Granted: 2025-02-11 - Earliest claimed: 2022-07-27 - Assignees: Ford Global Technologies, LLC - Applicants: Ford Global Technologies, LLC - Inventors: Raj Sohmshetty; Kyle Saul; Ramesh Parameswaran; Elizabeth Bullard - Web page: https://patentgrep.com/patent/US12222704B2 ## CPC - G05B 19/4155: characterised by program execution, i.e. part program or machine function execution, e.g. selection of a program - B25J 9/163: {learning, adaptive, model based, rule based expert control} - B25J 9/1679: {characterised by the tasks executed} - G05B 13/0265: {the criterion being a learning criterion} - G05B 2219/40269: Naturally compliant robot arm - G05B 2219/45031: Manufacturing semiconductor wafers - G05B 2219/45204: Die, mould making - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) ## Abstract A method for automating a die fabrication process includes obtaining, by one or more controllers, die sensor data from one or more die sensors and image data from one or more image sensors, generating, by the one or more controllers, a command to perform one or more machining tool operations based on the die sensor data and the image data, and controlling, by the one or more controllers, a machining tool to perform the one or more machining tool operations. The method includes selectively adjusting, by the one or more controllers, a position of a robotic arm based on the one or more machining tool operations and training an artificial intelligence system to autonomously perform the die fabrication process based on the one or more machining tool operations, the position of the robotic arm, the image data, the die sensor data, or a combination thereof. ## Claims ### Claim 1 A method for automating a die fabrication process, the method comprising: obtaining, by one or more controllers, die sensor data from one or more die sensors and image data from one or more image sensors; generating, by the one or more controllers, a command to perform one or more machining tool operations based on the die sensor data and the image data; controlling, by the one or more controllers, a machining tool to perform the one or more machining tool operations; selectively adjusting, by the one or more controllers, a position of a robotic arm based on the one or more machining tool operations; and training, by the one or more controllers, an artificial intelligence system to autonomously perform the die fabrication process based on the one or more machining tool operations, the position of the robotic arm, the image data, the die sensor data, or a combination thereof. ### Claim 2 (depends on 1) The method of claim 1, wherein the command is based on an input generated by a haptic glove. ### Claim 3 (depends on 1) The method of claim 1, wherein the die sensor data is indicative of a surface roughness of a die, one or more dimensional parameters of the die, or a combination thereof, and wherein the one or more die sensors include an optical sensor, a range sensor, a digital micrometer, or a combination thereof. ### Claim 4 (depends on 1) The method of claim 1 further comprising generating, by a display device, a graphical element based on the image data, the die sensor data, or a combination thereof, wherein the display device is one of an augmented reality display device and a virtual reality display device. ### Claim 5 (depends on 4) The method of claim 4, wherein the graphical element includes the image data that is overlayed with the die sensor data. ### Claim 6 (depends on 4) The method of claim 4, wherein: the graphical element includes a first graphical element associated with a first portion of a die; the graphical element includes a second graphical element associated with a second portion of the die; the first graphical element is indicative of the first portion of the die satisfying a die tolerance condition; and the second graphical element is indicative of the second portion of the die not satisfying the die tolerance condition. ### Claim 7 (depends on 1) The method of claim 1, wherein the artificial intelligence system is a reinforcement learning network. ### Claim 8 (depends on 7) The method of claim 7, wherein training the reinforcement learning network to autonomously perform the die fabrication process further comprises: generating a state vector based on the one or more machining tool operations, the position of the robotic arm, an input received by the one or more controllers, and the die sensor data; defining one or more actions associated with the state vector, wherein each action from among the one or more actions includes performing a machining tool operation, adjusting the position of the robotic arm, or a combination thereof; determining a reward for each of the one or more actions; and generating a plurality of entries of a state-action database based on the one or more actions and the reward associated with the one or more actions, wherein the reinforcement learning network is configured to autonomously perform the die fabrication process based on the plurality of entries. ### Claim 9 (depends on 1) The method of claim 1, wherein the artificial intelligence system is a convolutional neural network. ### Claim 10 (depends on 9) The method of claim 9, wherein training the convolutional neural network to autonomously perform the die fabrication process further comprises: generating a training image based on the one or more machining tool operations, the position of the robotic arm, the image data, and the die sensor data; labeling the training image based on a comparison between the die sensor data and a die tolerance condition; and selectively adjusting one or more parameters of the convolutional neural network based on the training image and one or more additional training images. ### Claim 11 A system for automating a die fabrication process, the system comprising: a processor; and a nontransitory computer-readable medium including instructions that are executable by the processor, wherein the instructions include: obtaining die sensor data from one or more die sensors and image data from one or more image sensors; generating a command to perform one or more machining tool operations based on the die sensor data and the image data; controlling a machining tool to perform the one or more machining tool operations; selectively adjusting a position of a robotic arm based on the one or more machining tool operations; and training an artificial intelligence system to autonomously perform the die fabrication process based on the one or more machining tool operations, the position of the robotic arm, the image data, the die sensor data, or a combination thereof, wherein the artificial intelligence system is one of a reinforcement learning network and a convolutional neural network. ### Claim 12 (depends on 11) The system of claim 11, wherein the command is based on an input generated by a haptic glove. ### Claim 13 (depends on 11) The system of claim 11, wherein the die sensor data is indicative of a surface roughness of a die, one or more dimensional parameters of the die, or a combination thereof, and wherein the one or more die sensors include an optical sensor, a range sensor, a digital micrometer, or a combination thereof. ### Claim 14 (depends on 11) The system of claim 11, wherein the instructions further comprise generating, by a display device, a graphical element based on the image data, the die sensor data, or a combination thereof, wherein the display device is one of an augmented reality display device and a virtual reality display device. ### Claim 15 (depends on 14) The system of claim 14, wherein the graphical element includes the image data that is overlayed with the die sensor data. ### Claim 16 (depends on 14) The system of claim 14, wherein: the graphical element includes a first graphical element associated with a first portion of a die; the graphical element includes a second graphical element associated with a second portion of the die; the first graphical element is indicative of the first portion of the die satisfying a die tolerance condition; and the second graphical element is indicative of the second portion of the die not satisfying the die tolerance condition. ### Claim 17 (depends on 11) The system of claim 11, wherein the artificial intelligence system is the reinforcement learning network, and wherein the instructions for training the reinforcement learning network to autonomously perform the die fabrication process further comprises: generating a state vector based on the one or more machining tool operations, the position of the robotic arm, an input received by the one or more controllers, and the die sensor data; defining one or more actions associated with the state vector, wherein each action from among the one or more actions includes performing a machining tool operation, adjusting the position of the robotic arm, or a combination thereof; determining a reward for each of the one or more actions; and generating a plurality of entries of a state-action database based on the one or more actions and the reward associated with the one or more actions, wherein the reinforcement learning network is configured to autonomously perform the die fabrication process based on the plurality of entries. ### Claim 18 (depends on 11) The system of claim 11, wherein the artificial intelligence system is a convolutional neural network, and wherein the instructions for wherein training the convolutional neural network to autonomously perform the die fabrication process further comprise: generating a training image based on the one or more machining tool operations, the position of the robotic arm, the image data, and the die sensor data; labeling the training image based on a comparison between the die sensor data and a die tolerance condition; and selectively adjusting one or more parameters of the convolutional neural network based on the training image and one or more additional training images. ### Claim 19 A method for automating a die fabrication process, the method comprising: obtaining, by one or more controllers, die sensor data from one or more die sensors and image data from one or more image sensors; generating, by the one or more controllers, a command to perform one or more machining tool operations based on the die sensor data and the image data; controlling, by the one or more controllers, a machining tool to perform the one or more machining tool operations; selectively adjusting, by the one or more controllers, a position of a robotic arm based on the one or more machining tool operations; and training, by the one or more controllers, an artificial intelligence system to autonomously perform the die fabrication process based on the one or more machining tool operations, the position of the robotic arm, the image data, the die sensor data, or a combination thereof, wherein the artificial intelligence system is one of a reinforcement learning network and a convolutional neural network. ### Claim 20 (depends on 19) The method of claim 19, wherein the artificial intelligence system is the reinforcement learning network, and wherein training the reinforcement learning network to autonomously perform the die fabrication process further comprises: generating a state vector based on the one or more machining tool operations, the position of the robotic arm, an input received by the one or more controllers, and the die sensor data; defining one or more actions associated with the state vector, wherein each action from among the one or more actions includes performing a machining tool operation, adjusting the position of the robotic arm, or a combination thereof; determining a reward for each of the one or more actions; and generating a plurality of entries of a state-action database based on the one or more actions and the reward associated with the one or more actions, wherein the reinforcement learning network is configured to autonomously perform the die fabrication process based on the plurality of entries.