# US 12,179,367 B2 · Translation and automation of protocols in a robotic lab > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,179,367 B2 - Document ID: US12179367B2 - Status: Granted - Kind: B2 - Application number: 17/392,096 - Filed: 2021-08-02 - Published: 2022-02-10 - Granted: 2024-12-31 - Earliest claimed: 2020-08-04 - Assignees: Artificial, Inc. - Applicants: Artificial, Inc. - Inventors: Jeff Washington; Geoffrey J. Budd; Nikhita Singh; Jake Sganga; Alexander Li Honda - Web page: https://patentgrep.com/patent/US12179367B2 ## CPC - B25J 9/1671: {characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems} - B25J 9/1605: {Simulation of manipulator lay-out, design, modelling of manipulator} - B25J 9/163: {learning, adaptive, model based, rule based expert control} - B25J 9/1653: {parameters identification, estimation, stiffness, accuracy, error analysis} - B25J 9/1656: {characterised by programming, planning systems for manipulators} - B25J 9/1658: {characterised by programming language} - B25J 9/1661: {characterised by task planning, object-oriented languages} - B25J 9/1679: {characterised by the tasks executed} - B25J 9/1692: {Calibration of manipulator} - B25J 9/1697: {Vision controlled systems} - B25J 13/089: {Determining the position of the robot with reference to its environment} - B25J 19/023: {including video camera means} - G01N 35/0099: {comprising robots or similar manipulators (robots per se B25J)} - G06F 3/04812: Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 40/10: Text processing (natural language analysis G06F40/20; semantic analysis G06F40/30; processing or translation of natural language G06F40/40) - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06N 20/00: Machine learning - G06T 7/50: Depth or shape recovery - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/73: using feature-based methods - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G01N 2035/0091: {GUI [graphical user interfaces]} - G01N 2035/0491: {Position sensing, encoding; closed-loop control} - G05B 2219/40392: Programming, visual robot programming language - G05B 2219/40393: Learn natural high level command, associate its template with a plan, sequence - G06T 2207/30204: Marker - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract A lab system configures robots to performs protocols in labs. The lab automation system receives, via a user interface, an instruction from a user to perform a protocol within a lab. The instruction may comprise text, and the lab may comprise a robot configured to perform the protocol. The lab system converts, using a machine learned model, the text into steps and, for each step, identifies one or more of an operation, lab equipment, and reagent associated with the step. In response to detecting an ambiguity/error associated with the step, the lab system notifies the user via the user interface of the ambiguity/error. The lab system may receive one or more indications from the user that resolve the ambiguity/error and update the associated steps. For each step, the lab system configures the robot to perform an identified operation, interact with identified lab equipment, and/or access/use an identified reagent. ## Claims ### Claim 1 A method comprising: receiving, via a graphic user interface, an instruction from a user to perform a protocol within a lab, the instruction comprising text, the lab comprising a robot configured to perform the protocol; converting, using a machine learned model, the text of the instruction into steps; for each step, identifying one or more of an operation associated with the step, lab equipment associated with the step, and a reagent associated with the step, wherein at least one step is modified to include a replacement operation, replacement lab equipment, or a replacement reagent based on a capability of the lab such that an outcome of the modified step is not substantially different from the at least one step; in response to detecting an ambiguity or error associated with a first step of the steps, notifying the user via the graphic user interface; and for each step, configuring a robot to perform the step by: performing an identified operation associated with the step; interacting with identified lab equipment associated with the step; and/or accessing and using an identified reagent associated with the step. ### Claim 2 (depends on 1) The method of claim 1, wherein the graphic user interface displays a virtual representation of the lab and notifying the user via the graphic user interface comprises: selecting candidate lab equipment for the ambiguity or error; and highlighting representative lab equipment within the virtual representation associated with the candidate lab equipment. ### Claim 3 (depends on 1) The method of claim 1, wherein the graphic user interface displays a virtual representation of the lab and notifying the user via the graphic user interface comprises: selecting a candidate operation for the ambiguity or error; and displaying an indication of the first step along with a representative action associated with the candidate operation within the virtual representation. ### Claim 4 (depends on 1) The method of claim 1, wherein notifying the user via the graphic user interface comprises displaying a list of one or more candidate operations, lab equipment, or reagents. ### Claim 5 (depends on 1) The method of claim 1, wherein the machine learned model uses natural language processing to extract generic capabilities from the text. ### Claim 6 (depends on 1) The method of claim 1, wherein identifying one or more of an operation associated with the step, lab equipment associated with the step, and a reagent associated with the step comprises accessing a lab database indicative of what lab equipment and reagents are available in the lab. ### Claim 7 (depends on 6) The method of claim 6, wherein the lab database further comprises information indicating how lab equipment stacks, overlaps, and interacts within the lab. ### Claim 8 (depends on 1) The method of claim 1, wherein the machine learned model is trained on historical operations performed with lab equipment and reagents within a lab. ### Claim 9 A non-transitory computer-readable storage medium storing instructions executable by a processor, the instructions comprising: instructions for receiving, via a graphic user interface, an instruction from a user to perform a protocol within a lab, the instruction comprising text, the lab comprising a robot configured to perform the protocol; instructions for converting, using a machine learned model, the text of the instruction into steps; instructions for, for each step, identifying one or more of an operation associated with the step, lab equipment associated with the step, and a reagent associated with the step, wherein at least one step is modified to include a replacement operation, replacement lab equipment, or a replacement reagent based on a capability of the lab such that an outcome of the modified step is not substantially different from the at least one step; instructions for, in response to detecting an ambiguity or error associated with a first step of the steps, notifying the user via the graphic user interface; and instructions for, for each step, configuring a robot to perform the step by: performing an identified operation associated with the step; interacting with identified lab equipment associated with the step; and/or accessing and using an identified reagent associated with the step. ### Claim 10 (depends on 9) The non-transitory computer-readable storage medium of claim 9, wherein the graphic user interface displays a virtual representation of the lab and the instructions for notifying the user via the graphic user interface comprise: instructions for selecting candidate lab equipment for the ambiguity or error; and instructions for highlighting representative lab equipment within the virtual representation associated with the candidate lab equipment. ### Claim 11 (depends on 9) The non-transitory computer-readable storage medium of claim 9, wherein the graphic user interface displays a virtual representation of the lab and the instructions for notifying the user via the graphic user interface comprise: instructions for selecting a candidate operation for the ambiguity or error; and instructions for displaying an indication of the first step along with a representative action associated with the candidate operation within the virtual representation. ### Claim 12 (depends on 9) The non-transitory computer-readable storage medium of claim 9, wherein the instructions for notifying the user via the graphic user interface comprise instructions for displaying a list of one or more candidate operations, lab equipment, or reagents. ### Claim 13 (depends on 9) The non-transitory computer-readable storage medium of claim 9, wherein the machine learned model uses natural language processing to extract generic capabilities from the text. ### Claim 14 (depends on 9) The non-transitory computer-readable storage medium of claim 9, wherein the instructions for identifying one or more of an operation associated with the step, lab equipment associated with the step, and a reagent associated with the step comprise instructions for accessing a lab database indicative of what lab equipment and reagents are available in the lab. ### Claim 15 (depends on 14) The non-transitory computer-readable storage medium of claim 14, wherein the lab database further comprises information indicating how lab equipment stacks, overlaps, and interacts within the lab. ### Claim 16 (depends on 9) The non-transitory computer-readable storage medium of claim 9, wherein the machine learned model is trained on historical operations performed with lab equipment and reagents within a lab. ### Claim 17 A computer system comprising: a processor; and a non-transitory computer-readable storage medium storing instructions that, when executed, cause the processor to: receive, via a graphic user interface, an instruction from a user to perform a protocol within a lab, the instruction comprising text, the lab comprising a robot configured to perform the protocol; convert, using a machine learned model, the text of the instruction into steps; for each step, identify one or more of an operation associated with the step, lab equipment associated with the step, and a reagent associated with the step, wherein at least one step is modified to include a replacement operation, replacement lab equipment, or a replacement reagent based on a capability of the lab such that an outcome of the modified step is not substantially different from the at least one step; in response to detecting an ambiguity or error associated with a first step of the steps, notify the user via the graphic user interface; and for each step, configure a robot to perform the step by: performing an identified operation associated with the step; interacting with identified lab equipment associated with the step; and/or accessing and using an identified reagent associated with the step. ### Claim 18 (depends on 17) The computer system of claim 17, wherein the graphic user interface displays a virtual representation of the lab and the instructions the cause the processor to notify the user via the graphic user interface further cause the processor to: select candidate lab equipment for the ambiguity or error; and highlight representative lab equipment within the virtual representation associated with the candidate lab equipment. ### Claim 19 (depends on 17) The computer system of claim 17, wherein the graphic user interface displays a virtual representation of the lab and the instructions the cause the processor to notify the user via the graphic user interface further cause the processor to: selecting a candidate operation for the ambiguity or error; and displaying an indication of the first step along with a representative action associated with the candidate operation within the virtual representation. ### Claim 20 (depends on 17) The computer system of claim 17, wherein the instructions that cause the processor to notify the user via the graphic user interface further cause the processor to display a list of one or more candidate operations, lab equipment, or reagents.