# US 11,794,349 B2 · System and method of integrating robot into warehouse management software > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,794,349 B2 - Document ID: US11794349B2 - Status: Granted - Kind: B2 - Application number: 17/038,321 - Filed: 2020-09-30 - Published: 2021-09-02 - Granted: 2023-10-24 - Earliest claimed: 2020-02-28 - Assignees: Nimble Robotics, Inc. - Applicants: Nimble Robotics, Inc. - Inventors: Simon Kalouche; Siva Chaitanya Mynepalli; John Vashchhenko; Suyash Nigam; Jordan Dawson; George Marchman - Web page: https://patentgrep.com/patent/US11794349B2 ## CPC - B25J 9/1697: {Vision controlled systems} - B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor} - B25J 9/1679: {characterised by the tasks executed} - B25J 13/08: by means of sensing devices, e.g. viewing or touching devices - B25J 19/04: Viewing devices - G06Q 10/08741: {by picking of items from inventory for fulfillment} - G05B 2219/40013: Kitting, place parts from belt into tray, place tray on conveyor belt - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract A parsing system includes an imaging device and one or more processors. The imaging device may capture visual instructions displayed on a monitor. The one or more processors may be configured to receive, from the imaging device, the captured visual instructions, generate functions for a robot to perform based on the captured visual instructions, and transmit, to a robot, the functions to perform. ## Claims ### Claim 1 A parsing system, comprising: an imaging device configured to capture visual instructions provided by a warehouse or fulfillment center system and displayed on a monitor; a custom device configured to emulate signals generated by input devices within the warehouse or fulfillment center system; and one or more processors, wherein the one or more processors are configured to: receive, from the imaging device, captured visual instructions; process the received visual instructions to determine task instructions included in the received visual instructions; and generate functions for a robot to perform based on the determined task instructions, the functions configured to cause the robot to perform the determined task instructions. ### Claim 2 (depends on 1) The parsing system of claim 1, further comprising: the robot, wherein the robot includes a picking arm configured to grasp an item from a first location and place the item in a second location. ### Claim 3 (depends on 2) The parsing system of claim 2, wherein the robot is further configured to: perform the functions; and provide a confirmation of completion of the functions to the one or more processors after performing the functions. ### Claim 4 (depends on 1) The parsing system of claim 1, wherein the one or more processors are further configured to transmit a completion response indicating completion of the captured visual instructions to the warehouse or fulfillment center system, after receiving a confirmation of completion of the functions from the robot. ### Claim 5 (depends on 1) The parsing system of claim 1, wherein the functions configured to cause the robot to perform the determined task instructions include one or more functions configured to cause the robot to pick an item from a first location and place the item in a second location. ### Claim 6 (depends on 1) The parsing system of claim 1, wherein the parsing system further includes a physical button-pushing device, wherein the physical button-pushing device pushes a button connected to the warehouse or fulfillment center system to provide a completion response to the warehouse or fulfillment center system. ### Claim 7 (depends on 6) The parsing system of claim 6, wherein the completion response provided by the physical button-pushing device notifies the warehouse or fulfillment center system of completion of the captured visual instructions in response to receiving a confirmation of completion of the functions from the robot. ### Claim 8 (depends on 1) The parsing system of claim 1, wherein the custom device is further configured to: transmit, in response to receiving a confirmation of completion of the functions from the robot, one or more emulated signals to provide a completion response notifying the warehouse or fulfillment center system of completion of the captured visual instructions. ### Claim 9 (depends on 1) The parsing system of claim 1, wherein the imaging device is a camera configured to capture an image of the visual instructions displayed on the monitor. ### Claim 10 (depends on 1) The parsing system of claim 1, wherein the imaging device is an image or video capture card. ### Claim 11 A system, comprising: robot having a picking arm configured to grasp an item from a first location and place the grasped item in a second location spaced from the first location; an imaging device to capture visual instructions provided by a warehouse or fulfillment center system and displayed on a monitor; a custom device configured to transmit a completion response to the warehouse or fulfillment center system; and one or more processors, wherein the one or more processors are configured to: receive, from the imaging device, captured visual instructions; generate functions for the robot to perform based on the captured visual instructions; and transmit, via the custom device, the completion response to the warehouse or fulfillment center system indicating completion of the captured visual instructions after receiving a confirmation of completion of the generated functions from the robot. ### Claim 12 (depends on 11) The system of claim 11, wherein the robot is further configured to: perform the generated functions; and provide the confirmation of completion of the generated functions. ### Claim 13 (depends on 11) The system of claim 11, wherein the first location is a first container and the second location is a second container. ### Claim 14 A method comprising: receiving, by one or more processors, an image including visual instructions displayed on a monitor and provided by a warehouse or fulfillment center system; by the one or more processors, task instructions included in the visual instructions; generating, by the one or more processors, task functions for a robot, the task functions configured to cause the robot to perform the determined task instructions; receiving, by the one more processors, a confirmation of completion of the task functions from the robot; and transmitting, via a custom device configured to emulate signals generated by input devices, a completion response to the warehouse or fulfillment center system indicating completion of the task instructions after receiving the confirmation of completion of the task functions from the robot. ### Claim 15 (depends on 14) The method of claim 14, wherein the one or more processors is distinct from the warehouse or fulfillment center system; and the one or more processors are located within the robot or external to the robot. ### Claim 16 (depends on 14) The method of claim 14, further comprising: performing, by the robot, the task functions; and providing, by the robot, the confirmation of completion of the task functions to the one or more processors after performing the task functions. ### Claim 17 (depends on 14) The method of claim 14, wherein the task functions are further configured to cause the robot to pick an item from a first section of a first container and place the item into a second container, the first section of the first container and/or the second container being determined from the task instructions. ### Claim 18 (depends on 11) The system of claim 11, wherein the warehouse or fulfillment center system is pre-existing.