# US 11,731,792 B2 · Kitting machine

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,731,792 B2
- Document ID: US11731792B2
- Status: Granted
- Kind: B2
- Application number: 16/143,278
- Filed: 2018-09-26
- Published: 2020-03-26
- Granted: 2023-08-22
- Earliest claimed: 2018-09-26
- Assignees: Dexterity, Inc.
- Applicants: Dexterity, Inc.
- Inventors: Samir Menon; Gerald Brantner; Zhouwen Sun; Robert Hegdahl; Kevin Jose Chavez
- Web page: https://patentgrep.com/patent/US11731792B2

## CPC

- B65B 5/08: Packaging groups of articles, the articles being individually gripped or guided for transfer to the containers or receptacles
- B25J 9/1679: {characterised by the tasks executed}
- B25J 9/1697: {Vision controlled systems}
- B65B 5/12: Introducing successive articles, e.g. confectionery products, of different shape or size in predetermined positions
- B65B 35/54: Feeding articles along multiple paths to a single packaging position
- B65B 43/52: using roller-ways or endless conveyors
- B65B 43/54: Means for supporting containers or receptacles during the filling operation
- B65B 57/00: Automatic control, checking, warning, or safety devices (registering wrapping or container-forming material fed from rolls B65B41/18)
- B65B 57/14: and operating to control, or stop, the feed of articles or material to be packaged
- B25J 9/1689: {Teleoperation}
- B65B 59/003: {Arrangements to enable adjustments related to the packaging material}
- B65B 59/005: {Adjustable conveying means}
- B65B 2210/02: Plurality of alternative input or output lines or plurality of alternative packaging units on the same packaging line for improving machine flexibility
- B65B 2220/14: Adding more than one type of material or article to the same package
- G05B 2219/40013: Kitting, place parts from belt into tray, place tray on conveyor belt
- B65B: MACHINES, APPARATUS OR DEVICES FOR, OR METHODS OF, PACKAGING ARTICLES OR MATERIALS; UNPACKING (bundling and pressing devices for cigars A24C1/44; devices for tensioning and securing binders adapted to be supported by the article or articles to be bound B25B25/00; applying closure members to bottles, jars or similar containers B67B1/00-B67B6/00; concurrent cleaning, filling and closing of bottles B67C7/00; emptying bottles, jars, cans, casks, barrels or similar containers B67C9/00)
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A kitting machine is disclosed. In various embodiments, the machine includes an actuator configured to move a surface or receptacle to a position associated with item retrieval; and a controller configured to control operation of the actuator to position an item in the position associated with item retrieval, in a manner that is synchronized at least in part with operation of a robotic retrieval device configured to retrieve the item from the position associated with item retrieval.

## Claims

### Claim 1

A system, comprising:
an actuator configured to move a surface or receptacle to a position associated with item retrieval;
one or more cameras or sensors within an environment in which a robotic retrieval device operates to retrieve one or more items and pack the one or more items into a box or other container; and
a first controller associated with the actuator;
a second controller associated with the robotic retrieval device;
a processor configured to:
determine the one or more items to retrieve and pack into the box or other container, wherein determining the one or more items includes determining a location from which an item of the one or more items is to be retrieved, and determining one or more attributes associated with the item;
obtain a plan to retrieve the item and to pack the item into the box or other container, wherein:
the plan is based at least in part on the one or more items to be retrieved, the location from which the item is to be retrieved, and the one or more attributes associated with the item; and
the plan includes a coordination of controlling the robotic retrieval device and the actuator so that the actuator moves the item to the position associated with item retrieval and the robotic retrieval device moves to retrieve the item from the position associated with item retrieval; and
control the robotic retrieval device via the second controller and the actuator via the first controller at least in part by:
using at least the robotic retrieval device to attempt to perform an individual task according to a first predefined strategy;
in response to determining that the performing the individual task according to the first predefined strategy was attempted, determining whether the individual task was completed successfully according to the attempt to implement the first predefined strategy;
in response to a determination by the processor that the individual task was not completed using the first predefined strategy:
determining that a plurality of alternative strategies are available to perform the individual task;
selecting a first alternative strategy to perform the individual task from the plurality of alternative strategies based on a respective confidence score associated with the first alternative strategy; and
iteratively using at least the robotic retrieval device to perform the individual task according to the plurality of alternative strategies for the individual task until the earlier of (i) the individual task is completed, or (ii) a prescribed number of attempts have been performed, wherein the plurality of alternative strategies are iteratively used based on their respective confidence scores; and
in response to a determination by the processor that the prescribed number of attempts have been performed, invoking a teleoperation for human intervention in which a user remotely controls the robotic retrieval device to perform the individual task.

### Claim 2 (depends on 1)

The system of claim 1, wherein the processor is configured to:
control the actuator via the first controller to position the item in the position associated with item retrieval; and
control the robotic retrieval device via the second controller to retrieve the item from the position associated with item retrieval.

### Claim 3 (depends on 2)

The system of claim 2, wherein the system further includes a conveyor configured to position the box or other container in a loading position and the processor is further configured to:
control the conveyor to position the box or other container in the loading position;
coordinate operation of the actuator and the robotic retrieval device to position the item in the position associated with item retrieval;
use the robotic retrieval device to retrieve the item from the position associated with item retrieval; and
use the robotic retrieval device to place the item in the box or other container.

### Claim 4 (depends on 1)

The system of claim 1, wherein the one or more cameras or sensors comprises one or more of a camera or other image sensor, a weight sensor, a pressure sensor, a limit switch, and a contact sensor.

### Claim 5 (depends on 1)

The system of claim 1, wherein the processor is configured to do one or both of the following based at least in part on sensor output data generated by the one or more cameras or sensors:
control the actuator via the first controller to position the item in the position associated with item retrieval; and
control the robotic retrieval device via the second controller to retrieve the item from the position associated with item retrieval.

### Claim 6 (depends on 1)

The system of claim 1, wherein the actuator comprises one or more of a motor, a linear motor, and a motor-driven linear bearing.

### Claim 7 (depends on 1)

The system of claim 1, wherein the surface comprises a conveyor belt and the first controller controls the actuator to advance the item under control of the first controller from a first end of the conveyor belt at which the item is loaded onto the conveyor belt to a pick up zone at a second end of the conveyor belt.

### Claim 8 (depends on 1)

The system of claim 1, wherein the processor is configured to control one or more of the first controller, the second controller, the robotic retrieval device, and a carriage to position the item in the position associated with item retrieval and retrieve the item using the robotic retrieval device.

### Claim 9 (depends on 1)

The system of claim 1, wherein the actuator is associated with a first kitting machine and the item comprises a first item of a first item type; and the system further comprises one or more additional kitting machines, each associated with a corresponding item of an associated item type.

### Claim 10 (depends on 9)

The system of claim 9, wherein the processor is configured to detect that a new kitting machine has become associated with the system and to integrate the new kitting machine into the system.

### Claim 11 (depends on 10)

The system of claim 10, wherein:
the processor is configured to integrate the new kitting machine into the system by registering the new kitting machine with a control computer, and performing one or more of the following:
initializing the new kitting machine;
determining one or more of a position, location, and orientation of the new kitting machine;
determining one or both of a corresponding item and a corresponding item type associated with the new kitting machine;
determine a capability of the new kitting machine;
adding the new kitting machine to a network associated with the system; and
performing an operational test on the new kitting machine.

### Claim 12 (depends on 1)

The system of claim 1, wherein the processor is configured to:
control the actuator via the first controller to position the item in the position associated with item retrieval;
control the robotic retrieval device via the second controller to retrieve the item from the position associated with item retrieval;
receive from the one or more cameras or sensors an indication that the system has entered a state in which the system cannot continue a current kitting operation through automated operation; and
initiate on demand human teleoperation based at least in part on the indication.

### Claim 13 (depends on 12)

The system of claim 12, wherein the on demand human teleoperation is provided via a teleoperation device associated with the system.

### Claim 14 (depends on 13)

The system of claim 13, wherein the teleoperation device is associated with a plurality of robotic kitting systems that includes the system.

### Claim 15 (depends on 14)

The system of claim 14, wherein at least a subset of the plurality of robotic kitting systems comprises robotic kitting systems located in one or more locations other than a first location in which the system is located; and wherein the on demand human teleoperation is configured to be performed using the teleoperation device remotely via a network.

### Claim 16 (depends on 1)

The system of claim 1, further comprising:
a surface on which a bin comprising a plurality of items is disposed,
wherein:
an actuation of the actuator causes the actuator to move the bin to a pickup zone;
the plan comprises one or more instructions for controlling the actuator to move the bin from an initial position to the pickup zone in coordination with controlling movement of the robotic retrieval device to retrieve the item; and
the plan comprises one or more instructions for controlling to move the bin from the pickup zone after the robotic retrieval device retrieves the item from the bin.

### Claim 17 (depends on 16)

The system of claim 16, wherein controlling to move the bin from the pickup zone after the robotic retrieval device retrieves the item from the bin comprises:
determining whether the bin is empty;
in response to determining that the bin is not empty, controlling the actuator to return the bin to the initial position; and
in response to determining that the bin is empty, controlling the robotic retrieval device to move the bin from the pickup zone to a predefined empty bin location.

### Claim 18 (depends on 17)

The system of claim 17, wherein in response to determining that the bin is empty, controlling to move a replacement bin comprising a plurality of items to the initial position.

### Claim 19

The system of 1, wherein:
the plan comprises a plurality of tasks; and
the system stores a plurality of predefined strategies for performing an individual task of the plurality of tasks.

### Claim 20

The system of 1, wherein:
the robotic retrieval device comprises a robotic arm mounted on a carriage configured to translate along a rail or other linear guide; and
the second controller controls the carriage to move along the rail or other linear guide in connection with positioning the robotic arm in relation to one or more surfaces or receptacles comprising items for the robotic arm to operate to retrieve the item from one or more surfaces or receptacles and to place the item into the box or other container.

### Claim 21 (depends on 1)

The system of claim 1, wherein an alternative strategy of the plurality of alternative strategies includes at least one of (i) using an alternate grip, (ii) approaching the item from a different angle, (iii) agitating or repositioning the item using the robotic retrieval device or terminal device attached to the robotic retrieval device, (iv) agitating or repositioning the item by controlling the actuator to move the surface or receptacle, (v) changing the terminal device attached to the robotic retrieval device to an alternate end effector and using the alternate end effector to engage the item.

### Claim 22 (depends on 1)

The system of claim 1, wherein the second controller is further configured to:
control operation of the robotic retrieval device to collectively pack a plurality of items into the box or other container according to the plan.

### Claim 23

A method to operate a kitting system comprising an actuator configured to move a surface or receptacle to a position associated with item retrieval, a first controller configured to control operation of the actuator, and a second controller associated with a robotic retrieval device, the method comprising:
using information obtained from one or more cameras or sensors in connection with determining one or more items to retrieve and pack into a box or other container, wherein determining the one or more items includes determining a location from which an item of the one or more items is to be retrieved, and determining one or more attributes associated with the item, and the one or more cameras or sensors are located within an environment in which a robotic retrieval device operates to retrieve the one or more items and pack the one or more items into a box or other container;
obtaining a plan to retrieve the item and to pack the item into the box or other container, wherein:
the plan is based at least in part on the one or more items to be retrieved, the location from which the item is to be retrieved, and the one or more attributes associated with the item;
the plan includes a coordination of controlling the robotic retrieval device via the second controller and the actuator via the first controller so that the actuator moves the item to the position associated with item retrieval and the robotic retrieval device moves to retrieve the item from the position associated with item retrieval; and
the obtaining the plan includes:
determining a plan to retrieve and collectively pack a plurality of items into the box or other container, the plurality of items comprising the item that is retrieved from the position associated with item retrieval;
using the actuator to position the item into the position associated with item retrieval; and
using the robotic retrieval device to retrieve the item from the position associated with item retrieval and place the item in a destination location, wherein retrieval and placement of the item is based at least in part on the plan, including:
using at least the robotic retrieval device to attempt to perform an individual task according to a first predefined strategy;
in response to determining that the performing the individual task according to the first predefined strategy was attempted, determining whether the individual task was completed successfully according to the attempt to implement the first predefined strategy;
in response to a determination by a processor that the individual task was not completed using the first predefined strategy;
determining that a plurality of alternative strategies are available to perform the individual task;
selecting a first alternative strategy to perform the individual task from the plurality of alternative strategies based on a respective confidence score associated with the first alternative strategy; and
iteratively using at least the robotic retrieval device to perform the individual task according to the plurality of alternative strategies for the individual task until the earlier of (i) the individual task is completed, or (ii) a prescribed number of attempts have been performed, wherein the plurality of alternative strategies are iteratively used based on their respective confidence scores; and
in response to a determination by the processor that the prescribed number of attempts have been performed, invoking a teleoperation for human intervention in which a user remotely controls the robotic retrieval device to perform the individual task,
wherein:
the actuator is used to position the item in the position associated with item retrieval in a manner that is synchronized at least in part with operation of the robotic retrieval device.

### Claim 24 (depends on 23)

The method of claim 23, wherein the robotic retrieval device comprises a robotic arm having limited reach, and using the actuator to position the item in the position associated with item retrieval results in the item being placed within reach of the robotic arm.

### Claim 25 (depends on 23)

The method of claim 23, wherein in response to the determination by a control computer that no further strategies exist, human operation in connection with performing the individual task is requested and further comprising:
receiving teleoperation instructions corresponding to human operation, the teleoperation instructions being received from a mobile device that is remote with respect to a physical site at which the system is located.

### Claim 26 (depends on 25)

The method of claim 25, further comprising updating a set of strategies to enable the kitting system to repeat or emulate the actions corresponding to the teleoperation instructions to perform the individual task for which teleoperation instructions were received.

### Claim 27 (depends on 23)

The method of claim 23, wherein the processor determines the plan to collectively pack the plurality of items into the box or other container.

### Claim 28 (depends on 23)

The method of claim 23, wherein the robotic retrieval device comprises a plurality of terminal devices, wherein a first terminal device of the plurality of terminal devices is used in connection with performing the individual task according to the first predefined strategy and a second terminal device of the plurality of terminal devices is used in connection with performing the individual task according to at least one of the plurality of alternative strategies.

### Claim 29 (depends on 23)

The method of claim 23, wherein the first predefined strategy is selected from among all strategies corresponding to performance of the individual task based on a determined preference or confidence score for the first predefined strategy compared to the plurality of alternative strategies.

### Claim 30 (depends on 23)

The method of claim 23, wherein an order in which the plurality of alternative strategies are attempted is based at least in part on corresponding confidence scores for the plurality of alternative strategies.

### Claim 31 (depends on 23)

The method of claim 23, wherein iteratively performing the individual task according to the plurality of alternative strategies comprise repositioning the item to a different position to facilitate the robotic retrieval device being able to retrieve the item.

### Claim 32 (depends on 31)

The method of claim 31, wherein:
the using the actuator to position an item into the position, or the repositioning the item to a different position comprises moving the surface or receptacle to the position associated with item retrieval or the different position to facilitate the robotic retrieval device being able to retrieve the item the first controller controls the actuator to move the surface or receptacle, and the second controller controls the robotic retrieval device to retrieve the item from the position associated with item retrieval.

### Claim 33

A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions to operate a kitting system comprising an actuator configured to move a surface or receptacle to a position associated with item retrieval, a first controller configured to control operation of the actuator, and a second controller associated with a robotic retrieval device, the computer program product comprising computer instructions for:
using information obtained from one or more cameras or sensors in connection with determining one or more items to retrieve and pack into the box or other container, wherein determining the one or more items includes determining a location from which an item of the one or more items is to be retrieved, and determining one or more attributes associated with the item, and the one or more cameras or sensors are located within an environment in which the robotic retrieval device operates to retrieve the one or more items and pack the one or more items into a box or other container;
obtaining a plan to retrieve the item and to pack the item into the box or other container, wherein:
the plan is based at least in part on the one or more items to be retrieved, the location from which the item is to be retrieved, and the one or more attributes associated with the item;
the plan includes a coordination of controlling the robotic retrieval device via the second controller and the actuator via the first controller so that the actuator moves the item to the position associated with item retrieval and the robotic retrieval device moves to retrieve the item from the position associated with item retrieval; and
the obtaining the plan includes:
determining a plan to retrieve and collectively pack a plurality of items into the box or other container, the plurality of items comprising the item that is retrieved from the position associated with item retrieval;
using the actuator to position the item into the position associated with item retrieval; and
using the robotic retrieval device to retrieve the item from the position associated with item retrieval and place the item in a destination location, wherein retrieval and placement of the item is based at least in part on the plan, including:
using at least the robotic retrieval device to attempt to perform an individual task according to a first predefined strategy;
in response to determining that the performing the individual task according to the first predefined strategy was attempted, determining whether the individual task was completed successfully according to the attempt to implement the first predefined strategy;
in response to a determination by a processor that the individual task was not completed using the first defined strategy;
determining that a plurality of alternative strategies are available to perform the individual task;
selecting a first alternative strategy to perform the individual task from the plurality of alternative strategies based on a respective confidence score associated with the first alternative strategy; and
iteratively using at least the robotic retrieval device to perform the individual task according to one or more other predefined strategies for the individual task until the earlier of (i) the individual task is completed, or (ii) a prescribed number of attempts have been performed, wherein the plurality of alternative strategies are iteratively used based on their respective confidence scores; and
in response to a determination by the processor that the prescribed number of attempts have been performed, invoking a teleoperation for human intervention in which a user remotely controls the robotic retrieval device to perform the individual task;
wherein:
the actuator is used to position the item in the position associated with item retrieval in a manner that is synchronized at least in part with operation of the robotic retrieval device.