# US 12,002,007 B2 · Robotic system with automated package scan and registration mechanism and methods of operating the same

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,007 B2
- Document ID: US12002007B2
- Status: Granted
- Kind: B2
- Application number: 17/944,117
- Filed: 2022-09-13
- Published: 2023-01-12
- Granted: 2024-06-04
- Earliest claimed: 2018-10-30
- Assignees: MUJIN, Inc.
- Applicants: MUJIN, Inc.
- Inventors: Rosen Nikolaev Diankov; Huan Liu; Xutao Ye; Jose Jeronimo Moreira Rodrigues; Yoshiki Kanemoto; Jinze Yu; Russell Islam
- Web page: https://patentgrep.com/patent/US12002007B2

## CPC

- G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders
- G06T 7/001: {using an image reference approach}
- B25J 9/1676: {Avoiding collision or forbidden zones}
- B25J 9/1687: {Assembly, peg and hole, palletising, straight line, weaving pattern movement}
- B25J 9/1697: {Vision controlled systems}
- B65B 43/46: using grippers
- B65G 61/00: Use of pick-up or transfer devices or of manipulators for stacking or de-stacking articles not otherwise provided for
- G05B 19/40935: {Selection of predetermined shapes and defining the dimensions with parameter input}
- G06T 7/11: Region-based segmentation
- G06T 7/12: Edge-based segmentation
- G06T 7/13: Edge detection
- G06T 7/174: involving the use of two or more images
- G06T 7/50: Depth or shape recovery
- G06T 7/62: of area, perimeter, diameter or volume
- G06T 7/73: using feature-based methods
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 20/64: Three-dimensional [3D] objects
- G05B 2219/40006: Placing, palletize, un palletize, paper roll placing, box stacking
- G05B 2219/40607: Fixed camera to observe workspace, object, workpiece, global
- G06T 2200/04: involving 3D image data
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20164: Salient point detection; Corner detection
- G06T 2207/30108: Industrial image inspection
- G06T 2207/30164: Workpiece; Machine component
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 2201/06: Recognition of objects for industrial automation
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- B65B: MACHINES, APPARATUS OR DEVICES FOR, OR METHODS OF, PACKAGING ARTICLES OR MATERIALS; UNPACKING (bundling and pressing devices for cigars A24C1/44; devices for tensioning and securing binders adapted to be supported by the article or articles to be bound B25B25/00; applying closure members to bottles, jars or similar containers B67B1/00-B67B6/00; concurrent cleaning, filling and closing of bottles B67C7/00; emptying bottles, jars, cans, casks, barrels or similar containers B67C9/00)
- B65G: TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS (packaging B65B; handling thin or filamentary materials, e.g. paper sheets or thread, B65H; cranes B66C; portable mobile or lifting or hauling appliances, e.g. hoists, B66D; devices for lifting or lowering goods for loading or unloading purposes, e.g. fork-lift trucks, B66F9/00; emptying bottles, jars, casks, barrels or similar containers, not otherwise provided for, B67C9/00; delivering or transferring liquids B67D; filling or discharging vessels for liquefied, solidified or compressed gases F17C; pipe-line systems for fluids F17D)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system and method for operating a robotic system to scan and register unrecognized objects is disclosed. The robotic system may use an image data representative of an unrecognized object located at a start location to implement operations for transferring the unrecognized object from the start location. While implementing the operations, the robotic system may obtain additional data, including scanning results of one or more portions of the unrecognized object not included in the image data. The robotic system may use the additional data to register the unrecognized object.

## Claims

### Claim 1

A method for operating a robotic system, the method comprising:
receiving an image data representative of an unrecognized object located at a start location;
implementing operations for transferring the unrecognized object to a task location;
deriving a location of a surface or a portion thereof of the unrecognized object based on the image data;
using the image data, deriving a scanning position based on the location of the surface of the unrecognized object;
implementing operations for positioning the unrecognized object to expose the surface and/or one or more portions of the unrecognized object to a sensor, during the operations for transferring, based on the scanning position;
generating a scanning result representative of an output from the sensor scanning the surface and/or the one or more portions; and
creating new registration data representative of a new record for the unrecognized object, wherein the new registration data includes the scanning result.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
generating one or more commands for operating the sensor based on implementing the operations for positioning the unrecognized object to expose the surface to the sensor; and
wherein:
the scanning result is representative of a feature on or accessible through the surface of the unknown object.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
deriving a scanning maneuver based on the scanning position, wherein the scanning maneuver is for positioning the unrecognized object to expose multiple portions thereof to the sensor; and
implementing operations according to the scanning maneuver for exposing the multiple portions of the unrecognized object to the sensor to scan the multiple portions.

### Claim 4 (depends on 3)

The method of claim 3, wherein the scanning maneuver corresponds to rotating the unrecognized object about one or more of the axes of the unrecognized object.

### Claim 5 (depends on 3)

The method of claim 3, further comprising:
calculating a dimension of the surface based on an object height and/or a first set of peripheral edges of the unknown object;
deriving a second scanning position based on the location of the surface when the dimension exceeds a threshold, wherein the scanning position and the second scanning position are for exposing different portions of the unrecognized object to the sensor; and
wherein:
the scanning maneuver is for positioning the unrecognized object from the scanning position to the second scanning position while the scanner is configured to continuously perform a scanning operation.

### Claim 6

A robotic system comprising:
at least one processor; and
at least one memory device connected to the at least one processor and having stored thereon instructions executable by the processor for:
receiving an image data representative of an unrecognized object located at a start location;
implementing operations for transferring the unrecognized object to a task location;
while implementing the operations for transferring the unrecognized object, implement operations for scanning at least one portion of the unrecognized object that was unrecognizable or hidden in the image data;
generating a scanning result based on implementing the operations for scanning the at least one portion of the unrecognized object; and
creating new registration data representative of a new record for cataloging the unrecognized object, wherein the new registration data includes the scanning result.

### Claim 7 (depends on 6)

The robotic system of claim 6, wherein the at least one memory device includes instructions for:
obtaining additional data corresponding to the unrecognized object while implementing the transfer operations;
deriving a scanning position based on the additional data, wherein the scanning position represents a pose for presenting to a sensor the at least one portion of the unrecognized object;
generating the scanning result based on a command and/or a setting corresponding to the scanning operation for positioning the unrecognized object according to the scanning position, wherein the scanning operation is coordinated with operating the sensor.

### Claim 8 (depends on 7)

The robotic system of claim 7, wherein:
the image data depicts a surface of the unrecognized object;
the at least one memory device includes instructions for:
identifying one or more locations of corresponding to the surface or a portion thereof based on the image data;
deriving a target location of the unrecognized object based on the surface or the portion thereof, wherein the target location represents a position of the at least one portion of the unrecognized object; and
deriving the scanning position based on the target location and a scanning zone located about the sensor.

### Claim 9 (depends on 7)

The robotic system of claim 7, wherein the at least one portion of the unrecognized object includes a surface or a peripheral portion thereof of the unrecognized object.

### Claim 10 (depends on 9)

The robotic system of claim 9, wherein:
the scanning position corresponds to a first corner of the surface; and
the at least one memory device includes instructions for implementing operations for scanning based on:
a second scanning position that represents a pose for exposing to the sensor a second corner of the surface; and
a scanning maneuver derived according to the scanning position and the second scanning position for scanning between the first corner and the second corner of the surface.

### Claim 11 (depends on 10)

The robotic system of claim 10, wherein the at least one memory device includes instructions for generating the scanning result including an identifier location based on:
receiving an identifier value resulting from scanning an identifier accessible on or through the surface;
determining a gripper location at a time of receiving the identifier value;
deriving an identifier location based on the gripper location, wherein the identifier location represents a position of the identifier relative to the surface; and
generating the scanning result including the identifier value and the identifier location.

### Claim 12 (depends on 7)

The robotic system of claim 7, wherein the at least one memory device includes instructions for implementing operations for scanning at least one portion of the unrecognized object based on further generating a command and/or a setting for rotating the unrecognized object about one or more axes to expose multiple surfaces to the sensor.

### Claim 13 (depends on 6)

The robotic system of claim 6, wherein the at least one memory device includes instructions for:
receiving a surface image, wherein the surface image is a visual representation of the at least one portion of the unrecognized object; and
generating the scanning result based on the surface image.

### Claim 14 (depends on 13)

The robotic system of claim 13, wherein the at least one memory device includes instructions for:
determining an object feature based on visually processing the surface image, wherein the object feature corresponds to an identifier exposed on the depicted surface;
deriving a feature location based on visually processing the surface image, wherein the feature location represents a position of the object feature on the unrecognized object; and
generating the scanning result including the object feature and the feature location.

### Claim 15 (depends on 13)

The robotic system of claim 13, wherein the at least one memory device includes instructions for:
identifying peripheral edges of the unrecognized object in the surface image; and
generating the scanning result including a cropped portion of the surface image within the identified peripheral edges.

### Claim 16

A tangible, non-transient computer-readable medium having processor instructions stored thereon that, when executed by a robotic system via one or more processors thereof, cause the robotic system to perform a method, the method comprising:
receiving an image data representative of an unrecognized object located at a start location;
implementing operations for transferring the unrecognized object to a task location;
deriving a location of a surface or a portion thereof of the unrecognized object based on the image data;
using the image data, deriving a scanning position based on the location of the surface of the unrecognized object;
implementing operations for positioning the unrecognized object to expose the surface and/or one or more portions of the unrecognized object to a sensor, during the operations for transferring, based on the scanning position;
generating a scanning result representative of an output from the sensor scanning the surface and/or the one or more portions; and
creating new registration data representative of a new record for the unrecognized object, wherein the new registration data includes the scanning result.

### Claim 17 (depends on 16)

The tangible, non-transient computer-readable medium of claim 16 having processor instructions to perform the method that includes:
generating one or more commands for operating the sensor based on implementing the operations for positioning the unrecognized object to expose the surface to the sensor; and
wherein:
the scanning result is representative of a feature on or accessible through the surface of the unknown object.

### Claim 18 (depends on 16)

The tangible, non-transient computer-readable medium of claim 16 having processor instructions to perform the method that includes:
deriving a scanning maneuver based on the scanning position, wherein the scanning maneuver is for positioning the unrecognized object to expose multiple portions thereof to the sensor; and
implementing operations according to the scanning maneuver for exposing the multiple portions of the unrecognized object to the sensor to scan the multiple portions.

### Claim 19 (depends on 18)

The tangible, non-transient computer-readable medium of claim 18 having processor instructions to perform the method, wherein the scanning maneuver corresponds to rotating the unrecognized object about one or more of the axes of the unrecognized object.

### Claim 20 (depends on 18)

The tangible, non-transient computer-readable medium of claim 18 having processor instructions to perform the method that includes:
calculating a dimension of the surface based on an object height and/or a first set of peripheral edges of the unknown object;
deriving a second scanning position based on the location of the surface when the dimension exceeds a threshold, wherein the scanning position and the second scanning position are for exposing different portions of the unrecognized object to the sensor; and
wherein:
the scanning maneuver is for positioning the unrecognized object from the scanning position to the second scanning position while the scanner is configured to continuously perform a scanning operation.