# US 12,602,806 B2 · Systems and methods for machine vision robotic processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,602,806 B2
- Document ID: US12602806B2
- Status: Granted
- Kind: B2
- Application number: 17/963,156
- Filed: 2022-10-10
- Published: 2023-06-22
- Granted: 2026-04-14
- Earliest claimed: 2021-10-08
- Assignees: University of Maryland, College Park
- Applicants: University of Maryland, College Park
- Inventors: Yang Tao; Dongyi Wang; Mohamed Amr Ali
- Web page: https://patentgrep.com/patent/US12602806B2

## CPC

- G06T 7/593: from stereo images
- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1697: {Vision controlled systems}
- G06T 7/11: Region-based segmentation
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06V 10/141: Control of illumination
- G06V 10/143: Sensing or illuminating at different wavelengths
- G06V 10/82: using neural networks
- G05B 2219/40053: Pick 3-D object from pile of objects
- G05B 2219/45063: Pick and place manipulator
- G06T 2207/10152: Varying illumination
- G06T 2207/20084: Artificial neural networks [ANN]
- G06V 2201/06: Recognition of objects for industrial automation
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

Systems, methods, and media for machine vision-guided robotic loading are provided. Such systems and methods can allow for more accurate object detection so as to allow for robotic picking or loading of an object from a pile or group of objects. In some embodiments, a camera is controlled to capture a background image of the object or group of objects. One or more illumination sources (such as, e.g., laser illuminators) are controlled to project immunization toward the object or group of objects within the field-of-view of the camera. The camera may be controlled to capture a scan image comprising the projected illumination. Based on the scan image, a depth map can be determined, which is provided to a machine learning model along with the background image. The output of the machine learning model can then provide information associated with one or more objects, such as location information, orientation information, and the like.

## Claims

### Claim 1

A system for detecting an object on a surface, comprising:
a camera oriented to include the surface in a field-of-view (FOV) of the camera;
a plurality of aimable illumination sources configured to project corresponding illuminations toward the surface within the FOV of the camera; and
a controller comprising a processor that is configured:
to control the camera to capture, with the camera, a first background image of the object on the surface;
to control the plurality of aimable illumination sources to project the corresponding illuminations simultaneously toward the surface to form a projected illumination on the surface;
to control the camera to capture a scan image comprising the projected illumination formed by simultaneously projected corresponding illuminations that intersect the surface at respective positions;
to determine a first depth map based on separating the scan image into a plurality of object scan mask sets, each object scan mask set representing a corresponding image formed only by an illumination of an aimable illumination source chosen from the plurality of aimable illumination sources;
to provide the first depth map and the first background image to a trained machine learning model executed by the processor; and
to receive an output from the machine learning model, wherein the output comprises information associated with the object;
wherein the plurality of aimable illumination sources comprises a first aimable illumination source and a second aimable illumination source, and wherein the first aimable illumination source and the second aimable illumination source have different spectra, and
wherein the controller is further configured to separate the scan image into the plurality of scan image sets each of which is formed only by light having a spectrum of an illumination of said aimable illumination source chosen from the plurality of aimable illumination sources.

### Claim 2 (depends on 1)

The system of claim 1, wherein each of the aimable illumination sources comprises a line laser.

### Claim 3 (depends on 2)

The system of claim 2, wherein each of the aimable illumination sources further comprises:
a motor; and
a mirror coupled to the motor,
wherein the mirror is positioned to reflect illumination from the line laser;
wherein the motor is controllable to rotate the mirror based on a received control input.

### Claim 4 (depends on 1)

The system of claim 1, wherein the controller is further configured:
to controllably scan the projected illumination across the surface within the FOV in a plurality of scan steps by changing a position of a corresponding illumination of each of the plurality of aimable illumination sources on the surface;
to determine an object scan set, comprising a plurality of scan image sets, at each scan step of the plurality of scan steps; and
to determine the first depth map based on the object scan set.

### Claim 5 (depends on 4)

The system of claim 4, wherein the controller is further configured:
to determine a baseline scan set for each of the plurality of aimable illumination sources; and
to determine the first depth map based on the object scan set by subtracting, pixel by pixel, the baseline scan set from the object scan set to identify a pixel shift in the projected illumination captured in a given object scan mask set as compared to a corresponding baseline scan mask set, for each scan step, wherein the shift is due to the projected illumination intersecting with the object.

### Claim 6 (depends on 1)

The system of claim 1, wherein the controller is further configured to control a robotic picking device to acquire the object according to the output of the machine learning model.

### Claim 7 (depends on 1)

The system of claim 1, wherein the controller is further configured:
to determine, based on the output of the machine learning model, a first object among a plurality of objects to prioritize; and
to provide a first signal to a robotic picking device to aid in acquiring the first object.

### Claim 8 (depends on 7)

The system of claim 7, wherein the controller is further configured, after the first object has been removed from the surface:
to provide a second depth map and a second background image to the machine learning model;
to receive a second output from the machine learning model and, based on the second output, determine a second object among the plurality of objects to prioritize; and
to provide a second signal to the robotic picking device to aid in acquiring the second object.

### Claim 9 (depends on 1)

The system of claim 1, wherein the machine learning model is configured:
to identify the object; and
wherein the information associated with the object comprises at least one of a mask, a key a point, a class, and a bounding box for the object identified by the machine learning model.

### Claim 10

A method for detecting an object, the method comprising:
capturing, with a camera, a first background image of the object on a surface that is positioned in a field-of-view (FOV) of the camera;
controlling a plurality of aimable illumination sources to simultaneously project corresponding illuminations toward the surface within the FOV of the camera to form a projected illumination on the surface;
capturing, with the camera, a scan image comprising the projected illumination formed by simultaneously projected corresponding illuminations that intersect the surface at respective positions;
determining a first depth map based on separating the scan image into plurality of object scan mask sets, each object scan mask set representing a corresponding image formed only by an illumination of an aimable illumination source chosen from the plurality of aimable illumination sources; providing the first depth map and the first background image to a machine learning model; and receiving an output from the machine learning model, wherein the output comprises information associated with the object,
wherein the plurality of aimable illumination sources comprises a first aimable illumination source and a second aimable illumination source, and wherein the first aimable illumination source and the second aimable illumination source have different spectra, and
wherein the controller is configured to separate the scan image into the plurality of scan image sets each of which is formed only by light having a spectrum of an illumination of said aimable illumination source chosen from the plurality of aimable illumination sources.

### Claim 11 (depends on 10)

The method of claim 10, wherein each of the aimable illumination sources comprises a line laser.

### Claim 12 (depends on 11)

The method of claim 11, wherein each of the aimable illumination sources further comprises:
a motor; and
a mirror coupled to the motor and positioned to reflect illumination from the line laser; and
wherein the controlling the plurality of aimable illumination sources comprises providing a control input to the motor of each aimable illumination source.

### Claim 13 (depends on 10)

The method of claim 10, wherein the method further comprises:
controllably scanning the projected illumination across the surface within the FOV in a plurality of scan steps by changing a position, on the surface, of the corresponding illumination of an aimable illumination source of the plurality of aimable illumination sources,
determining an object scan set, comprising a plurality of scan image sets, at each scan step of the plurality of scan steps, and
determining the first depth map based on the object scan set.

### Claim 14 (depends on 13)

The method of claim 13, wherein the method further comprises:
determining a baseline scan set for each of the plurality of aimable illumination sources; and
wherein the determining the first depth map based on the object scan set comprises subtracting, pixel by pixel, the baseline scan set from the object scan set to identify a pixel shift in the projected illumination captured in a given object scan mask set as compared to a corresponding baseline scan mask set, for each scan step, wherein the shift is due to the projected illumination intersecting with the object.

### Claim 15 (depends on 10)

The method of claim 10, wherein the method further comprises generating a control signal for a robotic picking device to acquire the object according to the output of the machine learning model.

### Claim 16 (depends on 10)

The method of claim 10, wherein the method further comprises:
determining, based on the output of the machine learning model, a first object among a plurality of objects on the surface to prioritize; and
controlling a robotic picking device to acquire the first object.

### Claim 17 (depends on 16)

The method of claim 16, wherein the method further comprises determining a second object among the plurality of objects to prioritize, after the first object has been removed from the surface.

### Claim 18 (depends on 10)

The method of claim 10, wherein the machine learning model is configured to identify the object; and wherein the information associated with the object comprises at least one of a mask, a key point, a class, and a bounding box for the object identified by the machine learning model.

### Claim 19

A system for detecting an object on a surface, comprising:
a camera oriented to include the surface in a field-of-view (FOV) of the camera;
a plurality of aimable illumination sources configured to project corresponding illuminations toward the surface within the FOV of the camera; and
a controller comprising a processor that is configured:
to control the camera to capture, with the camera, a first background image of the object on the surface;
to control the plurality of aimable illumination sources to project the corresponding illuminations simultaneously toward the surface to form a projected illumination on the surface;
to control the camera to capture a scan image comprising the projected illumination formed by simultaneously projected corresponding illuminations that intersect the surface at respective positions;
to determine a first depth map based on separating the scan image into a plurality of object scan mask sets, each object scan mask set representing an image formed only by an illumination of an aimable illumination source chosen from the plurality of aimable illumination sources;
to provide the first depth map and the first background image to a trained machine learning model executed by the processor; and
to receive an output from the machine learning model, wherein the output comprises information associated with the object;
wherein the controller is further configured:
to controllably scan the projected illumination across the surface within the FOV in a plurality of scan steps by changing a position of the corresponding illumination of each of the plurality aimable illumination source on the surface;
to determine an object scan set, comprising a plurality of scan image sets, at each scan step of the plurality of scan steps; and
to determine the first depth map based on the object scan set.

### Claim 20 (depends on 19)

The system of claim 19, wherein the machine learning model is configured:
to identify the object; and
wherein the information associated with the object comprises at least one of a mask, a key point, a class, and a bounding box for the object identified by the machine learning model.