# US 11,991,982 B2 · Tool-pickup system, method, computer program and non-volatile data carrier

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,991,982 B2
- Document ID: US11991982B2
- Status: Granted
- Kind: B2
- Application number: 17/290,704
- Filed: 2019-10-28
- Published: 2022-01-20
- Granted: 2024-05-28
- Earliest claimed: 2018-11-01
- Assignees: DeLaval Holding AB
- Applicants: Delaval Holding AB
- Inventors: Andreas Eriksson
- Web page: https://patentgrep.com/patent/US11991982B2

## CPC

- A01J 5/007: Monitoring milking processes; Control or regulation of milking machines
- A01J 5/0175: {Attaching of clusters}
- A01J 7/02: for cleaning or sanitising milking machines or devices
- B25J 9/1697: {Vision controlled systems}
- B25J 11/0045: {Manipulators used in the food industry}
- B25J 13/08: by means of sensing devices, e.g. viewing or touching devices
- B25J 19/023: {including video camera means}
- G06T 1/0014: {Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023)}
- G06T 7/0008: {checking presence/absence}
- G06T 7/30: Determination of transform parameters for the alignment of images, i.e. image registration
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/73: using feature-based methods
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 20/64: Three-dimensional [3D] objects
- G08B 21/18: Status alarms (G08B21/02 takes precedence)
- H04N 13/207: using a single two-dimensional [2D] image sensor
- H04N 23/695: Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
- B25J 15/0052: {multiple gripper units or multiple end effectors}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30164: Workpiece; Machine component
- G06T 2207/30242: Counting objects in image
- A01J: MANUFACTURE OF DAIRY PRODUCTS
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Tools in an automatic milking arrangement are picked up by using a robotic arm (110). The robotic arm (110) moves a camera (130) to an origin location (PC) from which the camera (130) registers three-dimensional image data (Dimg3D) of at least one tool (141, 142, 143, 144). The three-dimensional image data is processed using an image-based object identification algorithm to identify objects in the form of the tools and hoses (152). In response to identifying at least one tool, a respective tool position (PT1, PT3, PT4) is determined for each identified tool based on the origin location (PC) and the three-dimensional image data. Then, a grip device (115) is exclusively controlled to the one or more of the respective tool positions (PT1, PT3, PT4) to perform a pick-up operation. Thus, futile attempts to pick-up non-existing or blocked tools can be avoided.

## Claims

### Claim 1

A tool-pickup system for an automatic milking arrangement, the tool-pickup system comprising:
a robotic arm (110) provided with a grip device (115) configured to pick up tools (141, 142, 143, 144), and a camera (130) configured to register three-dimensional image data (Dimg3D); and
a control unit (120) operatively connected to the robotic arm, the control unit (120) configured to:
control the robotic arm (110) to move the camera (130) to an origin location (PC) from which at least one tool of the tools (141, 142, 143, 144) is expected to be visible within a view field (VF) of the camera (130),
obtain three-dimensional image data (Dimg3D) registered by the camera (130) at the origin location (PC),
process the three-dimensional image data (Dimg3D) using an image-based object identification algorithm to identify objects in a form of the tools (141, 143, 144) and/or hoses (152), and
in response to identifying at least one of the tools (141, 143, 144):
i) determine a respective tool position (PT1, PT3, PT4) for each identified tool (141, 143, 144) based on the origin location (PC) and the three-dimensional image data (Dimg3D), and
ii) exclusively control the grip device (115) to one or more of the respective tool positions (PT1, PT3, PT4) to perform a pick-up operation,
wherein the control unit (120) is further configured to produce an alert (A) in response to identifying at least one hose (152) at a position where, in a current stage of a procedure executed by the automatic milking arrangement, one of said tools (141, 142, 143, 144) should be present.

### Claim 2 (depends on 1)

The tool-pickup system according to claim 1, wherein the control unit (120) is configured to process the three-dimensional image data (Dimg3D) by searching for the tools (141, 142, 143, 144) in at least one predefined volume (L, R; 441, 442) within the view field (VF).

### Claim 3 (depends on 2)

The tool-pickup system according to claim 2, wherein the at least one predefined volume comprises a respective line (L) or arc for each of the tools (141, 142, 143, 144) along which respective line (L) expected tool positions (PET1, PET2, PET3, PET4) are defined within a range (R).

### Claim 4 (depends on 3)

The tool-pickup system according to claim 3, wherein, after having controlled the grip device (115) to perform a pick-up operation at a particular one of said tool positions (PT1, PT3, PT4), the control unit (120) is configured to exclude the predefined volume for said particular one tool position from a subsequent search for at least one remaining tool of said tools in the three-dimensional image data (Dimg3D).

### Claim 5 (depends on 2)

The tool-pickup system according to claim 2, wherein the at least one predefined volume comprises a respective area (441; 442) for each of the tools (141, 142) within which respective area (441; 442) expected tool positions (PET1, PET2) are defined.

### Claim 6 (depends on 5)

The tool-pickup system according to claim 5, wherein, after having controlled the grip device (115) to perform a pick-up operation at a particular one of said tool positions (PT1, PT3, PT4), the control unit (120) is configured to exclude the predefined volume for said particular one tool position from a subsequent search for at least one remaining tool of said tools in the three-dimensional image data (Dimg3D).

### Claim 7 (depends on 2)

The tool-pickup system according to claim 2, wherein, after having controlled the grip device (115) to perform a pick-up operation at a particular one of said tool positions (PT1, PT3, PT4), the control unit (120) is configured to exclude the predefined volume for said particular one tool position from a subsequent search for at least one remaining tool of said tools in the three-dimensional image data (Dimg3D).

### Claim 8 (depends on 1)

The tool-pickup system according to claim 1, wherein the tools comprise at least one of:
one or more teatcups and one or more cleaning cups.

### Claim 9 (depends on 1)

The tool-pickup system according to claim 1, wherein the tools comprise at least one teatcup and at least one cleaning cup.

### Claim 10

A method for picking up tools in an automatic milking arrangement, the method comprising:
controlling a robotic arm (110) to move a camera (130) arranged on the robotic arm (110) to an origin location (PC) from which at least one tool of the tools (141, 142, 143, 144) is expected to be visible within a view field (VF) of the camera (130);
using the camera (13) to register three-dimensional image data (Dimg3D) within the view field (VF) of the camera (130);
obtaining the three-dimensional image data (Dimg3D) registered by the camera (130) at the origin location (PC);
processing the three-dimensional image data (Dimg3D) using an image-based object identification algorithm to identify objects in a form of tools (141, 143, 144) and/or hoses (152), and
in response to identifying at least one tool of the tools (141, 143, 144):
i) determining a respective tool position (PT1, PT3, PT4) for each identified tool (141, 143, 144) based on the origin location (PC) and the three-dimensional image data (Dimg3D), and
ii) exclusively controlling a grip device (115) on the robotic arm (110) to one or more of the respective tool positions (PT1, PT3, PT4) to perform a pick-up operation of each respective identified tool (141, 143, 144); and
producing an alert (A) in response to identifying at least one hose (152) at a position where, in a current stage of a procedure executed by the automatic milking arrangement, one of said tools (141, 142, 143, 144) should be present.

### Claim 11 (depends on 10)

The method according to claim 10, comprising:
processing the three-dimensional image data (Dimg3D) by searching for each of the tools (141, 142, 143, 144) in at least one predefined volume (L, R; 441, 442) within the view field (VF).

### Claim 12 (depends on 11)

The method according to claim 11, wherein the at least one predefined volume is represented by a respective line (L) or arc for each of the tools (141, 142, 143, 144) along which respective line (L) expected tool positions (PET1, PET2, PET3, PET4) are defined within a range (R).

### Claim 13 (depends on 12)

The method according to claim 12, wherein, after having controlled the grip device (115) to perform a pick-up operation at a particular one of said tool positions (PT1, PT3, PT4), the method comprises excluding the predefined volume for said particular one tool position from a subsequent search for at least one remaining tool of said tools in the three-dimensional image data (Dimg3D).

### Claim 14 (depends on 11)

The method according to claim 11, wherein the at least one predefined volume is represented by a respective area (441; 442) for each of the tools (141, 142) within which area (441; 442) expected tool positions (PET1, PET2) are defined.

### Claim 15 (depends on 14)

The method according to claim 14, wherein, after having controlled the grip device (115) to perform a pick-up operation at a particular one of said tool positions (PT1, PT3, PT4), the method comprises excluding the predefined volume for said particular one tool position from a subsequent search for at least one remaining tool of said tools in the three-dimensional image data (Dimg3D).

### Claim 16 (depends on 11)

The method according to claim 11, wherein, after having controlled the grip device (115) to perform a pick-up operation at a particular one of said tool positions (PT1, PT3, PT4), the method comprises excluding the predefined volume for said particular one tool position from a subsequent search for at least one remaining tool of said tools in the three-dimensional image data (Dimg3D).

### Claim 17 (depends on 10)

The method according to claim 10, wherein the tools comprise at least one of:
one or more teatcups and one or more cleaning cups.

### Claim 18 (depends on 10)

A non-transitory data carrier (126) containing a computer program (127) loadable into a processing unit (125), the computer program (127) comprising software, when executed by the processing unit (125), causes the processing unit to perform the method according claim 10.