# US 9,434,072 B2 · Vision-guided robots and methods of training them

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,434,072 B2
- Document ID: US9434072B2
- Status: Granted
- Kind: B2
- Application number: 13/621,687
- Filed: 2012-09-17
- Published: 2013-12-26
- Granted: 2016-09-06
- Earliest claimed: 2012-06-21
- Assignees: Rethink Robotics, Inc.
- Applicants: Rethink Robotics, Inc.
- Inventors: Christopher J. Buehler; Michael Siracusa
- Web page: https://patentgrep.com/patent/US9434072B2

## CPC

- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/0087: {Dual arms (double SCARA arms B25J9/043)}
- B25J 9/1671: {characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems}
- B25J 9/1676: {Avoiding collision or forbidden zones}
- B25J 9/1682: {Dual arm manipulator; Coordination of several manipulators}
- B25J 9/1697: {Vision controlled systems}
- B25J 11/0005: {Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means}
- B25J 13/084: {Tactile sensors (in general G01L5/16, G01L5/22)}
- G05B 19/42: Recording and playback systems, i.e. in which the program is recorded from a cycle of operations, e.g. the cycle of operations being manually controlled, after which this record is played back on the same machine
- G05B 19/423: Teaching successive positions by walk-through, i.e. the tool head or end effector being grasped and guided directly, with or without servo-assistance, to follow a path
- G06F 17/00: Digital computing or data processing equipment or methods, specially adapted for specific functions (information retrieval, database structures or file system structures therefor G06F16/00)
- G06F 18/28: Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
- G06N 20/00: Machine learning
- G06V 10/235: {based on user input or interaction}
- G06V 30/1914: {Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries, e.g. user dictionaries}
- G06V 30/194: References adjustable by an adaptive method, e.g. learning
- G05B 2219/39451: Augmented reality for robot programming
- G05B 2219/39466: Hand, gripper, end effector of manipulator
- G05B 2219/40116: Learn by operator observation, symbiosis, show, watch
- G06N 5/025: Extracting rules from data
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- Y02P 90/80: Management or planning
- Y10S 901/01: Mobile robot
- Y10S 901/03: Teaching system
- Y10S 901/09: Closed loop, sensor feedback controls arm movement
- Y10S 901/31: Gripping jaw
- Y10S 901/46: Sensing device
- Y10S 901/47: Optical
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- Y02P: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y10S: TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS

## Abstract

Via intuitive interactions with a user, robots may be trained to perform tasks such as visually detecting and identifying physical objects and/or manipulating objects. In some embodiments, training is facilitated by the robot's simulation of task-execution using augmented-reality techniques.

## Claims

### Claim 1

A method of training a robot to complete a specified task comprising performing an action on an object, the robot having at least one appendage for manipulating the object and a machine-vision system for capturing images within a camera field of view, the method comprising:
(a) selecting, by the robot, an object in the field of view and displaying the selection on a robot-controlled visual display, the displayed selection including a camera view of the selected object;
(b) overlaying, by the robot, a representation of an end-effector onto the camera view and displaying, on the display, performance of an action on the object by the end-effector;
(c) based on the displayed performance, providing feedback to the robot indicating whether the action is performed correctly, the robot modifying the performance of the action in response to the feedback; and
(d) repeating steps (b) through (c) until the action is performed correctly on the display.

### Claim 2 (depends on 1)

The method of claim 1, wherein the robot selects the action from a plurality of stored action templates.

### Claim 3 (depends on 1)

The method of claim 1, wherein the end-effector is a gripper and the action comprises gripping the object, the feedback indicating whether the displayed grippers have gripped a proper part of the displayed object.

### Claim 4 (depends on 1)

The method of claim 1, wherein the feedback comprises movement of the at least one appendage by the user.

### Claim 5

A vision-guided robot trainable, via interactions with a human trainer, to complete a specified task comprising performing an action on an object, the robot comprising:
at least one movable appendage comprising an end-effector for performing the action on the object;
a machine-vision system comprising at least one camera for capturing images within a camera field of view and an image-processing module for identifying objects within the captured images;
a user interface comprising (i) a robot-controlled visual display for visually displaying performance of the action on the object in a camera view thereof using a robot-generated graphic representation of the end-effector, and (ii) at least one input device for receiving feedback from the trainer indicating whether the action is performed correctly on the display; and
a control system for modifying performance of the action on the display in response to the feedback until the action is performed correctly on the display.

### Claim 6 (depends on 5)

The robot of claim 5, wherein the end-effector is a gripper.

### Claim 7 (depends on 5)

The robot of claim 5, wherein the control system comprises a data storage medium for storing a plurality of action templates.

### Claim 8 (depends on 5)

The robot of claim 5, wherein the at least one input device comprises the at least one appendage.

### Claim 9

A robot-implemented method of manipulating objects based on visual recognition thereof, the method comprising causing a robot to execute steps comprising:
(a) selecting an object in a camera field of view;
(b) computationally selecting a visual model from among a plurality of computer-implemented visual models, each of the visual models including an associated algorithm, the associated algorithm for each visual model being distinct from the other algorithms;
(c) using the selected visual model to computationally identify the object based on a stored representation thereof associated with the model;
(d) determining whether an object-specific manipulation routine is stored in association with the stored representation; and
(e) if so, executing the object-specific manipulation routine, and if not, executing a generic manipulation routine.

### Claim 10

A vision-guided robot for manipulating objects based on visual recognition thereof, the robot comprising:
at least one movable appendage for manipulating objects;
a camera for capturing an image within a camera field of view; and
a control system for (i) computationally selecting a visual model from among a plurality of computer-implemented visual models, each of the visual models including an associated algorithm, the associated algorithm for each visual model being distinct from the other algorithms, (ii) computationally identifying an object in the image based on a representation thereof stored in association with the selected visual model, (iii) determining whether an object-specific manipulation routine is stored in association with the stored representation, and (iv) if so, causing the at least one movably appendage to execute the object-specific manipulation routine, and if not, causing the at least one movable appendage to execute a generic manipulation routine.

### Claim 11 (depends on 10)

The robot of claim 10, wherein the control system comprises a data storage medium storing a plurality of visual models and object representations associated therewith.