# US 2016/0243704 A1 · Image-based trajectory robot programming planning approach

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0243704 A1
- Document ID: US20160243704A1
- Status: Published application
- Kind: A1
- Application number: 15/031,779
- Filed: 2014-10-21
- Published: 2016-08-25
- Earliest claimed: 2013-10-25
- Inventors: Aleksandar Vakanski; Farrokh Janabi-Sharifi
- Web page: https://patentgrep.com/patent/US20160243704A1

## CPC

- B25J 9/1697: {Vision controlled systems}
- B25J 9/1664: {characterised by motion, path, trajectory planning}
- G05B 2219/36436: Arm follows movement of handheld device, camera detects, analyses motion
- G05B 2219/36442: Automatically teaching, teach by showing
- G05B 2219/39393: Camera detects projected image, compare with reference image, position end effector
- G05B 2219/40116: Learn by operator observation, symbiosis, show, watch
- G05B 2219/40607: Fixed camera to observe workspace, object, workpiece, global
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A method of programming at least one robot by demonstration comprising: performing at least one demonstration of at least one task in the Held of view of at least one fixed camera to obtain at least one observed task trajectory of at least one manipulated object, preferably at least one set of observed task trajectories; generating a generalized task trajectory from said at least one observed task trajectory, preferably from said at least one set of observed task trajectories; and executing said at least one task by said at least one robot in the field of view of said at least one fixed camera, preferably using image-based visual servoing to minimize the difference between the executed trajectory during said execution and the generalized task trajectory.

## Claims

### Claim 1

A method of programming at least one robot by demonstration comprising:
performing at least one demonstration of at least one task in the field of view of at least one fixed camera to obtain at least one observed task trajectory of at least one manipulated object, preferably at least one set of observed task trajectories;
generating a generalized task trajectory from said at least one observed task trajectory, preferably from said at least one set of observed task trajectories; and
executing said at least one task by said at least one robot in the field of view of said at least one fixed camera, preferably using image-based visual servoing to minimize the difference between the executed trajectory during said execution and the generalized task trajectory.

### Claim 2 (depends on 1)

The method of claim 1, whereas Cartesian positions and velocities of said at least one manipulated object are calculated from image measurements from said observed task trajectories.

### Claim 3 (depends on 2)

The method of claim 2, whereas said image-based visual servoing comprises following Cartesian positions and velocities of said at least one manipulated object by minimizing the difference between the observed trajectory during task execution and the generalized task trajectory.

### Claim 4 (depends on 1)

The method of claim 1, whereas a reference task trajectory is obtained by performing a smoothing of said set of observed task trajectories.

### Claim 5 (depends on 4)

The method of claim 4, whereas said smoothing is performed using a smoothing method selected from a group consisting of Kalman smoothing, hidden Markov model filtering, Gaussian mixture regression, dynamical systems smoothing and filtering.

### Claim 6 (depends on 4)

The method of claim 4, whereas said generalized task trajectory is obtained by performing a second order conic optimization of the reference task trajectory within a set of constraints imposed on said observed task trajectories and said at least one robot.

### Claim 7 (depends on 6)

The method of claim 6, whereas said constraints limit the generalized task trajectory to the field of view of said at least one camera.

### Claim 8 (depends on 6)

The method of claim 6, whereas said constraints limit the generalized task trajectory to the dexterous workspace of said at least one robot.

### Claim 9 (depends on 6)

The method of claim 6, whereas said constraints limit the maximum velocity of at least one end-effector of said at least one robot.

### Claim 10 (depends on 1)

The method of claim 1, whereas said at least one camera further comprises at least one sensor for the perception of depth of pixels for each image acquired from said at least one camera.

### Claim 11 (depends on 10)

The method of claim 10, whereas said depth of said pixels provides depth information on said observed task trajectories and during said image-based visual servoing.

### Claim 12 (depends on 1)

The method of claim 1, whereas said at least one robot further comprises at least one tactile force sensor for monitoring tactile force during said execution of said task by said at least one robot.

### Claim 13

A system for executing at least one task learned by at least one observation of at least one demonstration, comprising:
at least one robot with at least one movable end-effector;
at least one object to be manipulated by the at least one end-effector;
at least one fixed camera for observing said at least one demonstration of the at least one task, resulting in at least one observed task trajectory, preferably at least one set of observed task trajectories and observing the execution of said at least one task by said at least one robot;
at least one processing unit for generating a generalized task trajectory from said set of observed task trajectories;
at least one controller for servoing said at least one robot to reproduce the task with its end-effector in the field of view of said at least one said fixed camera using image-based visual servoing to minimize the difference between the observed trajectory during the execution and the generalized task trajectory.

### Claim 14 (depends on 13)

The system of claim 13, whereas said controller and said processing unit are combined in a single computer.

### Claim 15 (depends on 13)

The system of claim 13, whereas said image-based visual servoing is programmed so as to minimize the difference between the observed trajectories during execution and the generalized task trajectory.

### Claim 16 (depends on 13)

The system of claim 13, whereas a reference task trajectory is obtained by performing a smoothing of said set observed task trajectories.

### Claim 17 (depends on 16)

The system of claim 16, whereas said smoothing is performed using a smoothing method selected from a group consisting of Kalman smoother, hidden-Markov model smoother, Gaussian mixture regression, dynamical systems smoothing and filtering.

### Claim 18 (depends on 13)

The system of claim 13, whereas said generalized task trajectory is obtained by performing a second order conic optimization of the reference task trajectory within a set of constraints imposed on said observed task trajectories and said at least one robot.

### Claim 19 (depends on 13)

The system of claim 13, whereas said at least one robot, cameras, processing unit and controller are integrated into a single device.

### Claim 20 (depends on 13)

The system of claim 13, whereas at least one said camera includes at least one sensor for perception of the depth of pixels for each image acquired from said at least one camera.
