# US 2019/0168390 A1 · Programming of a Robotic Arm Using a Motion Capture System > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0168390 A1 - Document ID: US20190168390A1 - Status: Published application - Kind: A1 - Application number: 16/273,361 - Filed: 2019-02-12 - Published: 2019-06-06 - Earliest claimed: 2013-03-05 - Assignees: X Development LLC - Applicants: X Development LLC - Inventors: Jeffrey Linnell - Web page: https://patentgrep.com/patent/US20190168390A1 ## CPC - B25J 9/1697: {Vision controlled systems} - B25J 9/1664: {characterised by motion, path, trajectory planning} - G05B 19/427: Teaching successive positions by tracking the position of a joystick or handle to control the positioning servo of the tool head, leader-follower control (G05B19/423 takes precedence) - G05B 2219/36442: Automatically teaching, teach by showing - G05B 2219/37563: Ccd, tv camera - Y10S 901/01: Mobile robot - Y10S 901/46: Sensing device - Y10S 901/47: Optical - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - Y10S: TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS ## Abstract An example method includes receiving position data indicative of position of a demonstration tool. Based on the received position data, the method further includes determining a motion path of the demonstration tool, wherein the motion path comprises a sequence of positions of the demonstration tool. The method additionally includes determining a replication control path for a robotic device, where the replication control path includes one or more robot movements that cause the robotic device to move a robot tool through a motion path that corresponds to the motion path of the demonstration tool. The method also includes providing for display of a visual simulation of the one or more robot movements within the replication control path. ## Claims ### Claim 1 A computer-implemented method comprising: receiving position data indicative of position of a demonstration tool; based on the received position data, determining a motion path of the demonstration tool, wherein the motion path comprises a sequence of positions of the demonstration tool; determining a replication control path for a robotic device, wherein the replication control path comprises one or more robot movements that cause the robotic device to move a robot tool through a motion path that corresponds to the motion path of the demonstration tool; and providing the replication control path. ### Claim 2 (depends on 1) The method of claim 1, wherein receiving the position data indicative of the position of the demonstration tool comprises receiving the position data from one or more sensors. ### Claim 3 (depends on 1) The method of claim 1, wherein receiving the position data indicative of the position of the demonstration tool comprises receiving the position data from one or more cameras. ### Claim 4 (depends on 3) The method of claim 3, wherein the position data received from the one or more cameras comprises data indicative of position of one or more control markers attached to the demonstration tool. ### Claim 5 (depends on 4) The method of claim 4, wherein the one or more control markers comprise one or more light emitting diode markers. ### Claim 6 (depends on 5) The method of claim 5, wherein: the one or more light emitting diode markers have a distinguishing characteristic that is distinguishable by the one or more cameras; and determining the motion path of the demonstration tool comprises identifying the distinguishing characteristic of the one or more light emitting diode markers using the one or more cameras. ### Claim 7 (depends on 1) The method of claim 1, wherein the position data indicative of the position of the demonstration tool is received only while a button on the demonstration tool indicative of when to record movements is pressed. ### Claim 8 (depends on 1) The method of claim 1, wherein the motion path of the robot tool comprises scaled positions from the motion path of the demonstration tool. ### Claim 9 (depends on 1) The method of claim 1, wherein: the demonstration tool is a camera used to film a scene; and the robot tool is a camera used by the robotic device to film a different iteration of the scene. ### Claim 10 (depends on 1) The method of claim 1, wherein: the demonstration tool is a manufacturing tool used to fabricate an output product; and the robot tool is a manufacturing tool used by the robotic device to fabricate a copy of the output product. ### Claim 11 (depends on 10) The method of claim 10, wherein the robot tool is a welding tool. ### Claim 12 (depends on 10) The method of claim 10, wherein the robot tool is a paint brush. ### Claim 13 (depends on 1) The method of claim 1, wherein the sequence of positions within the motion path of the demonstration tool comprises a sequence of positions resulting from a force of nature applied to the demonstration tool. ### Claim 14 (depends on 13) The method of claim 13, wherein the force of nature comprises gravity. ### Claim 15 (depends on 1) The method of claim 1, further comprising determining one or more refinements to the one or more robot movements within the replication control path in order to smooth one or more curves of robot movements within the replication control path before providing the replication control path. ### Claim 16 (depends on 1) The method of claim 1, further comprising: receiving one or more refinements to the one or more robot movements within the replication control path via a user interface; and updating a displayed visual simulation of the one or more robot movements within the replication control path to reflect the one or more refinements. ### Claim 17 A system comprising: a demonstration tool; and a control system configured to: receive position data indicative of position of the demonstration tool; based on the received position data, determine a motion path of the demonstration tool, wherein the motion path comprises a sequence of positions of the demonstration tool; determine a replication control path for a robotic device, wherein the replication control path comprises one or more robot movements that cause the robotic device to move a robot tool through a motion path that corresponds to the motion path of the demonstration tool; and provide the replication control path. ### Claim 18 (depends on 17) The system of claim 17, further comprising one or more sensors, wherein the position data indicative of the position of the robot tool is received from the one or more sensors. ### Claim 19 (depends on 18) The system of claim 18, wherein the one or more sensors comprise one or more cameras. ### Claim 20 A non-transitory computer readable medium having stored therein instructions, that when executed by a computing system, cause the computing system to perform functions comprising: receiving position data indicative of position of a demonstration tool; based on the received position data, determining a motion path of the demonstration tool, wherein the motion path comprises a sequence of positions of the demonstration tool; determining a replication control path for a robotic device, wherein the replication control path comprises one or more robot movements that cause the robotic device to move a robot tool through a motion path that corresponds to the motion path of the demonstration tool; and providing the replication control path.