# US 10,864,633 B2 · Automated personalized feedback for interactive learning applications > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,864,633 B2 - Document ID: US10864633B2 - Status: Granted - Kind: B2 - Application number: 15/962,267 - Filed: 2018-04-25 - Published: 2018-11-01 - Granted: 2020-12-15 - Earliest claimed: 2017-04-28 - Assignees: Southe Autonomy Works, LLC - Applicants: Southie Autonomy Works, LLC - Inventors: Rahul D. Chipalkatty; Raphael Segal; Jay Ming Wong - Web page: https://patentgrep.com/patent/US10864633B2 ## CPC - B25J 9/163: {learning, adaptive, model based, rule based expert control} - B25J 9/0081: {with leader teach-in means} - B25J 9/1656: {characterised by programming, planning systems for manipulators} - B25J 9/1664: {characterised by motion, path, trajectory planning} - B25J 13/06: Control stands, e.g. consoles, switchboards - G05B 19/42: Recording and playback systems, i.e. in which the program is recorded from a cycle of operations, e.g. the cycle of operations being manually controlled, after which this record is played back on the same machine - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G05B 2219/35444: Gesture interface, controlled machine observes operator, executes commands - G05B 2219/40414: Man robot interface, exchange of information between operator and robot - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A robot-training system permits a user touch, click on or otherwise select items from a display projected in the actual workspace in order to define task goals and constraints for the robot. A planning procedure responds to task definitions and constraints, and creates a sequence of robot instructions implementing the defined tasks. ## Claims ### Claim 1 A method of training a robot situated in a workspace, the method comprising the steps of: computationally representing, by a controller, the robot in a 3D coordinate space corresponding to at least a portion of the workspace; detecting, by the controller, a user gesture within the workspace, the gesture indicating a location, and based on the detected gesture, projecting an interface onto the workspace at the indicated location; detecting, by the controller, gestural user selections of an element of the projected interface, the element of the projected interface corresponding to a robot control function involving manipulation of a physical object; and causing the robot to execute the control function. ### Claim 2 (depends on 1) The method of claim 1, wherein the user gesture is movement of a device, selections of elements being detected based on a position of the device relative to the projected interface. ### Claim 3 (depends on 2) The method of claim 2, wherein the device is a wand having a fiducial thereon. ### Claim 4 (depends on 1) The method of claim 1, wherein the control function includes at least one task goal and at least one task constraint. ### Claim 5 (depends on 4) The method of claim 4, further comprising the step of analyzing a recorded image of the workspace to computationally define constraints associated with the task goals. ### Claim 6 (depends on 4) The method of claim 4, further comprising the step of computing motion plans for the robot based on the task goals and the task constraints, the robot being caused to execute the control functions in accordance with the computed motion plans. ### Claim 7 (depends on 1) The method of claim 1, further comprising the step of, following the gestural user selection of an interface element, detecting gestural user selection of an item within the workspace upon which the robot control function is to operate, the robot executing the control function on the selected item. ### Claim 8 (depends on 4) The method of claim 4, wherein the at least one task goal comprises an object placement location. ### Claim 9 (depends on 8) The method of claim 8, wherein the task constraint comprises an obstacle between an initial object location and the object placement location. ### Claim 10 For use with a robot in a workspace including a projector and a camera, a robot controller comprising: a processor; a memory for storing images obtained by the camera and controller instructions executable by the processor to provide: perception and interaction modules configured for (i) computationally representing the robot in a 3D coordinate spacer corresponding to at least a portion of the workspace, (ii) analyzing at least one first stored image to detect a user gesture within the workspace, the gesture indicating a location, (iii) based on the detected movement, causing the projector to project an interface onto the workspace at the indicated location, and (iv) analyzing at least one second stored image obtained by the camera subsequent to the at least one first stored image to detect a gestural user selection of an element of the projected interface, the element of the projected interface corresponding to a robot control function involving manipulation of a physical object; and a planning module configured for computing robot instructions corresponding to the robot control function and causing the robot to execute the robot instructions. ### Claim 11 (depends on 10) The robot controller of claim 10, wherein the robot controller is within the robot. ### Claim 12 (depends on 10) The robot controller of claim 10, wherein the controller is external to the robot and communicates therewith via a network. ### Claim 13 (depends on 10) The robot controller of claim 10, wherein the user gesture is movement of a device, selections of elements being detected based on a position of the device relative to the projected interface. ### Claim 14 (depends on 10) The robot controller of claim 10, wherein the control function includes at least one task goal and at least one task constraint. ### Claim 15 (depends on 14) The robot controller of claim 14, wherein the perception module is further configured for analyzing a stored image of the workspace to computationally define constraints associated with the task goals. ### Claim 16 (depends on 14) The robot controller of claim 14, wherein the planning module is further for computing motion plans for the robot based on the task goals and the task constraints. ### Claim 17 (depends on 10) The robot controller of claim 10, wherein the perception module is further configured for detecting gestural user selection of an item within the workspace upon which the robot control function is to operate. ### Claim 18 (depends on 14) The robot controller of claim 14, wherein the at least one task goal comprises an object placement location. ### Claim 19 (depends on 18) The robot controller of claim 18, wherein the task constraint comprises an obstacle between an initial object location and the object placement location.