# US 11,285,607 B2 · Systems and methods for distributed training and management of AI-powered robots using teleoperation via virtual spaces > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,285,607 B2 - Document ID: US11285607B2 - Status: Granted - Kind: B2 - Application number: 16/511,492 - Filed: 2019-07-15 - Published: 2020-01-16 - Granted: 2022-03-29 - Earliest claimed: 2018-07-13 - Assignees: Massachusetts Institute of Technology - Applicants: Massachusetts Institute of Technology - Inventors: Daniela Rus; Jeffrey Lipton; Aidan Fay; Changhyun Choi - Web page: https://patentgrep.com/patent/US11285607B2 ## CPC - B25J 9/1689: {Teleoperation} - B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor} - B25J 9/163: {learning, adaptive, model based, rule based expert control} - G05B 2219/39311: Multilayer, MNN, four layer perceptron, sigmoidal neural network - G05B 2219/39536: Planning of hand motion, grasping - G05B 2219/40131: Virtual reality control, programming of manipulator - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract In some aspects, a system comprises a computer hardware processor and a non-transitory computer-readable storage medium storing processor-executable instructions for receiving, from one or more sensors, sensor data relating to a robot; generating, using a statistical model, based on the sensor data, first control information for the robot to accomplish a task; transmitting, to the robot, the first control information for execution of the task; and receiving, from the robot, a result of execution of the task. ## Claims ### Claim 1 A system, comprising: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: receiving, from one or more sensors, sensor data relating to a robot, wherein the sensor data comprises a voxel grid relating to the robot, wherein each voxel in the voxel grid is either occupied or not occupied, wherein the voxel grid is generated by capturing a three-dimensional point cloud relating to the robot, segmenting the three-dimensional point cloud into one or more object point clouds, and converting the one or more object point clouds into the voxel grid; generating, using a statistical model, based on the sensor data, first control information for the robot to accomplish a task; transmitting, to the robot, the first control information for execution of the task; and receiving, from the robot, a result of execution of the task. ### Claim 2 (depends on 1) The system of claim 1, wherein the processor-executable instructions cause the at least one computer hardware processor to further perform: in response to the result of execution of the task being unsuccessful: receiving, from a user, input relating to second control information for the robot to accomplish the task; transmitting, to the robot, the second control information for execution of the task; receiving, from the robot, the result of execution of the task; and updating the statistical model based on the sensor data, the second control information, and the result of execution of the task. ### Claim 3 (depends on 2) The system of claim 2, wherein the processor-executable instructions cause the at least one computer hardware processor to further perform: in response to the result of execution of the task being unsuccessful: updating a count of unsuccessful executions of tasks; and in response to the count of unsuccessful executions exceeding a threshold, receiving, from the user, the input relating to the second control information for the robot to accomplish the task. ### Claim 4 (depends on 2) The system of claim 2, wherein the processor-executable instructions cause the at least one computer hardware processor to further perform: generating, using the statistical model, a confidence value for the first control information; in response to the confidence value not exceeding a confidence threshold, receiving, from the user, the input relating to the second control information for the robot to accomplish the task; and in response to the confidence value exceeding the confidence threshold, transmitting, to the robot, the first control information for execution of the task. ### Claim 5 (depends on 1) The system of claim 1, wherein the first control information relates to a grasp pose for an end effector of the robot. ### Claim 6 (depends on 5) The system of claim 5, wherein the grasp pose comprises a position vector and an orientation vector for the end effector of the robot. ### Claim 7 (depends on 1) The system of claim 1, wherein the statistical model comprises a convolutional neural network including an input layer, one or more convolution layers, one or more pooling layers, one or more dense layers, and an output layer. ### Claim 8 (depends on 1) The system of claim 1, wherein the result of execution of the task indicates whether execution of the task was successful or unsuccessful. ### Claim 9 (depends on 8) The system of claim 8, wherein the result of execution of the task is based on an indication from a user regarding whether the execution of the task was successful or unsuccessful. ### Claim 10 (depends on 8) The system of claim 8, wherein the task relates to a grasp pose, wherein a torque across an end effector of the robot is measured, and wherein the result of execution of the task is successful or unsuccessful based on whether the measured torque exceeds or does not exceed a torque threshold. ### Claim 11 At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform: receiving, from one or more sensors, sensor data relating to a robot, wherein the sensor data comprises a voxel grid relating to the robot, wherein each voxel in the voxel grid is either occupied or not occupied, wherein the voxel grid is generated by capturing a three-dimensional point cloud relating to the robot, segmenting the three-dimensional point cloud into one or more object point clouds, and converting the one or more object point clouds into the voxel grid; generating, using a statistical model, based on the sensor data, first control information for the robot to accomplish a task; transmitting, to the robot, the first control information for execution of the task; and receiving, from the robot, a result of execution of the task. ### Claim 12 (depends on 11) The computer-readable storage medium of claim 11, wherein the processor-executable instructions cause the at least one computer hardware processor to further perform: in response to the result of execution of the task being unsuccessful: receiving, from a user, input relating to second control information for the robot to accomplish the task; transmitting, to the robot, the second control information for execution of the task; receiving, from the robot, the result of execution of the task; and updating the statistical model based on the sensor data, the second control information, and the result of execution of the task. ### Claim 13 (depends on 12) The computer-readable storage medium of claim 12, wherein the processor-executable instructions cause the at least one computer hardware processor to further perform: in response to the result of execution of the task being unsuccessful: updating a count of unsuccessful executions of tasks; and in response to the count of unsuccessful executions exceeding a threshold, receiving, from the user, the input relating to the second control information for the robot to accomplish the task. ### Claim 14 (depends on 12) The computer-readable storage medium of claim 12, wherein the processor-executable instructions cause the at least one computer hardware processor to further perform: generating, using the statistical model, a confidence value for the first control information; in response to the confidence value not exceeding a confidence threshold, receiving, from the user, the input relating to the second control information for the robot to accomplish the task; and in response to the confidence value exceeding the confidence threshold, transmitting, to the robot, the first control information for execution of the task. ### Claim 15 (depends on 11) The computer-readable storage medium of claim 11, wherein the first control information relates to a grasp pose for an end effector of the robot, wherein the grasp pose comprises a position vector and an orientation vector for the end effector of the robot. ### Claim 16 (depends on 11) The computer-readable storage medium of claim 11, wherein the result of execution of the task indicates whether execution of the task was successful or unsuccessful, wherein the result of execution of the task is based on an indication from a user regarding whether the execution of the task was successful or unsuccessful. ### Claim 17 (depends on 11) The computer-readable storage medium of claim 11, wherein the result of execution of the task indicates whether execution of the task was successful or unsuccessful, wherein the task relates to a grasp pose, wherein a torque across an end effector of the robot is measured, and wherein the result of execution of the task is successful or unsuccessful based on whether the measured torque exceeds or does not exceed a torque threshold. ### Claim 18 A method, comprising: receiving, from one or more sensors, sensor data relating to a robot, wherein the sensor data comprises a voxel grid relating to the robot, wherein each voxel in the voxel grid is either occupied or not occupied, wherein the voxel grid is generated by capturing a three-dimensional point cloud relating to the robot, segmenting the three-dimensional point cloud into one or more object point clouds, and converting the one or more object point clouds into the voxel grid; generating, using a statistical model, based on the sensor data, first control information for the robot to accomplish a task; transmitting, to the robot, the first control information for execution of the task; and receiving, from the robot, a result of execution of the task.