System and method to reconstruct human motion for mobile robot teleoperation using shared control

A system to reconstruct human motion for mobile robot teleoperation using a shared control, comprising:

• a memory storing instructions;
• one or more communication interfaces;
• and
• one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions:
• acquiring by a control unit of a mobile robot an input feed of a human operator, wherein the input feed is acquired during an assistance provided by the human operator to the mobile robot, to perform a task in a remote environment, wherein the input feed comprises one or more actions of the human operator providing the assistance to perform the task comprising at least one goal among a plurality of goals, wherein the remote environment is shared or not shared between the human operator and the mobile robot, wherein at least one goal intended by the human in the remote environment is determined by performing the steps of:
• computing a goal score for each goal among the plurality of goals based on (i) a distance between instantaneous human operator position and each goal position, and (ii) human operator velocity towards each goal;
• computing a goal confidence from the corresponding goal score of each goal to determine the human operator intended goal;
• and
• computing a long term confidence using an exponential smoothing of the goal confidence to transfer control from the human operator to the mobile robot to reach at least one goal intended by the human operator based on the long term confidence in the remote environment;
• receiving by the control unit of the mobile robot executed in the remote environment, (i) a delayed kinematic state associated with a plurality of measurements of the human operator (ii) a timestamp of the delayed kinematic state, wherein the plurality of measurements corresponds to an arm position, a shoulder velocity, an elbow angular velocity, and a wrist angular velocity;
• and
• reconstructing by the control unit of the mobile robot to follow a trajectory of the human operator towards the intended goal in the remote environment, wherein the step comprises:
• identifying a future trajectory of the human operator based on (i) the delayed kinematic state associated with the plurality of measurements of the human operator and (ii) the timestamp of the delayed kinematic state to determine the intended goal;
• determining at least one goal intended by the human operator in the remote environment based on (i) a previously occurred state associated with the plurality of measurements, (ii) the current kinematic state associated with the plurality of measurements of the human, (iii) the future trajectory of the human operator, and (iv) a known position of the plurality of goals;
• generating at least one instruction by a model predictive control of the mobile robot to control the movement of the mobile robot to perform at least one of (i) following the trajectory of the human operator, and (ii) reaching the operator intended goal based on a joint angle position and a velocity of the mobile robot.

One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:

• acquiring by a control unit of a mobile robot executed an input feed of a human operator, wherein the input feed is acquired during an assistance provided by the human operator to the mobile robot, to perform a task in a remote environment, wherein the input feed comprises of one or more actions of the human operator providing the assistance to perform the task comprising of at least one goal among a plurality of goals, wherein the remote environment is shared or not shared between the human operator and the mobile robot, wherein at least one goal intended by the human in the remote environment is determined by performing the steps of:
• computing a goal score for each goal among the plurality of goals based on (i) a distance between instantaneous human operator position and each goal position, and (ii) human operator velocity towards each goal;
• computing a goal confidence from the corresponding goal score of each goal to determine the human operator intended goal;
• and
• computing a long term confidence using an exponential smoothing of the goal confidence to transfer control from the human operator to the mobile robot to reach at least one goal intended by the human operator based on the long term confidence in the remote environment;
• receiving by the control unit of the mobile robot executed in the remote environment, (i) a delayed kinematic state associated with a plurality of measurements of the human operator (ii) a timestamp of the delayed kinematic state, wherein the plurality of measurements corresponds to the position of operators shoulder, elbow and the wrist angular velocity;
• and
• reconstructing by the control unit of the mobile robot to follow a trajectory of the human operator towards the intended goal in the remote environment, wherein the step comprises:
• identifying a future trajectory of the human operator based on (i) the delayed kinematic state associated with the plurality of measurements of the human operator and (ii) the timestamp of the delayed kinematic state to determine the intended goal;
• determining at least one goal intended by the human operator in the remote environment based on (i) a previously occurred state associated with the plurality of measurements, (ii) the current kinematic state associated with the plurality of measurements of the human, (iii) the future trajectory of the human operator, and (iv) a known position of the plurality of goals;
• and
• generating at least one instruction by a model predictive control of the mobile robot to control the movement of the mobile robot to perform at least one of (i) following the trajectory of the human operator, and (ii) reaching the operator's intended goal based on a joint angle position and a velocity of the mobile robot.