Drone assisted adaptive robot control

A method for adaptively controlling a programmable robot, the method employing a drone device comprising at least one processor configured to execute computer program instructions for performing the method comprising:

• dynamically capturing a plurality of images of a target object to be operated on in a work environment from a plurality of directions by the drone device using one or more three-dimensional cameras configured in the drone device, while navigating independently in the directions around the target object in the work environment and accessing inaccessible areas with respect to the target object in the work environment;
• recording a spatial location of the target object by the drone device using the dynamically captured images of the target object in the work environment and sensor data obtained from one or more of a plurality of sensors configured in the drone device;
• dynamically calibrating the recorded spatial location of the target object in communication with the one or more three-dimensional cameras and the one or more of the sensors by the drone device based on environmental parameters of the work environment and a discernment of the target object;
• determining parts geometry of the target object by the drone device based on the dynamically calibrated spatial location of the target object, geometrical information of the target object, and predetermined dimensions of a plurality of objects stored in a parts geometry database;
• determining, by the drone device, a task trajectory of a task to be performed on the target object by the programmable robot based on the determined parts geometry of the target object, the predetermined dimensions of the objects stored in the parts geometry database, and predetermined task trajectories of the target objects stored in a procedure database;
• determining a collision-free robotic motion trajectory for the programmable robot by the drone device based on the determined task trajectory, a model of the target object extracted from the determined parts geometry, and the environmental parameters of the work environment;
• and
• dynamically transmitting the dynamically calibrated spatial location of the target object, the determined parts geometry, the determined task trajectory, and the determined collision-free robotic motion trajectory to the programmable robot by the drone device via a communication network to adaptively control the programmable robot to perform the task on the target object while accounting for misalignments of the target object in the work environment.

A system for adaptively controlling a programmable robot, the system comprising:

• a drone device comprising:
• one or more three-dimensional cameras mounted thereon for dynamically capturing a plurality of images of a target object to be operated on in a work environment from a plurality of directions, while the drone device navigates independently in the directions around the target object in the work environment and accesses inaccessible areas with respect to the target object in the work environment;
• and
• one or more sensors for detecting a configuration of the work environment and characteristics, positioning, and movements of the target object;
• and
• at least one processor configured to execute computer program instructions defined by modules deployed in one of the drone device and in an electronic device positioned external to and in operable communication with the drone device via a communication network, the modules comprising:
• a location recorder deployed in the drone device, the location recorder configured to record a spatial location of the target object using the dynamically captured images of the target object in the work environment and sensor data obtained from the one or more sensors configured in the drone device;
• a spatial location calibration module deployed in the drone device, the spatial location calibration module configured to dynamically calibrate the recorded spatial location of the target object in communication with the one or more three-dimensional cameras and the one or more sensors based on environmental parameters of the work environment and a discernment of the target object;
• a parts geometry determination module deployed in the one of the drone device and the electronic device, the parts geometry determination module configured to determine parts geometry of the target object based on the dynamically calibrated spatial location of the target object, geometrical information of the target object, and predetermined dimensions of a plurality of objects stored in a parts geometry database;
• a task trajectory determination module deployed in the one of the drone device and the electronic device, the task trajectory determination module configured to determine a task trajectory of a task to be performed on the target object by the programmable robot based on the determined parts geometry of the target object, the predetermined dimensions of the objects stored in the parts geometry database, and predetermined task trajectories of the target objects stored in a procedure database;
• a motion trajectory determination module deployed in the one of the drone device and the electronic device, the motion trajectory determination module configured to determine a collision-free robotic motion trajectory for the programmable robot based on the determined task trajectory, a model of the target object extracted from the determined parts geometry, and the environmental parameters of the work environment;
• and
• a data communication module deployed in the one of the drone device and the electronic device, the data communication module configured to dynamically transmit the dynamically calibrated spatial location of the target object, the determined parts geometry, the determined task trajectory, and the determined collision-free robotic motion trajectory to the programmable robot via the communication network to adaptively control the programmable robot to perform the task on the target object while accounting for misalignments of the target object in the work environment.

A non-transitory computer readable storage medium having embodied thereon, computer program codes comprising instructions executable by at least one processor for adaptively controlling a programmable robot, the computer program codes comprising:

• a first computer program code for processing a plurality of images of a target object to be operated on in a work environment, the images of the target object dynamically captured by a drone device from a plurality of directions using one or more three-dimensional cameras configured in the drone device, while navigating independently in the directions around the target object in the work environment and accessing inaccessible areas with respect to the target object in the work environment;
• a second computer program code for recording a spatial location of the target object using the dynamically captured images of the target object in the work environment and sensor data obtained from one or more of a plurality of sensors configured in the drone device;
• a third computer program code for dynamically calibrating the recorded spatial location of the target object in communication with the one or more three-dimensional cameras and the one or more of the sensors based on environmental parameters of the work environment and a discernment of the target object;
• a fourth computer program code for determining parts geometry of the target object based on the dynamically calibrated spatial location of the target object, geometrical information of the target object, and predetermined dimensions of a plurality of objects stored in a parts geometry database;
• a fifth computer program code for determining a task trajectory of a task to be performed on the target object by the programmable robot based on the determined parts geometry of the target object, the predetermined dimensions of the objects stored in the parts geometry database, and predetermined task trajectories of the target objects stored in a procedure database;
• a sixth computer program code for determining a collision-free robotic motion trajectory for the programmable robot based on the determined task trajectory, a model of the target object extracted from the determined parts geometry, and the environmental parameters of the work environment;
• and
• a seventh computer program code for dynamically transmitting the dynamically calibrated spatial location of the target object, the determined parts geometry, the determined task trajectory, and the determined collision-free robotic motion trajectory to the programmable robot via a communication network to adaptively control the programmable robot to perform the task on the target object while accounting for misalignments of the target object in the work environment.