Apparatus and method for action-level planning of coal mine robot based on digital cousin and mr

An apparatus for action-level planning of a coal mine robot based on digital cousin and MR (Mixed Reality), comprising:

• an information data acquisition module, configured to acquire scene information data, wherein the scene information data comprises single-frame red, green, and blue (RGB) images collected from an underground coal mine;
• a digital cousin scene information determining module, connected to the information data acquisition module, and configured to determine digital cousin scene information, wherein the digital cousin scene information is determined according to the scene information data based on a 3D asset library virtual model, and the 3D asset library virtual model is a preset model library, wherein the preset model library covers operational objects, operational tools, a terrain structure, mechanical devices and auxiliary devices, and faces an operation task of the underground coal mine;
• an action data determining module, connected to the digital cousin scene information determining module, and configured to input the digital cousin scene information to an imitation learning model to obtain action data, wherein the action data comprises motion pose information of the coal mine robot and operation pose information of a robotic arm, the imitation learning model is obtained by performing simulation training and zero-shot transfer based on a coal mine robot behavior model, the coal mine robot behavior model is obtained by performing reinforcement learning and simulation training on a coal mine robot model, and the coal mine robot model is a physical model constructed by a three-dimensional (3D) modeling tool based on entity parameters of the coal mine robot, wherein the entity parameters comprise:
• a size proportion, structure data and shape data;
• an action pre-planning module, connected to the action data determining module, and configured to determine an action pre-planning result according to the action data;
• an adjusting module, connected to the action pre-planning module, and configured to adjust the action pre-planning result by a multimodal interaction method to obtain an adjusted planning result;
• a spatiotemporal deduction module, connected to the adjusting module, and configured to process the adjusted planning result by a preset action planning algorithm, and perform spatiotemporal deduction according to a time dimension to obtain a spatiotemporal deduction result, wherein the spatiotemporal deduction result comprises a movement path of the coal mine robot and a movement trajectory of the robotic arm;
• and
• an action planning module, connected to the spatiotemporal deduction module, and configured to determine an action planning result according to the spatiotemporal deduction result, wherein the action planning result comprises position coordinates, pose information, and path points.