Intelligent material completeness detection and configuration method based on digital twin and augmented reality (AR)

An intelligent material completeness detection and configuration method based on digital twin and augmented reality (AR), comprising the following steps:

• S1:
• constructing a digital twin model base of incomplete material;
• S2:
• constructing a digital twin knowledge base of incomplete material;
• S3:
• importing the digital twin model base of incomplete material in step S1 and the digital twin knowledge base of incomplete material in step S2 respectively into a digital twin model database and a digital twin knowledge database in a digital twin system database;
• S4:
• sending materials to be detected to an automated visual detection platform, acquiring images of the materials, performing completeness detection on the materials using visual algorithms such as image preprocessing and feature point matching, and sorting out incomplete materials;
• S5:
• recording, through an algorithm, incomplete material data acquired by the machine-vision detection in step S4, sorting the incomplete material data according to material types and incomplete types and locations, coding to form a three-level model tree, and setting a corresponding identification (ID) to form final incomplete material data;
• S6:
• uploading the incomplete material data sorted in step S5 to an incomplete material information database in the digital twin system database;
• S7:
• matching data information in the digital twin model database, the digital twin knowledge database and the incomplete material information database in the digital twin system database according to a matching rule in a digital twin relational database to obtain a configuration plan for the incomplete materials;
• S8:
• performing AR-based material perception and reconstruction:
• manually configuring the incomplete materials, extracting a feature of a configuration scene through an AR device, and performing material perception and reconstruction by the AR device by:
• identifying and tracking materials through a line of sight of human eyes, performing three-dimensional (3D) reconstruction of incomplete materials through environment sensing cameras, and sensing, by an inertial sensor, directions of the AR cameras to calculate positions and attitudes of the AR cameras;
• S9:
• inputting the configuration plan for the incomplete materials obtained in step S7 into the AR device for coordinates matching with the configuration scene of the incomplete material acquired by the AR-based material perception and reconstruction method in step S8;
• and if the match succeeds, extracting and matching image feature points of virtual-real scenes, and further correcting the positions and attitudes in fields of view of the AR cameras, such that an incomplete material configuration guide image and a real incomplete material configuration image are superimposed and displayed in AR glasses to complete a virtual-real fusion;
• and
• S10:
• allowing, according to the configuration plan and an AR interactive prompt displayed on the AR device, a worker to rapidly and accurately find a location of an incomplete material, determine a problem type and rapidly configuring the incomplete material according to a prompt of the configuration plan for the incomplete materials.