Three-dimensional (3D) integrated teaching field system based on flipped platform and method for operating same

A three-dimensional (3D) integrated teaching field system based on a flipped platform, comprising:

• a device deployment module;
• a teaching resource matching module;
• an acquisition and processing module;
• and
• an edge computing module;
• wherein the device deployment module is configured to deploy displaying, acquiring, computing and interactive devices and a lighting system in a teaching activity area in a classroom to support the 3D integrated teaching field system;
• the teaching resource matching module is configured to select a teaching resource according to an instruction requested by a user in accordance with a weighting order of parameters, and realize a loading service of virtual teaching resources along a cloud-edge-end link based on local caching, hot updating by using an edge computing server, and cloud batch updating;
• the acquisition and processing module is configured to acquire environment of the teaching activity area and point cloud sequence data of a teacher using a red-green-blue-depth (RGB-D) camera, extract skeleton data of the teacher using a Mask region-based convolutional neural network (R-CNN) algorithm on the edge computing server, and feed the skeleton data back to a local processing module to extract skeletal joint points of the teacher and coordinates of the skeletal joint points of the teacher to perform grouping of the skeletal joint points;
• and
• the edge computing module is configured to receive the point cloud sequence data obtained by the acquisition and processing module, calculate and label the skeletal joint points of the teacher, analyze a motion trajectory of the skeletal joint points, detect states of the teacher in the teaching activity area, and track teaching behaviors of the teacher and changes thereof;
• wherein the skeletal joint points of the teacher are calculated and labeled through steps of:
• receiving, by the edge computing server, the point cloud sequence data obtained by the acquisition and processing module using a mobile edge computing (MEC) architecture;
• searching and positioning the skeletal data of the teacher in the point cloud sequence data using a random decision tree algorithm and a random forest algorithm;
• segmenting skeletal joints of the teacher in the point cloud sequence data using the Mask R-CNN algorithm;
• and calculating and labeling the skeletal joint points of the teacher using a nonlinear solver;
• the motion trajectory of the skeletal joint points is analyzed through steps of:
• defining codes of common teaching actions according to teaching action meaning of movement of the skeletal joint points;
• determining moving speeds and angles of body movements, facial expressions or hand gestures of the teacher according to the skeletal joint points at different moments on a time axis;
• and analyzing the motion trajectory of the skeletal joint points during a teaching process using a neural network algorithm;
• and
• the teaching behaviors of the teacher and the changes thereof are tracked through steps of:
• detecting states of the teacher in the teaching activity area at different moments on the time axis using 3D mapping and human body tracking technology, in combination with position and connection relationship of an inverse kinematics skeletal model;
• calculating and examining degrees of freedom of adjacent joints;
• and determining and examining behaviors of the teacher based on formed characteristics to track changes in the teaching behaviors of the teacher.