An artificial intelligence system based on spatial-temporal information pairs, wherein the artificial intelligence system is configured to provide artificial intelligence software and hardware system support for humanoid robots, unmanned vehicles, intelligent glasses and inspection devices, and comprises a data acquisition end, data storage circuitry and data processing circuitry, wherein
- the data acquisition end covers a 720° spatial object environment and is constructed by using data acquisition devices, the data acquisition devices are configured for capturing and recording multi-dimensional data in the 720° spatial object environment in real time to form a series of spatial-temporal information pairs, the data acquisition devices comprise paired vision acquisition devices, paired auditory acquisition devices and paired olfactory acquisition devices, wherein the paired vision acquisition devices focus in a same direction or in a same approximate direction to simulate how human eyes observe any spatial object at the same time to form a stereo pair;
- the data storage circuitry is configured for storing multi-dimensional spatial-temporal information pairs collected at a same moment in pairs and according to a time series, and establishing a time-based index at the same time, so that the multi-dimensional spatial-temporal information pairs collected at the same moment form a data form that mutually confirms and complements and satisfies three-dimensional data processing;
- and
- the data processing circuitry is configured for processing and analyzing all of the multi-dimensional spatial-temporal information pairs in the data storage circuitry, performing synchronization and fusion processing to the multi-dimensional spatial-temporal information pairs from different types of the data acquisition devices to form a three-dimensional processing result or video stream with a depth of field, and identifying and understanding complex patterns and events in the 720° spatial object environment, wherein the complex patterns and events comprise identification and tracking of spatial objects and motion states of the spatial objects in a three-dimensional space, understanding and prediction of events, and comprehensive analysis and simulation of the 720° spatial object environment;
- the processing and analyzing of all of the multi-dimensional spatial-temporal information pairs in the data storage circuitry comprises:
- data pre-processing, comprising cleaning the collected multi-dimensional spatial-temporal information pairs, removing noise and irrelevant information, performing standardization processing to a vision spatial-temporal information pair, an auditory spatial-temporal information pair and an olfactory spatial-temporal information pair, and correcting and processing the stereo pair;
- spatial-temporal synchronization, comprising ensuring that data that are captured by the different types of the data acquisition devices are synchronous in time, and aligning the data of the different types of the data acquisition devices to ensure consistency of the data in a spatial-temporal relationship;
- multimodal data fusion, by performing three-dimensional (x, y, z) and four-dimensional (x, y, z, t) spatial-temporal relationship analyzing and restructuring by using information pairs, comprising performing feature extraction to the vision spatial-temporal information pair, the auditory spatial-temporal information pair and the olfactory spatial-temporal information pair by using a deep-learning model, and, by using a fusion algorithm, combining feature information of different modalities to form a richer representation;
- three-dimensional reconstruction, comprising, by using a stereo-matching algorithm, extracting depth information from the stereo pair, and, by referring to the depth information and vision data, reconstructing objects and scenes in the three-dimensional space to form the three-dimensional processing result or the video stream with the depth of field;
- object recognition and tracking, comprising, by using target detecting algorithms and tracking algorithms, tracking a motion state of an identified object;
- event comprehension and prediction, comprising, by analyzing a behavior pattern of an object and environmental change, understanding occurrence and development of an event, or predicting a future event by using a sequence predicting model;
- environment analysis and simulation, comprising comprehensively analyzing the multi-dimensional spatial-temporal information pairs, fully analyzing the 720° spatial object environment, and, by using a simulation technique, simulating the 720° spatial object environment to provide an interactive experience;
- and
- decision support, comprising, based on a result of the processing and analyzing, providing decision support for the artificial intelligence system.