Data processing method and system for the preparation of a dataset
AssigneeBull SAS
InventorKaoutar SGHIOUER
Life of this patent
Estimating the term…
Abstract
The invention relates to a data processing method (100) for preparing a dataset comprising a data processing unit receiving a first plurality (2, 3, 4) of data input streams to prepare the output dataset (5), the plurality of data input streams and the output dataset being different, said method comprising: A step of standardizing (120) the plurality of data input streams, A step of encoding (130) the normalized data, A step of preprocessing (140) the missing data, A step of transmitting (190) a preprocessed dataset. The invention further relates to a data processing system, and a recording medium on which the data processing program is recorded.
Claims16
A data processing method for preparing a dataset implemented by a computing device comprising a data processing unit, said data processing unit receiving a plurality of data input streams from different data providers, and comprising a group of interconnected processing modules through which the first plurality of data streams pass to prepare the output dataset, the plurality of data input streams and the output dataset being different, said method comprising:
- A step of standardizing the plurality of data input streams, by a data descriptive analysis module, from the plurality of data input streams, for normalizing the data at the input of the processing unit,
- A step of encoding the normalized data, by the data descriptive analysis module, for transforming the plurality of normalized data input streams into a plurality of normalized and encrypted data input streams,
- A step of preprocessing the dataset, by a data processing module, from the plurality of normalized and encrypted data input streams, said preprocessing step comprising:
- A step of data augmentation for the completion of the missing data taking into account the consistency of the normalized and encrypted data with predefined correlation tables,
- A step of removing outliers from the plurality of normalized and encrypted data input streams, by processing by predetermined functions said normalized and encrypted data,
- A step of verifying the fit of the plurality of data input streams processed with the predetermined functions , said verification step comprising the data processing module providing feedback to take actions aiming at injecting the plurality of processed data input streams back into the data processing module when the fit of the plurality of data input streams processed with the predetermined functions is divergent,
- A step of transmitting a preprocessed dataset, said dataset including normalized and encrypted data as well as new data generated in the step of data augmentation and preferably new data replacing outliers.
A data processing system for preparing a dataset implemented by a computing device, receiving a plurality of data input streams from different data providers, the computing device comprising a data processing unit and comprising a group of interconnected processing modules through which the first plurality of data streams pass to prepare the output dataset, the plurality of data input streams and the output dataset being different, said data processing unit comprising:
- A data descriptive analysis module, configured to
- from the plurality of data input streams, standardize the data at the input of the processing unit, and
- transform the plurality of normalized data input streams into a plurality of normalized and encrypted data input streams,
- a dataset preprocessing module, configured to, from the plurality of normalized and encrypted data input streams:
- complete the missing data taking into account the consistency of the normalized and encrypted data with predefined correlation tables,
- remove outliers from the plurality of normalized and encrypted data input streams, by processing by predetermined functions said normalized and encrypted data,
- verify the fit of the plurality of data input streams processed with the predetermined functions, taking actions aiming at injecting the plurality of processed data input streams back into the data processing module when the fit of the plurality of data input streams processed with the predetermined functions is divergent,
- A data transmission means configured to transmit the preprocessed dataset, said dataset including normalized and encrypted data as well as new data generated and preferably new data replacing outliers from the plurality of data input streams processed by the data processing unit.
A data processing unit for preparing a dataset implemented by a computing device, receiving a plurality of data input streams from different data providers, and comprising a group of interconnected processing modules through which the plurality of data input streams pass to prepare the output dataset, the plurality of data input streams and the output dataset being different, said processing unit comprising:
- A data descriptive analysis module, configured to
- from the plurality of data input streams, standardize the data at the input of the processing unit, and
- transform the plurality of normalized data input streams into a plurality of normalized and encrypted data input streams,
- a dataset preprocessing module, configured to, from the plurality of normalized and encrypted data input streams:
- complete the missing data taking into account the consistency of the normalized and encrypted data with predefined correlation tables,
- remove outliers from the plurality of normalized and encrypted data input streams, by processing by predetermined functions said normalized and encrypted data,
- verify the fit of the plurality of data input streams processed with the predetermined functions, taking actions aiming at injecting the plurality of processed data input streams back into the data processing module when the fit of the plurality of data input streams processed with the predetermined functions is divergent,
- A data transmission means configured to transmit the preprocessed dataset, said dataset including normalized and encrypted data as well as new data generated and preferably new data replacing outliers from the plurality of data input streams processed by the data processing unit.
Dependencies are inferred from the claim text.