Method for automatically mapping fields having similar characteristics

A method for automatically mapping fields having similar characteristics from a first dataset to a second dataset, comprising:

• monitoring, by a processor, changes in a database structure during updates to the first dataset and extracting modified fields as metadata;
• performing data cleaning operations, by the processor, on the metadata to unify the formality of the metadata and standardize field descriptions;
• embedding multiple text fields of the metadata into a plurality of vectors respectively by at least one language embedding model for capturing semantic relations;
• building an index for a vector database, wherein the vector database is populated with the plurality of vectors from the second dataset;
• selecting a target field from the first dataset to identify the similar matches in the second dataset;
• averaging the plurality of vectors from the first dataset corresponding to the selected target field to obtain an average vector;
• performing a similarity search against the vector database by using the average vector to return a ranked list of the semantically similar vector embeddings in the second dataset, along with their associated fields;
• retrieving a set of top-N most similar fields in the second dataset to use as context for a large language model;
• feeding a prompt to the large language model, wherein the prompt comprises:
• target field information of the first dataset, a retrieved context from the set of top-N most similar fields in the second dataset, and instruction of providing the most similar fields;
• identifying, by the large language model, the set of top-N most similar fields of the second dataset based on the prompt to output a first most similar field in the second dataset having a similar characteristic to the field of the first dataset;
• partitioning, by the processor, the first dataset, the second datasets, and the plurality of vectors thereof, into a training set, a testing set, and a validation set;
• inputting, into a classification machine learning model, vector representations of field names, descriptions, average values, and data types, as input features associated with the fields of the first and second datasets;
• training, by the processor, the classification machine learning model using the training set, the training comprising using historical labeled field correspondence data as labels, wherein a label indicates a similarity or non-similarity between a field of the first dataset and a field of the second dataset;
• predicting, by the classification machine learning model, a similarity between a field of the first dataset and a field of the second dataset to output a second most similar field in the second dataset having a similar characteristic to the field in the first dataset;
• generating, by the processor, a mapping relationship between the first dataset and the second dataset if the first most similar field identified by the LLM is identical to the second most similar field predicted by the ML model;
• repeating, by the processor, the steps of selecting a target field through generating a mapping relationship, until the fields in the first dataset have all been processed, to create a list of mapping relationships between the first dataset and the second dataset;
• and
• storing, by the processor, the list of mapping relationships in a computer-readable storage medium.