Auto seed: an automatic crawler seeds adaptation mechanism

A system comprising memory for storing instructions, and a processor configured to execute the instructions to:

• train, using machine learning, an auto-seed model to predict a usefulness of relationship data;
• crawl for documents based on a seed Uniform Resource Locator (URL) list;
• generate relations data from the documents using a Natural Language Processing (NLP) model, wherein the relations data comprises a plurality of entities and relationship data indicating how the plurality of entities are related to each other, and wherein the NLP model has been trained using sample annotated documents to be able to perform automated annotations on the documents to generate the relations data;
• analyze the relations data using the auto-seed model to predict the usefulness of relationship data generated from the documents, wherein the relations data is deemed useful when more than a predetermined number of the entities in the relations data generated from the documents have a direct relationship indicating how a first entity is related to a second entity;
• and
• modify the seed URL list based on the predicted usefulness of the relationship data wherein modifying the seed URL list comprises expanding the seed URL list to include a new seed URL, the new seed URL predicted to produce quality relations data by the NLP model.

A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor of a system to cause the system to:

• train, using machine learning, an auto-seed model to predict a usefulness of relationship data;
• crawl for documents based on a seed Uniform Resource Locator (URL) list;
• generate relations data from the documents using a Natural Language Processing (NLP) model, wherein the relations data comprises a plurality of entities and relationship data indicating how the plurality of entities are related to each other, and wherein the NLP model has been trained using sample annotated documents to be able to perform automated annotations on the documents to generate the relations data;
• analyze the relations data using the auto-seed model to predict the usefulness of relationship data generated from the documents, wherein the relations data is deemed useful when more than a predetermined number of the entities in the relations data generated from the documents have a direct relationship indicating how a first entity is related to a second entity;
• and
• modify the seed URL list based on the predicted usefulness of the relationship data, wherein modifying the seed URL list comprises expanding the seed URL list to include a new seed URL, the new seed URL predicted to produce quality relations data by the NLP model.