Batch Data Ingestion In Database Systems

A method for data ingestion into a database, the method comprising:

• determining a notification channel associated with a database source, wherein the database source receives user files comprising data to be ingested into the database;
• associating one or more pipes with the notification channel, wherein each of the one or more pipes is assigned one or more target tables of the database and directs the data from the database source to an applicable target table of the one or more assigned target tables;
• and
• deploying a resource manager assigned to the notification channel, wherein the resource manager is configured to:
• determine a new user file has been added to the database source and received by the notification channel;
• identify new data in the new user file;
• and
• assign a pipe of the one or more pipes to the new data for the pipe to retrieve the new data and direct the new data to an applicable target table of the database.

A processor that is programmable to execute instructions stored in non-transitory computer readable storage media, the instructions comprising:

• determining a notification channel associated with a database source, wherein the database source receives user files comprising data to be ingested into the database;
• associating one or more pipes with the notification channel, wherein each of the one or more pipes is assigned one or more target tables of the database and directs the data from the database source to an applicable target table of the one or more assigned target tables;
• and
• deploying a resource manager assigned to the notification channel, wherein the resource manager is configured to:
• determine a new user file has been added to the database source and received by the notification channel;
• identify new data in the new user file;
• and
• assign a pipe of the one or more pipes to the new data for the pipe to retrieve the new data and direct the new data to an applicable target table of the database.