Separation of storage and compute in database management systems

A method for managing data in a database management system, the database management system comprising a first node, the first node being a master storage node for storing user data from a database, and a second node, the second node performing operations on user data stored in the database, wherein the operations include reading or writing the user data,

• wherein the method comprises configuring the second node to:
• write log data indicative of one or more operations that have been performed on user data in the database to a set of log files, wherein a said operation is made permanent in the database once the set of log files have been updated in the non-volatile storage of the second node to include the log data indicative of the operation and without requiring user data associated with the operation to be transmitted to the first node;
• segment the set of log files into a set of log file chunks, wherein each log file chunk is a portion of a log file in the set of log files;
• and
• for a said log file chunk, transmit user data from the database corresponding to the log file chunk to the first node, the transmitted user data comprising a binary large object, and thereafter transmit the log file chunk to the first node,
• wherein the second node segments the set of log files and transmits the log file chunks and associated user data to the first node asynchronously to its updating of the set of log files.

A database management system comprising:

• a first node, the first node comprising at least one processor and storage comprising first computer-executable instructions which, when executed by the at least one processor, cause the first node to implement a master storage node for storing user data from a database;
• and
• a second node, the second node performing operations on user data stored in the database, the operations comprising reading or writing the user data, wherein the second node comprises at least one processor and storage comprising second computer-executable instructions which, when executed by the least one processor of the second node, cause the second node to:
• write log data indicative of one or more operations that have been performed on user data in the database to a set of log files, wherein a said operation is made permanent in the database once the set of log files have been updated to include log data indicative of the operation and without requiring user data associated with the operation to be transmitted to the first node;
• segment the set of log files into a set of log file chunks, wherein each log file chunk is a portion of a log file in the set of log files;
• and
• for a said log file chunk, transmit user data corresponding to the log file chunk from the second node to the first node, the transmitted user data comprising a binary large object, and thereafter transmit the log file chunk to the first node,
• wherein the second node segments the set of log files and transmits the log file chunks and associated user data to the first node asynchronously to its updating of the set of log files.

A non-transitory computer-readable storage medium comprising computer executable instructions which, when executed by at least one processor, cause the at least one processor to:

• implement a first node, the first node being a master storage node for storing user data from a database;
• implement a second node for performing operations on data stored in the database, the operations comprising reading or writing the user data, the second node being configured to:
• write log data indicative of one or more operations that have been performed on user data in the database to a set of log files, wherein a said operation is made permanent in the database once the set of log files have been updated to include log data indicative of the operation and without requiring user data associated with the operation to be transmitted to the first node;
• segment the set of log files into a set of log file chunks, wherein each log file chunk is a portion of a log file in the set of log files;
• and
• for a said log file chunk, transmit user data corresponding to the log file chunk from the second node to the first node, the transmitted user data comprising a binary large object, and thereafter transmit the log file chunk to the first node,
• wherein the second node segments the set of log files and transmits the log file chunks and associated user data to the first node asynchronously to its updating of the set of log files.