Back to results
US 11,113,151 B1Granted
Cluster diagnostics data for distributed job execution
AssigneeSnowflake Inc.
InventorsBing Li, Zihan Li, Edward Ma, Mingli Rui, Srinath Shankar
Life of this patent
Estimating the term…
Abstract
A shared database platform can interface with a cluster computing platform over a network through a database connector and one or more cluster connectors. The data transferred over the network can include telemetry metadata that can be distributed to execution nodes of the cluster computing platform for generation and transmission of cluster data to the shared database platform.
Claims20
A method comprising:
- receiving, by a computing cluster, an application for processing by nodes of the computing cluster, the nodes including a driver node and a plurality of execution nodes for processing of tasks of the application;
- distributing, by the driver node, the tasks of the application for processing by the plurality of execution nodes;
- processing the tasks by the plurality of execution nodes;
- identifying, by the computing cluster, one or more errors in the application received by the computing cluster;
- in response to the one or more errors, receiving, by the driver node, telemetry metadata for access to a telemetry network service of a distributed database that provides data to the computing cluster through a database connector, the driver node receiving the telemetry metadata through the database connector of the distributed database;
- distributing, by the driver node, the telemetry metadata to the plurality of execution nodes for re-processing;
- re-processing the tasks of the application by the plurality of execution nodes;
- and
- transmitting, by the plurality of execution nodes, log data to the telemetry network service using one or more database connectors of the plurality of execution nodes, the log data being generated by the plurality of execution nodes while re-processing the tasks of the application.
6 more elements
A system comprising:
- one or more processors of a machine;
- and
- a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
- receiving, by a computing cluster, an application for processing by nodes of the computing cluster, the nodes including a driver node and a plurality of execution nodes for processing of tasks of the application;
- distributing, by the driver node, the tasks of the application for processing by the plurality of execution nodes;
- processing the tasks by the plurality of execution nodes;
- identifying, by the computing cluster, one or more errors in the application received by the computing cluster;
- in response to the one or more errors, receiving, by the driver node, telemetry metadata for access to a telemetry network service of a distributed database that provides data to the computing cluster through a database connector, the driver node receiving the telemetry metadata through the database connector of the distributed database;
- distributing, by the driver node, the telemetry metadata to the plurality of execution nodes for re-processing;
- re-processing the tasks of the application by the plurality of execution nodes;
- and
- transmitting, by the plurality of execution nodes, log data to the telemetry network service using one or more database connectors of the plurality of execution nodes, the log data being generated by the plurality of execution nodes while re-processing the tasks of the application.
9 more elements
A non-transitory machine storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
- receiving, by a computing cluster, an application for processing by nodes of the computing cluster, the nodes including a driver node and a plurality of execution nodes for processing of tasks of the application;
- distributing, by the driver node, the tasks of the application for processing by the plurality of execution nodes;
- processing the tasks by the plurality of execution nodes;
- identifying, by the computing cluster, one or more errors in the application received by the computing cluster;
- in response to the one or more errors, receiving, by the driver node, telemetry metadata for access to a telemetry network service of a distributed database that provides data to the computing cluster through a database connector, the driver node receiving the telemetry metadata through the database connector of the distributed database;
- distributing, by the driver node, the telemetry metadata to the plurality of execution nodes for re-processing;
- re-processing the tasks of the application by the plurality of execution nodes;
- and
- transmitting, by the plurality of execution nodes, log data to the telemetry network service using one or more database connectors of the plurality of execution nodes, the log data being generated by the plurality of execution nodes while re-processing the tasks of the application.
6 more elements
Dependencies are inferred from the claim text.