# US 2021/0081241 A1 · Pipeline task verification for a data processing platform

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0081241 A1
- Document ID: US20210081241A1
- Status: Published application
- Kind: A1
- Application number: 17/106,748
- Filed: 2020-11-30
- Published: 2021-03-18
- Earliest claimed: 2017-12-14
- Assignees: Palantir Technologies Inc.
- Applicants: Palantir Technologies Inc.
- Inventors: Kaan Tekelioglu
- Web page: https://patentgrep.com/patent/US20210081241A1

## CPC

- G06F 9/4881: {Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues}
- G05B 15/02: electric
- G06F 8/433: {Dependency analysis; Data or control flow analysis}
- G06F 9/3838: {Dependency mechanisms, e.g. register scoreboarding}
- G06F 9/4494: {data driven}
- G06F 9/451: Execution arrangements for user interfaces
- G06F 9/5072: {Grid computing}
- G06F 2209/486: Scheduler internals
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A pipeline task verification method and system is disclosed, and may use one or more processors. The method may comprise providing a data processing pipeline specification, wherein the data processing pipeline specification defines a plurality of data elements of a data processing pipeline. The method may further comprise identifying from the data processing pipeline specification one or more tasks defining a relationship between a first data element and a second data element. The method may further comprise receiving for a given task one or more data processing elements intended to receive the first data element and to produce the second data element. The method may further comprise verifying that the received one or more data processing elements receive the first data element and produce the second data element according to the defined relationship.

## Claims

### Claim 1

A computerized method, performed by a computing system having one or more hardware computer processors and one or more non-transitory computer readable storage device storing software instructions executable by the computing system to perform the computerized method comprising:
providing a data processing pipeline specification, wherein the data processing pipeline specification defines a plurality of data elements of a data processing pipeline;
identifying from the data processing pipeline specification one or more tasks defining or having a defined relationship between a first data element and a second data element;
displaying, on a graphical user interface, the data processing pipeline specification using placeholders, wherein the placeholders comprise one or more task placeholders for the one or more tasks;
receiving, from a user for a given task of the one or more tasks, one or more data processing elements intended to receive the first data element and to produce the second data element;
in response to receiving the one or more data processing elements for the given task from the user, verifying that the received one or more data processing elements receive the first data element and produce the second data element according to the defined relationship; and
providing, on the graphical user interface, a graphical indication of a verification result associated with the task placeholder for the given task.

### Claim 2 (depends on 1)

The method of claim 1, wherein the data processing pipeline specification defines pipeline dependencies of the plurality of data elements from which the one or more tasks may be identified.

### Claim 3 (depends on 2)

The method of claim 2, wherein the one or more tasks comprise metadata defining the relationship between the first data element and the second data element.

### Claim 4 (depends on 3)

The method of claim 3, wherein the metadata defines expected criteria for the one or more data processing elements and wherein the verification is based on whether the received one or more data processing elements agree with the expected criteria.

### Claim 5 (depends on 4)

The method of claim 4, wherein the metadata comprises one or more of:
expected dependencies of the one or more tasks;
an expected language or transformation; or
expected row and/or column names.

### Claim 6 (depends on 5)

The method of claim 5, wherein the placeholders further include one or more of data element placeholders representing one or more data elements.

### Claim 7 (depends on 6)

The method of claim 6, wherein the one or more data elements and/or tasks are user selectable to open an interface therefor, for receiving code or one or more links to code for performing one or more operations.

### Claim 8 (depends on 7)

The method of claim 7, further comprising providing on the graphical user interface a graphical indication of the verification result.

### Claim 9 (depends on 8)

The method of claim 8, wherein the graphical indication of the verification result is positioned in or adjacent a placeholder associated with the verification operation.

### Claim 10 (depends on 9)

The method of claim 9, further comprising providing on the graphical user interface an indication of a reason for a verification failure.

### Claim 11 (depends on 10)

The method of claim 10, wherein the indication of the reason for a verification failure is made responsive to a user hovering over, or selecting a placeholder associated with the verification operation.

### Claim 12 (depends on 11)

The method of claim 11, further comprising providing an indication of the expected criteria.

### Claim 13 (depends on 6)

The method of claim 6, wherein at least some of the metadata is automatically inferred based on the user defined graphical identification of dependencies.

### Claim 14 (depends on 13)

The method of claim 13, further comprising controlling an electrical system based on the verification.

### Claim 15 (depends on 13)

The method of claim 13, wherein at least one of the data elements is a first data set representing sensed data from one or more sensors, wherein the data processing element produces a second data set from the first data set, and wherein a positive verification controls the electrical system to perform a predetermined operation.

### Claim 16

A computing system comprising:
a hardware computer processor;
a non-transitory computer readable medium having software instructions stored thereon, the software instructions executable by the hardware computer processor to cause the computing system to perform operations comprising:
providing a data processing pipeline specification, wherein the data processing pipeline specification defines a plurality of data elements of a data processing pipeline;
identifying from the data processing pipeline specification one or more tasks defining or having a defined relationship between a first data element and a second data element;
displaying, on a graphical user interface, the data processing pipeline specification using placeholders, wherein the placeholders comprise one or more task placeholders for the one or more tasks;
receiving, from a user for a given task of the one or more tasks, one or more data processing elements intended to receive the first data element and to produce the second data element;
in response to receiving the one or more data processing elements for the given task from the user, verifying that the received one or more data processing elements receive the first data element and produce the second data element according to the defined relationship; and
providing, on the graphical user interface, a graphical indication of the verification result associated with the task placeholder for the given task.

### Claim 17 (depends on 16)

The computing system of claim 16, wherein the data processing pipeline specification defines pipeline dependencies of the plurality of data elements from which the one or more tasks may be identified.

### Claim 18 (depends on 17)

The computing system of claim 17, wherein the one or more tasks comprise metadata defining the relationship between the first data element and the second data element of the task.

### Claim 19 (depends on 18)

The computing system of claim 18, wherein the metadata defines expected criteria for the one or more data processing elements and wherein the verification is based on whether the received one or more data processing elements agree with the expected criteria.
