Logical upstream preprocessing at edge node of data to be stored in a data lake
AssigneeBank of America Corporation
InventorAnirudh Kumar Sharma
Life of this patent
Estimating the term…
Abstract
Preprocessing of data destined for storage in a data lake is accomplished upstream, such as at edge nodes. The preprocessing includes filtering data that is deemed to be unnecessary for subsequent analytical use purposes. An initial intelligent determination is performed on whether a data feed is to be preprocessed at (i) the data lake, or (ii) upstream of the data lake, such as at an edge node. Once upstream preprocessing has been determined, an intelligent determination of which edge node is to be chosen for preprocessing is performed. The determination on which edge node is to be chosen for preprocessing is based on response times between the application server and the edge nodes and network bandwidth usage encountered by the network transmitting the data feed.
Claims20
A system for managing preprocessing of data at a plurality of edge nodes prior to transmission and storage in a data lake, the system comprising;
- the data lake configured to receive data feeds and store data in the data feeds for purposes of supporting a plurality of different types of analytical use cases;
- a plurality of application servers configured to transmit the data feeds to the data lake;
- the plurality of edge nodes in network communication with the data lake and the plurality of application servers, each edge node physically located proximate at least one of the application servers;
- a computing platform comprising a memory and at least one processing device in communication with the memory, wherein the memory stores instructions, executable by the least one processing device and configured to:
- analyze the data feeds transmitted from the plurality of application servers to determine whether each data feed requires preprocessing (i) at the data lake, or (ii) upstream of the data lake,
- in response to determining that preprocessing of one or more of the data feeds requires preprocessing upstream of the data lake:
- determine, for each of the one or more data feeds, which one of the plurality of edge nodes to preprocess a corresponding one of the one or more data feeds based on (i) response times between the application server transmitting the corresponding data feed and each of the plurality of edge nodes, and (ii) network bandwidth usage of a network connecting the application server transmitting the corresponding data feed and each of the plurality of edge nodes, and
- generate preprocessing logic for the one or more of the data feeds and transmit the preprocessing logic to the corresponding determined one of the plurality of edge nodes.
A computer-implemented method for managing preprocessing of data at a plurality of edge nodes prior to transmission and storage in a data lake configured to receive data feeds and store data in the data feeds for purposes of supporting a plurality of different types of analytical use cases, the method executable by one or more processing devices and comprising;
- analyzing the data feeds, transmitted from a plurality of application servers that are configured to transmit the data feeds to the data lake, to determine whether each data feed requires preprocessing (i) at the data lake, or (ii) upstream of the data lake;
- in response to determining that preprocessing of one or more of the data feeds requires preprocessing upstream of the data lake:
- determining, for each of the one or more data feeds, which one of the plurality of edge nodes to preprocess a corresponding one of the one or more data feeds based on (i) response times between the application server transmitting the corresponding data feed and each of the plurality of edge nodes, and (ii) network bandwidth usage of a network connecting the application server transmitting the corresponding data feed and each of the plurality of edge nodes, wherein the plurality of edge nodes are in network communication with the data lake and the plurality of application servers, and each edge node is physically located proximate at least one of the application servers;
- and
- generating preprocessing logic for the one or more of the data feeds and transmitting the preprocessing logic to the corresponding determined one of the plurality of edge nodes.
A computer program product including a non-transitory computer-readable medium, the non-transitory computer-readable medium comprising:
- a first set of codes for causing a computer to analyze data feeds, transmitted from a plurality of application servers that are configured to transmit the data feeds to a data lake, to determine whether each data feed requires preprocessing (i) at the data lake that is configured to receive the data feeds and store data in the data feeds for purposes of supporting a plurality of different types of analytical use cases, or (ii) upstream of the data lake;
- and
- a second set of codes for causing a computer to, in response to determining that preprocessing of one or more of the data feeds requires preprocessing upstream of the data lake:
- (1) determine, for each of the one or more data feeds, which one of a plurality of edge nodes to preprocess a corresponding one of the one or more data feeds based on (i) response times between the application server transmitting the corresponding data feed and each of the plurality of edge nodes, and (ii) network bandwidth usage of a network connecting the application server transmitting the corresponding data feed and each of the plurality of edge nodes, wherein the plurality of edge nodes are in network communication with the data lake and the plurality of application servers, and each edge node is physically located proximate at least one of the application servers, and (2) generate preprocessing logic for the one or more of the data feeds and transmitting the preprocessing logic to the corresponding determined one of the plurality of edge nodes.
Dependencies are inferred from the claim text.