# US 2022/0027245 A1 · Data ingestion by distributed-computing systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0027245 A1
- Document ID: US20220027245A1
- Status: Published application
- Kind: A1
- Application number: 17/498,504
- Filed: 2021-10-11
- Published: 2022-01-27
- Earliest claimed: 2018-01-26
- Assignees: VMware, Inc.
- Applicants: VMware, Inc.
- Inventors: Sai Samrat KARLAPUDI; Mirza Mohsin BEG
- Web page: https://patentgrep.com/patent/US20220027245A1

## CPC

- G06F 16/27: Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F 17/40: Data acquisition and logging (for input to computer G06F3/00)
- G06F 9/45558: {Hypervisor-specific management and integration aspects}
- G06F 11/1456: {Hardware arrangements for backup}
- G06F 11/1464: {for networked environments}
- G06F 11/2048: {where the redundant components share neither address space nor persistent storage}
- G06F 11/2097: {maintaining the standby controller/processing unit updated (initialisation or re-synchronisation thereof G06F11/1658 and subgroups)}
- G06F 2009/4557: {Distribution of virtual machine instances; Migration and load balancing}
- G06F 2009/45595: {Network integration; Enabling network access in virtual machine instances}
- G06F 2201/805: Real-time
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Techniques for data ingestion by a distributed-computing system are provided. In one embodiment, data received from one or more data sources is processed at a management node of the distributed-computing system. The data is generated by one or more components of an information processing system external to the distributed-computing system. The data is stored at the management plane. The management plane selects, from a plurality of data cores, a data core to ingest the data. The plurality of data cores are stored across one or more data plane containers of a data plane of the distributed-computing system. The management plane processes the data to generate one or more event records corresponding to the data. The one or more event records are sent to the selected data core, which causes the data plane to store the one or more event records in the selected data core.

## Claims

### Claim 1

A method for data ingestion by a distributed-computing system implementing a management plane and a data plane, the method comprising:
at the management plane:
receiving data generated by one or more components of an information processing system;
replicating and storing the data across a plurality of separate management nodes of the management plane;
selecting, from a plurality of data cores, a data core to ingest the data;
processing the data to generate one or more event records corresponding to one or more log events that occurred at the information processing system; and
sending the one or more event records to the selected data core, wherein the sending causes the data plane to store the one or more event records in the selected data core.

### Claim 2 (depends on 1)

The method of claim 1, wherein receiving the data comprises receiving log data representing one or more log events that occurred at the information processing system.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
receiving, at the management plane, an ingestion request with the data, wherein the data is a payload of the ingestion request.

### Claim 4 (depends on 3)

The method of claim 3, wherein the data core is selected using information included in the received ingestion request.

### Claim 5 (depends on 1)

The method of claim 1, wherein the data is received from an agent of the information processing system, the method further comprising:
replicating the data across one or more separate management nodes of the management plane, including storing the replicated data at the one or more separate management nodes; and
upon replicating the data across the one or more separate management nodes, sending, by the management plane, a confirmation to the agent indicating that the data has been accepted.

### Claim 6 (depends on 5)

The method of claim 5, wherein the confirmation is sent to the agent prior to or while the one or more event records are being stored in the selected data core.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
at the management plane:
detecting a failure to store the one or more event records in the selected data core;
in response to detecting the failure, selecting, from the plurality of data cores, a second data core to ingest the data; and
sending the one or more event records to the selected second data core, including causing the data plane to store the one or more event records in the selected second data core.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
storing a back-up copy of the one or more event records in a back-up storage system external to the distributed-computing system; and
after storing the one or more event records in the selected data core and storing the back-up copy in the back-up storage system, removing the data from the management plane.

### Claim 9 (depends on 1)

The method of claim 1, wherein the data core is selected using metadata obtained from the data plane.

### Claim 10 (depends on 9)

The method of claim 9, wherein, for each respective data core of the plurality of data cores, the metadata includes information regarding available storage capacity of the respective data core, current availability of the respective data core for ingesting data, or ingestion processing capacity of a data plane container storing the respective data core.

### Claim 11 (depends on 1)

The method of claim 1, wherein the selected data core is stored in a first data plane container of the one or more data plane containers, and wherein selecting the data core further comprises:
determining, by the management plane, a current data ingestion rate of the first data plane container relative to one or more other data plane containers of the data plane, wherein the data core is selected based on the determined current data ingestion rate of the first data plane container relative to the one or more other data plane containers.

### Claim 12 (depends on 1)

The method of claim 1, wherein processing the data further comprises:
decompressing the data; and
parsing the decompressed data to generate the one or more event records.

### Claim 13 (depends on 12)

The method of claim 12, wherein:
parsing the decompressed data further comprises extracting, from the decompressed data, one or more fields for the one or more event records; and
causing the data plane to store the one or more event records in the selected data core further comprises causing the data plane to store the one or more fields in an index of the selected data core, the one or more fields stored in association with the one or more event records.

### Claim 14

A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a distributed-computing system, the distributed-computing system implementing a management plane and a data plane, the one or more programs including instructions for:
processing data received from one or more data sources, the data generated by one or more components of an information processing system external to the distributed-computing system;
replicating and storing the data across a plurality of separate management nodes of the management plane;
selecting, from a plurality of data cores, a data core to ingest the data;
processing the data to generate one or more event records corresponding one or more events that occurred at the information processing system; and
providing the one or more event records to the selected data core, wherein the providing causes the data plane to store the one or more event records in the selected data core.

### Claim 15 (depends on 14)

The computer-readable storage medium of claim 14, wherein the data is received from an agent of the information processing system, and wherein the one or more programs further include instructions for:
replicating the data across one or more separate management nodes of the management plane, including storing the replicated data at the one or more separate management nodes; and
upon replicating the data across the one or more separate management nodes, providing, by the management plane, a confirmation to the agent indicating that the data has been accepted.

### Claim 16 (depends on 14)

The computer-readable storage medium of claim 14, wherein the one or more programs further include instructions for:
detecting, by the management plane, a failure to store the one or more event records in the selected core;
in response to detecting the failure, selecting, from the plurality of data cores, a second data core to ingest the data; and
providing the one or more event records to the selected second data core, including causing the data plane to store the one or more event records in the selected second data core.

### Claim 17 (depends on 14)

The computer-readable storage medium of claim 14, wherein the one or more programs further include instructions for:
storing a back-up copy of the one or more event records in a back-up storage system external to the distributed-computing system; and
after storing the one or more event records in the selected data core and storing the back-up copy in the back-up storage system, removing the data from the management plane.

### Claim 18

A distributed-computing system for data ingestion, wherein the system implements a management plane and a data plane, the system comprising:
one or more processors; and
memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
processing data received from one or more data sources, the data generated by one or more components of an information processing system external to the distributed-computing system;
replicating and storing the data across a plurality of separate management nodes of the management plane;
selecting, from a plurality of data cores, a data core to ingest the data;
processing the data to generate one or more event records corresponding one or more events that occurred at the information processing system; and
providing the one or more event records to the selected data core, wherein the providing causes the data plane to store the one or more event records in the selected data core.

### Claim 19 (depends on 18)

The system of claim 18, wherein the data is received from an agent of the information processing system, and wherein the one or more programs further include instructions for:
replicating the data across one or more separate management nodes of the management plane, including storing the replicated data at the one or more separate management nodes; and
upon replicating the data across the one or more separate management nodes, providing, by the management plane, a confirmation to the agent indicating that the data has been accepted.

### Claim 20 (depends on 18)

The system of claim 18, wherein the one or more programs further include instructions for:
detecting, by the management plane, a failure to store the one or more event records in the selected core;
in response to detecting the failure, selecting, from the plurality of data cores, a second data core to ingest the data; and
providing the one or more event records to the selected second data core, including causing the data plane to store the one or more event records in the selected second data core.

### Claim 21 (depends on 18)

The system of claim 18, wherein the one or more programs further include instructions for:
storing a back-up copy of the one or more event records in a back-up storage system external to the distributed-computing system; and
after storing the one or more event records in the selected data core and storing the back-up copy in the back-up storage system, removing the data from the management plane.
