# US 2023/0267109 A1 · Indexing and relaying data to hot storage

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267109 A1
- Document ID: US20230267109A1
- Status: Published application
- Kind: A1
- Application number: 18/138,492
- Filed: 2023-04-24
- Published: 2023-08-24
- Earliest claimed: 2018-05-09
- Assignees: Palantir Technologies Inc.
- Applicants: Palantir Technologies Inc.
- Inventors: Robert Fink; Amr Al Mallah; Haithem Turki
- Web page: https://patentgrep.com/patent/US20230267109A1

## CPC

- G06F 16/1734: {Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs}
- G06F 16/2272: {Management thereof}
- G06F 16/13: File access structures, e.g. distributed indices (arrangements of input from, or output to, record carriers G06F3/06)
- G06F 16/212: {with details for data modelling support}
- G06F 16/2228: {Indexing structures}
- G06F 16/2379: {Updates performed during online database operations; commit processing}
- G06F 16/245: Query processing
- G06F 16/24568: {Data stream processing; Continuous queries}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method is disclosed. The method comprises receiving, from one or more search nodes of a distributed system, one or more requests for log data, the one or more search nodes being associated with one or more hot storage systems; identifying, from an index catalog, an indexed portion of the log data stored in a cold storage system of one or more cold storage systems based on at least part of the particular request, the index catalog containing pointers to indexed portions of the log data in the one or more cold storage systems, the indexing being performed by one or more indexing nodes independently from the receiving by the one or more search nodes; and sending the indexed portion to the one or more search nodes for storage in the associated one or more hot storage systems, wherein the method is performed using one or more processors.

## Claims

### Claim 1

A method comprising:
receiving, from one or more search nodes of a distributed system, one or more requests for log data,
the one or more search nodes being associated with one or more hot storage systems;
identifying, from an index catalog, an indexed portion of the log data stored in a cold storage system of one or more cold storage systems based on a particular request of the one or more requests,
the index catalog containing pointers to indexed portions of the log data in the one or more cold storage systems,
the log data being indexed by one or more indexing nodes independently from the receiving by the one or more search nodes;
sending the indexed portion to at least one search nodes of the one or more search nodes in response to the particular request for storage in the associated at least one hot storage systems,
wherein the method is performed using one or more processors.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
tracking availability of the one or more search nodes;
determining the at least one search nodes based on the availability of the one or more search nodes.

### Claim 3 (depends on 1)

The method of claim 1, the at least one search nodes comprising multiple search nodes.

### Claim 4 (depends on 1)

The method of claim 1, the particular request identifying timing of the log data, a type of the log data, an application, a system, or a server.

### Claim 5 (depends on 1)

The method of claim 1, the sending comprising fetching the indexed portion from the cold storage system.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
tracking availability of the indexed portion of the log data in the at least one hot storage systems;
receiving a new request for the log data;
serving the new request with a part of the log data in the at least one hot storage systems.

### Claim 7 (depends on 1)

The method of claim 1, the log data being immutable to the one or more search nodes.

### Claim 8 (depends on 1)

The method of claim 1, further comprising changing a first number of the one or more search nodes and updating a second number of the one or more indexing nodes independently.

### Claim 9 (depends on 8)

The method of claim 8, the updating being performed based on an amount or a streaming rate of the log data.

### Claim 10 (depends on 8)

The method of claim 8, the changing being performed based on a number of requests for the log data received over a period of time or an amount of time for which indexed portions have been stored at the one or more search nodes.

### Claim 11

One or more non-transitory computer-readable storage media storing one or more sequences of instructions which, when executed cause one or more processors to perform a method, the method comprising:
receiving, from one or more search nodes of a distributed system, one or more requests for log data,
the one or more search nodes being associated with one or more hot storage systems;
identifying, from an index catalog, an indexed portion of the log data stored in a cold storage system of one or more cold storage systems based on a particular request of the one or more requests,
the index catalog containing pointers to indexed portions of the log data in the one or more cold storage systems,
the log data being indexed by one or more indexing nodes independently from the receiving by the one or more search nodes;
sending the indexed portion to at least one search nodes of the one or more search nodes in response to the particular request for storage in the associated at least one hot storage systems.

### Claim 12 (depends on 11)

The one or more non-transitory computer-readable storage media of claim 11, the method further comprising:
tracking availability of the one or more search nodes;
determining the at least one search nodes based on the availability of the one or more search nodes.

### Claim 13 (depends on 11)

The one or more non-transitory computer-readable storage media of claim 11, the at least one search nodes comprising multiple search nodes.

### Claim 14 (depends on 11)

The one or more non-transitory computer-readable storage media of claim 11, the particular request identifying timing of the log data, a type of the log data, an application, a system, or a server.

### Claim 15 (depends on 11)

The one or more non-transitory computer-readable storage media of claim 11, the sending comprising fetching the indexed portion from the cold storage system.

### Claim 16 (depends on 11)

The one or more non-transitory computer-readable storage media of claim 11, the method further comprising:
tracking availability of the indexed portion of the log data in the at least one hot storage systems;
receiving a new request for the log data;
serving the new request with a part of the log data in the at least one hot storage systems.

### Claim 17 (depends on 11)

The one or more non-transitory computer-readable storage media of claim 11, the log data being immutable to the one or more search nodes.

### Claim 18 (depends on 11)

The one or more non-transitory computer-readable storage media of claim 11, the method further comprising changing a first number of the one or more search nodes and updating a second number of the one or more indexing nodes independently.

### Claim 19 (depends on 18)

The one or more non-transitory computer-readable storage media of claim 18, the updating being performed based on an amount or a streaming rate of the log data.

### Claim 20 (depends on 18)

The one or more non-transitory computer-readable storage media of claim 18, the changing being performed based on a number of requests for the log data received over a period of time or an amount of time for which indexed portions have been stored at the one or more search nodes.
