# US 11,907,196 B1 · Generating observability metrics for data lake usage based on data layer activity logs

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,907,196 B1
- Document ID: US11907196B1
- Status: Granted
- Kind: B1
- Application number: 18/326,889
- Filed: 2023-05-31
- Granted: 2024-02-20
- Earliest claimed: 2023-05-31
- Assignees: Intuit Inc.
- Applicants: Intuit Inc.
- Inventors: Saikiran Sri Thunuguntla; Keshav Bhashyam; Sreenivasulu Nallapati; Vijaykumar Hiremath
- Web page: https://patentgrep.com/patent/US11907196B1

## CPC

- G06F 16/2282: {Tablespace storage structures; Management thereof}
- G06F 16/25: Integrating or interfacing systems involving database management systems
- G06F 16/113: {Details of archiving (lifecycle management in storage systems G06F3/0649; point-in-time backing up or restoration of persistent data G06F11/1446)}
- G06F 16/21: Design, administration or maintenance of databases
- G06F 16/285: {Clustering or classification}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Incoming data requests from the perspective of the data lake, are gathered and analyzed to determine the usage of the data. Using the perspective of the data lake avoids the technical challenge of analyzing data usage by different computation points, which are at different locations, perform hard-to-track different operations, and are often reachable only through complicated access protocols. Another technical challenge of mapping between an object path and a table path is solved by generating object path datasets and table path datasets at different levels of abstractions. A comparison is performed, iteratively, from a lower level of granularity and the granularity is increased in the progressive steps. Matches from iterations are unionized to generate a final matching data. Observability metrics are generated using final matching data and are used to perform downstream operations such as controlling data table access, moving data tables to cold storage, decommissioning unused pipelines, etc.

## Claims

### Claim 1

A computer-implemented method comprising:
generating an object path dataset comprising a plurality of rows of object paths for accessing corresponding data tables in a data lake;
generating a table path dataset comprising a plurality of rows of table paths for physical addresses of the corresponding data tables in the data lake;
expanding columns of the object path dataset to generate an expanded object path dataset by adding a plurality of additional columns, each additional column including object paths with a portion of information stripped away from corresponding object paths in a previous column;
expanding rows of the table path dataset to generate an expanded table path dataset by adding an additional row for each original row, the additional row comprising a corresponding table path without partition information;
iteratively matching the columns of the expanded object path dataset to the original rows and the additional rows; and
generating observability metrics of access of the data tables based on the iterative matching.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
classifying a first data table as having low usage based on the observability metrics.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, further comprising:
changing an access permission of the first data table based on the classifying.

### Claim 4 (depends on 2)

The computer-implemented method of claim 2, further comprising:
moving the first data table to a cold storage based on the classifying.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, the generating the object path dataset comprising:
retrieving cloudtrail logs associated with an account having access to the data tables.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, the generating the object path comprising:
filtering the cloudtrail logs to retain only the cloudtrail logs of buckets having hive tables.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, the generating the object path dataset comprising:
retrieving cloudtrail logs from the data lake, the cloudtrail log being associated with an account having access to the data tables.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, the generating the observability metrics comprising:
generating the observability metrics based on a union of the iterative matching across a plurality of iterations.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, the generating the observability metrics comprising:
automatically generating the observability metrics at predetermined time intervals.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, further comprising:
automatically generating and transmitting a tabular report based on the observability metrics.

### Claim 11

A system comprising:
a non-transitory storage medium storing computer program instructions; and
a processor configured to execute the computer program instructions to cause operations comprising:
generating an object path dataset comprising a plurality of rows of object paths for accessing corresponding data tables in a data lake;
generating a table path dataset comprising a plurality of rows of table paths of physical addresses for the corresponding data tables in the data lake;
expanding columns of the object path dataset to generate an expanded object path dataset by adding a plurality of additional columns, each additional column including object paths with a portion of information stripped away from corresponding object paths in a previous column;
expanding rows of the table path dataset to generate an expanded table path dataset by adding an additional row for each original row, the additional row comprising a corresponding table path without partition information;
iteratively matching the columns of the expanded object path dataset to the original rows and the additional rows; and
generating observability metrics of access of the data tables based on the iterative matching.

### Claim 12 (depends on 11)

The system of claim 11, the operations further comprising:
classifying a first data table as having a low usage based on the observability metrics.

### Claim 13 (depends on 12)

The system of claim 12, the operations further comprising:
changing an access permission of the first data table based on the classifying.

### Claim 14 (depends on 12)

The system of claim 12, the operations further comprising:
moving the first data table to a cold storage based on the classifying.

### Claim 15 (depends on 12)

The system of claim 12, the generating the object path dataset comprising:
retrieving cloudtrail logs associated with an account having access to the data tables.

### Claim 16 (depends on 15)

The system of claim 15, the generating the object path comprising:
filtering the cloudtrail logs to retain only the cloudtrail logs of buckets having hive tables.

### Claim 17 (depends on 11)

The system of claim 11, the generating the object path dataset comprising:
retrieving cloudtrail logs from the data lake, the cloudtrail log being associated with an account having access to the data tables.

### Claim 18 (depends on 11)

The system of claim 11, the generating the observability metrics comprising:
generating the observability metrics based on a union of the iterative matching across a plurality of iterations.

### Claim 19 (depends on 11)

The system of claim 11, the generating the observability metrics comprising:
automatically generating the observability metrics at predetermined time intervals.

### Claim 20 (depends on 11)

The system of claim 11, the operations further comprising:
automatically generating and transmitting a tabular report based on the observability metrics.