# US 10,120,904 B2 · Resource management in a distributed computing environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,120,904 B2
- Document ID: US10120904B2
- Status: Granted
- Kind: B2
- Application number: 14/588,250
- Filed: 2014-12-31
- Published: 2016-06-30
- Granted: 2018-11-06
- Earliest claimed: 2014-12-31
- Assignees: Cloudera, Inc.
- Applicants: Cloudera, Inc.
- Inventors: Jairam Ranganathan
- Web page: https://patentgrep.com/patent/US10120904B2

## CPC

- G06F 16/24552: {Database cache management}
- G06F 9/5066: {Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs (mappping at compile time, see G06F8/451)}
- G06F 9/5083: {Techniques for rebalancing the load in a distributed system}
- G06F 16/182: Distributed file systems
- G06F 2209/505: Clust
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and methods are disclosed for resource management in a distributed computing environment. In some embodiments, a resource manager for a large distributed cluster needs to be able to provide resource responses very quickly. But each query may also not be accurate in initial resource request and will often have to come back to the resource manager multiple times. An artifact may provide low latency query responses by using resource request caching that can handle re-requests of resources. According to some embodiments, a queuing mechanism may take into account resources currently expended and any resource requirement estimates available in order to make queuing decisions that meet policies set by an administrator. In some embodiments, scheduling decisions are distribute across a cluster of computing systems while still maintaining approximate compliance with resource management policies set by an administrator.

## Claims

### Claim 1

A computer-implemented method for managing computing resources in a distributed computing cluster operable to execute queries on data stored in the cluster, the computing cluster having multiple data nodes storing the data, the method comprising:
instantiating the plurality of data nodes in the cluster, a respective data node operating an instance of a query engine that is capable of planning a query and executing at least a query portion that corresponds to data stored on the respective data node;
receiving, by a master node in the distributed computing cluster, an administrator usage policy, wherein the administrator usage policy includes limitations on usage of computing resources by a user and/or group of users;
receiving, by the master node, a query and an initial request for computing resources to process the query; and
determining, by the master node, if sufficient computing resources are available to process the query based on the initial request for computing resources and a limitation on usage of computing resources applicable to the query and resource request based on the administrator usage policy, wherein,
if sufficient resources are not available:
the master node is configured to place the query into a queue until sufficient resources are available.

### Claim 2 (depends on 1)

The method of claim 1, wherein computing resources include one or more of, memory, processing power, data storage, and network bandwidth.

### Claim 3 (depends on 1)

The method of claim 1, wherein the distributed computing cluster includes a distributed file system or a data store.

### Claim 4 (depends on 3)

The method of claim 3, wherein the distributed computing cluster is an Apache Hadoop cluster, the distributed file system is a Hadoop Distributed File System (HDFS) and the data store is a NoSQL (No Structured Query Language) data store, wherein the NoSQL data store includes Apache HBase.

### Claim 5 (depends on 1)

The method of claim 1, wherein,
if sufficient resources are available, the master node is configured to:
reserve a cache of computing resources at one or more slave nodes in the distributed computing cluster based on the initial request for resources and a heuristic associated with a temporal clustering of received queries;
allocate a first set computing resources from the reserved cache of computing resources based on the initial request for computing resources to process the query;
receive, during processing of the query, an updated computing resource request; and
if additional computing resources are required:
allocate an additional second set of computing resources from the reserved cache based on the updated resource request.

### Claim 6 (depends on 5)

The method of claim 5, wherein,
if fewer resources are required:
the master node is configured to remove a third set of computing resources from the reserved cache based on the updated resource request.

### Claim 7 (depends on 5)

The method of claim 5, wherein the heuristic includes a prediction of a plurality of queries to follow the received query.