Optimized resource metering in a multi tenanted distributed file system

A computer program product, comprising a computer readable hardware storage device storing a computer readable program code, said computer readable program code comprising an algorithm that when executed by a processor of a hardware device implements a distributed file system node metering and hardware usage technology improvement method, said method comprising:

• receiving from a user, by said processor, job data associated with jobs for execution via a distributed file system, wherein said hardware device comprises specialized discrete non-generic analog, digital, and logic based plugin circuitry including a specially designed integrated circuit designed for only implementing said distributed file system metering and hardware usage technology improvement method;
• uploading from said job data, by said processor to a memory device of said hardware device, characteristics of said jobs;
• retrieving, by said processor enabling a policy engine circuit of said hardware device, policy and cost metrics data associated with hardware usage metering, wherein said policy and cost metrics data comprises policies implemented as pluggable components defined in advance, by:
• uploading xml files, via descriptor files or json files, or via a command line, for said jobs;
• retrieving, by said processor enabling a hardware device cluster, resource requests describing hardware resource usage of said jobs and metered with respect to a fine grained level of the actual utilization of hardware resources;
• querying, by said processor enabling a job descriptor engine of said hardware device, a recommendation circuit for locating said policy and cost metrics data of said resource requests;
• determining, by said processor enabling a machine learning circuit with respect to results of said querying, a set of metrics of said policy and cost metrics data associated with said resource requests;
• updating, by said processor based on said set of metrics, said machine learning circuit with characteristics of said jobs associated with network delays, sudden change of memory data volumes, and hardware device failure;
• detecting, by said processor enabling voltage sensors comprised by said hardware device, voltages associated with a hardware cluster;
• detecting, by said processor enabling temperature sensors comprised by said hardware device, temperature readings associated with said hardware cluster, wherein said voltages and said temperature readings are analyzed to indicate a speed and usage of CPUs and a memory space available for said hardware cluster;
• determining, by said processor enabling a metrics circuit with respect to said set of metrics, said voltages, and said temperature readings, utilized hardware resources of said hardware cluster with respect to said hardware usage metering and said resource requests;
• and
• allocating, by said processor based on results of said characteristics of said jobs and results of said determining said utilized hardware resources, specified queue memory and associated processor cores to said hardware cluster thereby enabling control functionality for overall resource utilization of said hardware cluster and improving execution of said jobs being executed on said hardware cluster;
• generating, by said processor executing a resource descriptor circuit, resource infrastructure associated with hardware resources of said resource requests, wherein said generating is executed based on sail policy engine circuit providing various policies and cost metrics for calculating said hardware usage metering, and wherein said policy engine circuit comprises the pluggable components;
• monitoring, by said processor executing said job descriptor engine, a plurality of nodes, hardware resources, processors, memory devices, and distributed system networks;
• and
• determining, by said processor, a configuration of said hardware device cluster based on an analysis of types of workloads and job characterizations;
• determining, by said processor, workloads associated with said plurality of nodes, said hardware resources, said processors, said memory devices, and said distributed system networks;
• determining, by said processor a recommendation for improving said configuration of said hardware device cluster;
• and
• calculating, by said computer processor, a cost for improving said configuration of said hardware device cluster, wherein said calculating, by said computer processor, a cost for improving said configuration of said hardware device cluster, wherein said
• Cost
• =
• (
• Confidence
• ⁢
• ⁢
• Factor
• )
• *
• ∑
• 0
• n
• ⁢
• (
• (
• a
• ⁢
• ∑
• 0
• ⁢
• %
• 100
• ⁢
• %
• ⁢
• Memory
• )
• +
• (
• b
• ⁢
• ∑
• 0
• ⁢
• %
• 100
• ⁢
• %
• ⁢
• CPU
• ⁢
• ⁢
• Utilization
• )
• +
• (
• c
• ⁢
• ∑
• 1
• ∞
• ⁢
• Network
• ⁢
• ⁢
• Data
• ⁢
• ⁢
• Transfer
• )
• +
• (
• d
• ⁢
• ∫
• t
• =
• 0
• t
• =
• n
• ⁢
• ⁢
• 1
• ⁢
• Scheduler
• ⁢
• ⁢
• Time
• )
• +
• (
• e
• ⁢
• ∑
• 0
• n
• ⁢
• ⁢
• 2
• ⁢
• Preemption
• )
• +
• (
• ∑
• 0
• n
• ⁢
• ⁢
• 3
• ⁢
• Slots
• ⁢
• ⁢
• Used
• )
• )
• ,
• wherein n is a number of said plurality of nodes, n1 is an execution time of said jobs, n2 is a number of times said jobs were preempted, n3 is a number of slots allocated to said jobs, wherein a, b, c, d, and e are weightage factors, wherein (a) comprises a weightage factor associated with memory, wherein said weightage factor associated with said memory comprises a variable per unit cost associated with said memory, wherein (b) comprises a weightage factor for a CPU, wherein said weightage factor for said CPU comprises a variable per unit cost associated with said CPU, wherein (c) comprises a weightage factor for network data transfer, wherein said weightage factor for said network data transfer comprises a variable per unit cost associated with said network data transfer, wherein (d) comprises a weightage factor associated with preemptive processes utilized, wherein said weightage factor associated with said preemptive processes comprise a variable per unit cost associated with said preemptive processes, wherein (e) comprises a weightage factor for a total number of preemptions, wherein said weightage factor for said total number of preemptions comprises a variable per unit cost associated with said total number of preemptions, and wherein sail (a), (b), (c), (d), and (e) remain constant for a same type or category of said jobs if all the additional parameters of a distributed environment remain constant.

A hardware device comprising a processor coupled to a computer-readable memory unit, said memory unit comprising instructions that when executed by the processor executes a distributed file system node metering and hardware usage technology improvement method comprising:

• receiving from a user, by said processor, job data associated with jobs for execution via a distributed file system, wherein said hardware device comprises specialized discrete non-generic analog, digital and logic based plugin circuitry including a specially designed integrated circuit designed for only implementing said distributed file system metering and hardware usage technology improvement method;
• uploading from said job data, by said processor to a memory device of said hardware device, characteristics of said jobs;
• retrieving, by said processor enabling a policy engine circuit of said hardware device, policy and cost metrics data associated with hardware usage metering, wherein said policy and cost metrics data comprises policies implemented as pluggable components defined in advance, by:
• uploading xml files, via descriptor files or json files, or via a command line, for said jobs;
• retrieving, by said processor enabling a hardware device cluster, resource requests describing hardware resource usage of said jobs and metered with respect to a fine grained level of the actual utilization of hardware resources;
• querying, by sail processor enabling a job descriptor engine of said hardware device, a recommendation circuit for locating said policy and cost metrics data of said resource requests;
• determining, by said processor enabling a machine learning circuit with respect to results of said querying, a set of metrics of said policy and cost metrics data associated with said resource requests;
• updating, by said processor based on said set of metrics, said machine learning circuit with characteristics of said jobs associated with network delays, sudden change of memory data volumes, and hardware device failure;
• detecting, by said processor enabling voltage sensors comprised by said hardware device, voltages associated with a hardware cluster;
• detecting, by said processor enabling temperature sensors comprised by said hardware device, temperature readings associated with said hardware cluster, wherein said voltages and said temperature readings are analyzed to indicate a speed and usage of CPUs and a memory space available for said hardware cluster;
• determining, by said processor enabling a metrics circuit with respect to said set of metrics, said voltages, and said temperature readings, utilized hardware resources of said hardware cluster with respect to said hardware usage metering and said resource requests;
• and
• allocating, by said processor based on results of said characteristics of said jobs and results of said determining said utilized hardware resources, specified queue memory and associated processor cores to said hardware cluster thereby enabling control functionality for overall resource utilization of said hardware cluster and improving execution of said jobs being executed on said hardware cluster;
• generating, by said processor executing a resource descriptor circuit, resource infrastructure associated with hardware resources of said resource requests, wherein said generating is executed based on said policy engine circuit providing various policies and cost metrics for calculating said hardware usage metering, and wherein said policy engine circuit comprises the pluggable components;
• monitoring, by said processor executing said job descriptor engine, a plurality of nodes, hardware resources, processors, memory devices, and distributed system networks;
• and
• determining, by said processor, a configuration of said hardware device cluster based on an analysis of types of workloads and job characterizations;
• determining, by said processor, workloads associated with said plurality of nodes, said hardware resources, said processors, said memory devices, and said distributed system networks;
• determining, by said processor a recommendation for improving said configuration of said hardware device cluster;
• and
• calculating, by said computer processor, a cost for improving said configuration of said hardware device cluster, wherein said
• Cost
• =
• (
• Confidence
• ⁢
• ⁢
• Factor
• )
• *
• ∑
• 0
• n
• ⁢
• (
• (
• a
• ⁢
• ∑
• 0
• ⁢
• %
• 100
• ⁢
• %
• ⁢
• Memory
• )
• +
• (
• b
• ⁢
• ∑
• 0
• ⁢
• %
• 100
• ⁢
• %
• ⁢
• CPU
• ⁢
• ⁢
• Utilization
• )
• +
• (
• c
• ⁢
• ∑
• n
• =
• 1
• n
• =
• ∞
• ⁢
• Network
• ⁢
• ⁢
• Data
• ⁢
• ⁢
• Transfer
• )
• +
• (
• d
• ⁢
• ∫
• t
• =
• 0
• t
• =
• n
• ⁢
• ⁢
• 1
• ⁢
• Scheduler
• ⁢
• ⁢
• Time
• )
• +
• (
• e
• ⁢
• ∑
• 0
• n
• ⁢
• ⁢
• 2
• ⁢
• Preemption
• )
• +
• (
• ∑
• 0
• n
• ⁢
• ⁢
• 3
• ⁢
• Slots
• ⁢
• ⁢
• Used
• )
• )
• ,
• wherein n is a number of said plurality of nodes, n1 is an execution time of said jobs, n2 is a number of times said jobs were preempted, n3 is a number of slots allocated to said jobs, wherein a, b, c, d, and e are weightage factors, wherein (a) comprises a weightage factor associated with memory, wherein said weightage factor associated with said memory comprises a variable per unit cost associated with said memory, wherein (b) comprises a weightage factor for a CPU, wherein said weightage factor for said CPU comprises a variable per unit cost associated with said CPU, wherein (c) comprises a weightage factor for network data transfer, wherein said weightage factor for said network data transfer comprises a variable per unit cost associated with said network data transfer, wherein (d) comprises a weightage factor associated with preemptive processes utilized, wherein said weightage factor associated with said preemptive processes comprise a variable per unit cost associated with said preemptive processes, wherein (e) comprises a weightage factor for a total number of preemptions, wherein said weightage factor for said total number of preemptions comprises a variable per unit cost associated with said total number of preemptions, and wherein said (a), (b), (c), (d), and (e) remain constant for a same type or category of said jobs if all the additional parameters of a distributed environment remain constant.

A distributed file system resource utilization metering and hardware usage technology improvement method comprising:

• analyzing, by a processor enabling an analysis circuit of a hardware device, resource requests for resource usage of hardware devices in a multi-tenancy system, wherein said hardware device comprises specialized discrete non-generic analog, digital, and logic based plugin circuitry including a specially designed integrated circuit designed for only implementing said distributed file system resource utilization metering and hardware usage technology improvement method;
• monitoring, by said processor, said resource usage of said hardware devices with respect to functions executed by said hardware devices in response to said resource requests;
• determining, by said processor, that said functions executed by said hardware devices have been completed;
• determining, by said processor, billing amounts for said resource usage of said hardware devices;
• analyzing, by said processor, said billing amounts for said resource usage for each of said resource requests;
• modifying for optimization of resource usage, hardware and software utilization and functionality of components of said hardware devices based on said analyzing said billing amounts;
• updating, by said processor based on a set of determined metrics, a machine learning circuit with characteristics of jobs associated with network delays, sudden change of memory data volumes, and hardware device failure;
• detecting, by said processor enabling voltage sensors comprised by said hardware device, voltages associated with a hardware cluster;
• detecting, by said processor enabling temperature sensors comprised by said hardware device, temperature readings associated with said hardware cluster, wherein said voltages and said temperature readings are analyzed to indicate a speed and usage of CPUs and a memory space available for said hardware cluster;
• determining, by said processor enabling a metrics circuit with respect to said set of determined metrics, said voltages, and said temperature readings, utilized hardware resources of said hardware cluster with respect to said hardware usage metering and said resource requests;
• and
• allocating, by said processor based on results of said characteristics of said jobs and results of said determining said utilized hardware resources, specified queue memory and associated processor cores to said hardware cluster thereby enabling control functionality for overall resource utilization of said hardware cluster and improving execution of said jobs being executed on said hardware cluster;
• generating, by said processor executing a resource descriptor circuit, resource infrastructure associated with hardware resources of said resource requests, wherein said generating is executed based on said policy engine circuit providing various policies and cost metrics for calculating said hardware usage metering, and wherein said policy engine circuit comprises the pluggable components;
• monitoring, by said processor executing a job descriptor engine, a plurality of nodes, hardware resources, processors, memory devices, and distributed system networks;
• and
• determining, by said processor, a configuration of said hardware device cluster based on an analysis of types of workloads and job characterizations;
• determining, by said processor, workloads associated with said plurality of nodes, said hardware resources, said processors, said memory devices, and said distributed system networks;
• determining, by said processor a recommendation for improving said configuration of said hardware device cluster;
• and calculating, by said computer processor, a cost for improving said configuration of said hardware device cluster, wherein said
• Cost
• =
• (
• Confidence
• ⁢
• ⁢
• Factor
• )
• *
• ∑
• 0
• n
• ⁢
• (
• (
• a
• ⁢
• ∑
• 0
• ⁢
• %
• 100
• ⁢
• %
• ⁢
• Memory
• )
• +
• (
• b
• ⁢
• ∑
• 0
• ⁢
• %
• 100
• ⁢
• %
• ⁢
• CPU
• ⁢
• ⁢
• Utilization
• )
• +
• (
• c
• ⁢
• ∑
• n
• =
• 1
• n
• =
• ∞
• ⁢
• Network
• ⁢
• ⁢
• Data
• ⁢
• ⁢
• Transfer
• )
• +
• (
• d
• ⁢
• ∫
• t
• =
• 0
• t
• =
• n
• ⁢
• ⁢
• 1
• ⁢
• Scheduler
• ⁢
• ⁢
• Time
• )
• +
• (
• e
• ⁢
• ∑
• 0
• n
• ⁢
• ⁢
• 2
• ⁢
• Preemption
• )
• +
• (
• ∑
• 0
• n
• ⁢
• ⁢
• 3
• ⁢
• Slots
• ⁢
• ⁢
• Used
• )
• )
• ,
• wherein n is a number of said plurality of nodes, n1 is an execution time of said jobs, n2 is a number of times said jobs were preempted, n3 is a number of slots allocated to said jobs, wherein a, b, c, d, and e are weightage factors, wherein (a) comprises a weightage factor associated with memory, wherein said weightage factor associated with said memory comprises a variable per unit cost associated with said memory, wherein (b) comprises a weightage factor for a CPU, wherein said weightage factor for said CPU comprises a variable per unit cost associated with said CPU, wherein (c) comprises a weightage factor for network data transfer, wherein said weightage factor for said network data transfer comprises a variable per unit cost associated with said network data transfer, wherein (d) comprises a weightage factor associated with preemptive processes utilized, wherein said weightage factor associated with said preemptive processes comprise a variable per unit cost associated with said preemptive processes, wherein (e) comprises a weightage factor for a total number of preemptions, wherein said weightage factor for said total number of preemptions comprises a variable per unit cost associated with said total number of preemptions, and wherein said (a), (b), (c), (d), and (e) remain constant for a same type or category of said jobs if all the additional parameters of a distributed environment remain constant.

A hardware device comprising a processor coupled to a computer-readable memory unit, said memory unit comprising instructions that when executed by the processor executes a distributed file system resource utilization metering and hardware usage technology improvement method comprising:

• analyzing, by said processor enabling an analysis circuit of a hardware device, resource requests for resource usage of hardware devices in a multi-tenancy system, wherein said hardware device comprises specialized discrete non-generic analog, digital, and logic based plugin circuitry including a specially designed integrated circuit designed for only implementing said distributed file system resource utilization metering and hardware usage technology improvement method;
• monitoring, by said processor, said resource usage of said hardware devices with respect to functions executed by said hardware devices in response to said resource requests;
• determining, by said processor, that said functions executed by said hardware devices have been completed;
• determining, by said processor, billing amounts for said resource usage of said hardware devices;
• analyzing, by said processor, said billing amounts for said resource usage for each of said resource requests;
• modifying for optimization of resource usage, hardware and software utilization and functionality of components of said hardware devices based on said analyzing said billing amounts;
• updating, by said processor based on a set of determined metrics, a machine learning circuit with characteristics of jobs associated with network delays, sudden change of memory data volumes, and hardware device failure;
• detecting, by said processor enabling voltage sensors comprised by said hardware device, voltages associated with a hardware cluster;
• detecting, by said processor enabling temperature sensors comprised by said hardware device, temperature readings associated with said hardware cluster, wherein said voltages and said temperature readings are analyzed to indicate a speed and usage of CPUs and a memory space available for said hardware cluster;
• determining, by said processor enabling a metrics circuit with respect to said set of determined metrics, said voltages, and said temperature readings, utilized hardware resources of said hardware cluster with respect to said hardware usage metering and said resource requests;
• and
• allocating, by said processor based on results of said characteristics of said jobs and results of said determining said utilized hardware resources, specified queue memory and associated processor cores to said hardware cluster thereby enabling control functionality for overall resource utilization of said hardware cluster and improving execution of said jobs being executed on said hardware cluster;
• generating, by said processor executing a resource descriptor circuit, resource infrastructure associated with hardware resources of said resource requests, wherein said generating is executed based on said policy engine circuit providing various policies and cost metrics for calculating said hardware usage metering, and wherein said policy engine circuit comprises the pluggable components;
• monitoring, by said processor executing a job descriptor engine, a plurality of nodes, hardware resources, processors, memory devices, and distributed system networks;
• and
• determining, by said processor, a configuration of said hardware device cluster based on an analysis of types of workloads and job characterizations;
• determining, by said processor, workloads associated with said plurality of nodes, said hardware resources, said processors, said memory devices, and said distributed system networks;
• determining, by said processor based a recommendation for improving said configuration of said hardware device cluster;
• and
• calculating, by said computer processor, a cost for improving said configuration of said hardware device cluster, wherein said
• Cost
• =
• (
• Confidence
• ⁢
• ⁢
• Factor
• )
• *
• ∑
• 0
• n
• ⁢
• (
• (
• a
• ⁢
• ∑
• 0
• ⁢
• %
• 100
• ⁢
• %
• ⁢
• Memory
• )
• +
• (
• b
• ⁢
• ∑
• 0
• ⁢
• %
• 100
• ⁢
• %
• ⁢
• CPU
• ⁢
• ⁢
• Utilization
• )
• +
• (
• c
• ⁢
• ∑
• n
• =
• 1
• n
• =
• ∞
• ⁢
• Network
• ⁢
• ⁢
• Data
• ⁢
• ⁢
• Transfer
• )
• +
• (
• d
• ⁢
• ∫
• t
• =
• 0
• t
• =
• n
• ⁢
• ⁢
• 1
• ⁢
• Scheduler
• ⁢
• ⁢
• Time
• )
• +
• (
• e
• ⁢
• ∑
• 0
• n
• ⁢
• ⁢
• 2
• ⁢
• Preemption
• )
• +
• (
• ∑
• 0
• n
• ⁢
• ⁢
• 3
• ⁢
• Slots
• ⁢
• ⁢
• Used
• )
• )
• ,
• wherein n is a number of said plurality of nodes, n1 is an execution time of said jobs, n2 is a number of times said jobs were preempted, n3 is a number of slots allocated to said jobs, wherein a, b, c, d, and e are weightage factors, wherein (a) comprises a weightage factor associated with memory, wherein said weightage factor associated with said memory comprises a variable per unit cost associated with said memory, wherein (b) comprises a weightage factor for a CPU, wherein said weightage factor for said CPU comprises a variable per unit cost associated with said CPU, wherein (c) comprises a weightage factor for network data transfer, wherein said weightage factor for said network data transfer comprises a variable per unit cost associated with said network data transfer, wherein (d) comprises a weightage factor associated with preemptive processes utilized, wherein said weightage factor associated with said preemptive processes comprise a variable per unit cost associated with said preemptive processes, wherein (e) comprises a weightage factor for a total number of preemptions, wherein said weightage factor for said total number of preemptions comprises a variable per unit cost associated with said total number of preemptions, and wherein said (a), (b), (c), (d), and (e) remain constant for a same type or category of said jobs if all the additional parameters of a distributed environment remain constant.