Distributed data processing
AssigneeAlibaba Group Holding Limited
InventorMin Han
Life of this patent
Estimating the term…
Abstract
A method for distributed data processing includes selecting a first number of candidate key-value pairs from input key-value pairs stored in a distributed file system and identifying an invocation frequency for each of the first number of candidate key-value pairs in a predetermined period of time. The invocation frequency is a number of times the corresponding candidate key-value pair was called by a service system in the predetermined period of time. The method further includes selecting a second number of hot key-value pairs from the candidate key-value pairs based on the invocation frequencies of the candidate key-value pairs, mapping the second number of hot key-value pairs to intermediate key-value pairs, and reducing the intermediate key-value pairs to resultant key-value pairs to be called by the service system.
Claims20
A computer-implemented method, comprising:
- selecting a first number of candidate key-value pairs from input key-value pairs stored in a distributed file system, wherein selecting the first number of candidate key-value pairs comprises at least one of:
- random selection, or selection based on service types;
- identifying an invocation frequency for each of the first number of candidate key-value pairs in a predetermined period of time, wherein the invocation frequency is a number of times the corresponding candidate key-value pair was called by a service system in the predetermined period of time;
- selecting a second number of hot key-value pairs from the candidate key-value pairs, wherein the hot key-value pairs comprise at least one of:
- candidate key-value pairs that have invocation frequencies higher than a predetermined threshold, or candidate key-value pairs with key values within a key value interval;
- mapping, as intermediate key-value pairs, the second number of hot key-value pairs;
- and
- reducing, as resultant key-value pairs, the intermediate key-value pairs, wherein the resultant key-value pairs are to be called by the service system.
A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform operations comprising:
- selecting a first number of candidate key-value pairs from input key-value pairs stored in a distributed file system, wherein selecting the first number of candidate key-value pairs comprises at least one of:
- random selection, or selection based on service types;
- identifying an invocation frequency for each of the first number of candidate key-value pairs in a predetermined period of time, wherein the invocation frequency is a number of times the corresponding candidate key-value pair was called by a service system in the predetermined period of time;
- selecting a second number of hot key-value pairs from the candidate key-value pairs, wherein the hot key-value pairs comprise at least one of:
- candidate key-value pairs that have invocation frequencies higher than a predetermined threshold, or candidate key-value pairs with key values within a key value interval;
- mapping, as intermediate key-value pairs, the second number of hot key-value pairs;
- and
- reducing, as resultant key-value pairs, the intermediate key-value pairs, wherein the resultant key-value pairs are to be called by the service system.
A computer-implemented system, comprising:
- one or more computers;
- and
- one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing instructions, that when executed by the one or more computers, perform operations comprising:
- selecting a first number of candidate key-value pairs from input key-value pairs stored in a distributed file system, wherein selecting the first number of candidate key-value pairs comprises at least one of:
- random selection, or selection based on service types;
- identifying an invocation frequency for each of the first number of candidate key-value pairs in a predetermined period of time, wherein the invocation frequency is a number of times the corresponding candidate key-value pair was called by a service system in the predetermined period of time;
- selecting a second number of hot key-value pairs from the candidate key-value pairs, wherein the hot key-value pairs comprise at least one of:
- candidate key-value pairs that have invocation frequencies higher than a predetermined threshold, or candidate key-value pairs with key values within a key value interval;
- mapping, as intermediate key-value pairs, the second number of hot key-value pairs;
- and
- reducing, as resultant key-value pairs, the intermediate key-value pairs, wherein the resultant key-value pairs are to be called by the service system.
Dependencies are inferred from the claim text.