US 11,445,004 B2Granted

Method for processing shared data, apparatus, and server

AssigneePetal Cloud Technology Co., Ltd.

InventorsJun Tan, Zhengfan Yuan

Life of this patent

Estimating the term…

Loading patent…

Abstract

A method for processing shared data, an apparatus, and a server are provided, and relate to the field of communications technologies, so that shared data can be cached across application programs, which facilitates data sharing across applications programs, can reduce a quantity of repeated computations, and helps accelerate a computing speed of a Spark architecture. The method includes: receiving a first instruction; starting a first Spark context for a first application program, to create a DAG of the first application program, and caching the DAG of the first application program in a first area of a first server; receiving a second instruction; starting a second Spark context for a second application program reading m DAGs from the first area; and caching the to-be-cached shareable RDDs in a main memory of a second server, where the shareable RDD is an RDD included in at least two DAGs of the m DAGs.

Claims18

Dependent claims

A method for processing shared data, wherein the method is applied to a server cluster, and the method comprises:

receiving, by a first server, a first instruction for starting a first application program;
in response to the first instruction, starting, by the first server, a first Spark context for the first application program, to create a directed acyclic graph (DAG) of the first application program, and caching the DAG of the first application program in a first area of the first server;
receiving, by the first server, a second instruction for starting a second application program;
in response to the second instruction, starting, by the first server, a second Spark context for the second application program, to create a DAG of the second application program, and caching the DAG of the second application program in the first area of the first server;
reading, by the first server, two or more DAGs from the first area, wherein the two or more DAGs comprise the DAG of the first application program and the DAG of the second application program;
determining, by the first server, to-be-cached shareable resilient distributed datasets (RDDs) based on the two or more DAGs, including determining a quantity of shareable RDDs comprised in each of the two or more DAGs, determining a shareable RDD comprised in a first DAG and determining, based on remaining space in a main memory of a second server, a to-be-cached shareable RDD comprised in the first DAG;
and
caching the to-be-cached shareable RDDs comprised in the first DAG in the main memory of the second server,
wherein a shareable RDD is an RDD comprised in at least two DAGs of the two or more DAGs, and
wherein the first DAG is a DAG with a largest quantity of shareable RDDs in the two or more DAGs.

7 more elements

A server comprising a processor, a non-transitory computer-readable medium having computer-executable instructions stored thereon, and a communications interface, wherein the non-transitory computer-readable medium and the communications interface are coupled to the processor, wherein the computer-executable instructions, when executed by the processor, cause the processor to facilitate:

receiving a first instruction for starting a first application program, starting a first Spark context for the first application program to create a directed acyclic graph (DAG) of the first application program, and caching the DAG of the first application program in a first area of a first server;
receiving a second instruction for starting a second application program, starting a second Spark context for the second application program to create a DAG of the second application program, and caching the DAG of the second application program in the first area of the first server;
reading two or more DAGs from the first area, wherein the two or more DAGs comprise the DAG of the first application program and the DAG of the second application program;
determining to-be-cached shareable resilient distributed datasets (RDDs) based on the two or more DAGs, including determining a quantity of shareable RDDs comprised in each of the two or more DAGs, determining a shareable RDD comprised in a first DAG and determining, based on remaining space in a main memory of a second server, a to-be-cached shareable RDD comprised in the first DAG;
and
caching the to-be-cached shareable RDDs comprised in the first DAG in the main memory of the second server,
wherein a shareable RDD is an RDD comprised in at least two DAGs of the two or more DAGs, and
wherein the first DAG is a DAG with a largest quantity of shareable RDDs in the two or more DAGs.

5 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/415,573Filed 3 Dec 2019
Priority: CN 201811545173.617 Dec 2018
Published: US 2022/0060568 A124 Feb 2022
Granted: 13 Sep 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification8 codes

G06F 9/544
Interprogram communication: buffers; Shared memory; Pipes
G06F 9/5066
Partitioning or combining of resources: algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs
G06N 7/01
Computing arrangements based on specific mathematical models: probabilistic graphical models, e.g. probabilistic networks
H04L 67/1097
In which an application is distributed across nodes in the network: for distributed storage of data in networks, e.g. transport arrangements for…

Family

US 2022/0060568 A1Pre-grant publication · published 24 Feb 2022
US 11,445,004 B2This patent

From the US filing history. Foreign family members aren’t listed.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key