US 7,856,544 B2Granted

Stream processing in super node clusters of processors assigned with stream computation graph kernels and coupled by stream traffic optical links

AssigneeInternational Business Machines Corporation

InventorsEugen Schenfeld, Thomas B. Smith, III

Life of this patent

Estimating the term…

Loading patent…

Abstract

A method for implementing a stream processing computer architecture includes creating a stream computer processing (SCP) system by forming a super node cluster of processors representing physical computation nodes (“nodes”), communicatively coupling the processors via a local interconnection means (“interconnect”), and communicatively coupling the cluster to an optical circuit switch (OCS), via optical external links (“links”). The OCS is communicatively coupled to another cluster of processors via the links. The method also includes generating a stream computation graph including kernels and data streams, and mapping the graph to the SCP system, which includes assigning the kernels to the clusters and respective nodes, assigning data stream traffic between the kernels to the interconnection when the data stream is between nodes in the same cluster, and assigning traffic between the kernels to the links when the data stream is between nodes in different clusters. The method also includes configuring the OCSs to provide connectivity between mapped clusters.

Claims18

Dependent claims

A method for implementing a stream processing computer architecture, comprising:

forming a super node cluster of processors, each of the processors comprising at least one physical computation node within the super node cluster of processors, the super node cluster of processors formed by identifying relationships among processors with respect to associated data streams and using a multiplexer to aggregate and separate corresponding data streams between respective processors, wherein processors in the super node cluster of processors are communicatively coupled to one another via a local interconnection means;
communicatively coupling the super node cluster of processors to an optical circuit network, the optical circuit network comprising one or more optical external links and at least one optical circuit switch, the super node cluster of processors coupled to the optical circuit switch via the one or multiple optical external links, the optical external links each using multiple optical wavelengths in a wavelength division multiplexer, the optical circuit switch communicatively coupled to at least one other super node cluster of processors that respectively includes at least one other physical computation node, via one or more optical external links from the other super node cluster of processors, wherein the super node cluster of processors and the optical circuit network collectively represent a stream computer processing system;
generating a stream computation graph including kernels and data streams, the kernels representing software code elements that perform computations on one or more of the data streams that are input to corresponding kernels;
mapping components of the stream computation graph to physical components of the stream computer processing system, comprising:
assigning the kernels to the super node clusters and to respective physical computation nodes of each of the super node cluster of processors;
assigning data stream traffic between the kernels to the local interconnection means when the respective data stream is between physical computation nodes in the same super node cluster of processors;
assigning data stream traffic between the kernels to the optical external links when the respective data stream is between physical computation nodes in different super node cluster of processors;
and
configuring the optical circuit switch to provide connectivity between the super node clusters of processors corresponding to the assignments, via the optical external links;
and
executing operations on the stream computer processing system in accordance with the stream computation graph.

8 more elements

A system for implementing a stream processing computer architecture, comprising:

a stream computer processing system, comprising:
a super node cluster of processors, each of the processors comprising at least one physical computation node within the super node cluster of processors, the super node cluster of processors formed by identifying relationships among processors with respect to associated data streams and using a multiplexer to aggregate and separate corresponding data streams between respective processors, wherein processors in the super node cluster of processors are communicatively coupled to one another via a local interconnection means;
and
the super node cluster of processors communicatively coupled to an optical circuit network, the optical circuit network comprising one or more optical external links and at least one optical circuit switch, the super node cluster of processors coupled to the optical circuit switch via the one or multiple optical external links, the optical external links each using multiple optical wavelengths in a wavelength division multiplexer, the optical circuit switch communicatively coupled to at least one other super node cluster of processors that respectively includes at least one other physical computation node, via one or more optical external links from the other super node cluster of processors;
a stream computation graph including kernels and data streams, the kernels representing software code elements that perform computations on one or more of the data streams that are input to corresponding kernels;
wherein components of the stream computation graph are mapped to physical components of the stream computer processing system, the mapping comprising:
assigning the kernels to the super node cluster of processors and to respective physical computation nodes of each of the super node clusters of processors;
assigning data stream traffic between the kernels to the local interconnection means when the respective data stream is between physical computation nodes in the same super node cluster of processors;
assigning data stream traffic between the kernels to the optical external links when the respective data stream is between physical computation nodes in different super node cluster of processors;
and
configuring the optical circuit switch to provide connectivity between super node clusters of processors corresponding to the assignments, via the optical external links;
wherein operations on the stream computer processing system are executed in accordance with the stream computation graph.

9 more elements

Dependencies are inferred from the claim text.

Facts

Application: 12/193,125Filed 18 Aug 2008
Published: US 2010/0042809 A118 Feb 2010
Granted: 21 Dec 2010
Term ends: Not estimatedThe source records don’t support an estimate.

Classification6 codes

G06F 9/5061
Allocation of resources, e.g. of the central processing unit [CPU]: partitioning or combining of resources
G06F 9/5083
Allocation of resources, e.g. of the central processing unit [CPU]: techniques for rebalancing the load in a distributed system
G06F 15/17343
Direct connection machines, e.g. completely connected computers, point to point communication networks: wherein the interconnection is dynamically…
Y02D 10/00
Energy efficient computing