A system for managing communication between graphics processing units (GPUs) through a network comprising a plurality of forwarding elements, sets of the GPUs executing operations associated with a plurality of distributed applications, the system comprising:
- a first set of infrastructure services to configure a plurality of forwarding elements to implement a plurality of virtual private clouds (VPCs) with each VPC connecting different sets of GPUs that execute different distributed applications;
- and
- for each VPC, a second set of scheduling services to schedule forwarding of results computed by the VPC's GPUs through the plurality of forwarding elements that form the VPC.