Modifying computational graphs

A system comprising a plurality of processing components in one or more locations, the plurality of processing components implemented on one or more computers having stored on them instructions operable to cause the system to perform actions comprising:

• obtaining a computational graph for a neural network, the computational graph comprising a plurality of nodes representing respective operations of the neural network and a plurality of directed edges, each directed edge connecting a respective pair of nodes of the plurality of nodes and representing that an output of the operation represented by one node in the pair is processed as input by the operation represented by the other node in the pair;
• partitioning the computational graph into at least a first subgraph and a second subgraph, the first subgraph comprising a first subset of the plurality of nodes, the second subgraph comprising a different, second subset of the plurality of nodes;
• identifying that the first subgraph includes a first node representing an operation that produces output for processing as input by an operation represented by a second node in the second subgraph;
• inserting a send node into the first subgraph and a receive node into the second subgraph, the send node configured to receive the output from the first node and point to the receive node, the receive node configured to receive the output from the first node via the send node and to pass the output from the first node as input to the second node;
• assigning the first subgraph to a first processing component of the plurality of processing components and the second subgraph to a second processing component of the plurality of processing components;
• and
• using the first subgraph on the first processing component and the second subgraph on the second processing component to execute operations of the neural network;
• wherein:
• the first processing component comprises first hardware resources included in one or more first machines;
• the second processing component comprises second hardware resources included in one or more second machines;
• and
• data communicated between the first processing component and the second processing component is communicated according to a protocol that corresponds to each of the first hardware resources and the second hardware resources.

A non-transitory computer-readable storage device having instructions stored thereon that, when executed, cause performance of actions comprising:

• obtaining a computational graph for a neural network, the computational graph comprising a plurality of nodes representing respective operations of the neural network and a plurality of directed edges, each directed edge connecting a respective pair of nodes of the plurality of nodes and representing that an output of the operation represented by one node in the pair is processed as input by the operation represented by the other node in the pair;
• partitioning the computational graph into at least a first subgraph and a second subgraph, the first subgraph comprising a first subset of the plurality of nodes, the second subgraph comprising a different, second subset of the plurality of nodes;
• identifying that the first subgraph includes a first node representing an operation that produces output for processing as input by an operation represented by a second node in the second subgraph;
• inserting a send node into the first subgraph and a receive node into the second subgraph, the send node configured to receive the output from the first node and point to the receive node, the receive node configured to receive the output from the first node via the send node and to pass the output from the first node as input to the second node;
• assigning the first subgraph to a first processing component and the second subgraph to a second processing component;
• and
• using the first subgraph on the first processing component and the second subgraph on the second processing component to execute operations of the neural network;
• wherein:
• the first processing component comprises first hardware resources included in one or more first machines;
• the second processing component comprises second hardware resources included in one or more second machines;
• and
• data communicated between the first processing component and the second processing component is communicated according to a protocol that corresponds to each of the first hardware resources and the second hardware resources.