Generating output sequences using a token processing neural network and external tools

A method performed by one or more computers for training a token processing neural network that has parameters, the method comprising:

• generating, by the token processing neural network, a plurality of training output sequences, each training output sequence comprising a plurality of tokens, the plurality of tokens comprising special tokens representing a request for an external tool;
• for each of the plurality of training output sequences, determining a reward score for the training output sequence that is dependent on relative positions of the special tokens within the plurality of tokens included in the training output sequence;
• and
• training the auto-regressive token processing neural network based on the plurality of training output sequences to update values of the parameters of the auto-regressive token processing neural network through reinforcement learning to optimize the reward scores.

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform operations comprising:

• receiving a request from a user for an output sequence;
• generating, using a token processing neural network, a first partial output sequence and an external tool request sequence, wherein the external tool request sequence comprises tokens that represent a plurality of requests for one or more external tools, and wherein the plurality of requests comprises a first request and a second request;
• obtaining a first result from an external tool corresponding to the first request before obtaining a second result from an external tool corresponding to the second request;
• generating, using the token processing neural network, based on the first partial output sequence and the first result, a second partial output sequence;
• providing the second partial output sequence and the first partial output sequence for presentation to the user before obtaining the second result from the external tool corresponding to the second request;
• obtaining the second result from the external tool corresponding to the second request;
• generating, using the token processing neural network, based on the second partial output sequence and the second result, a third partial output sequence;
• and
• providing the third partial output sequence, the second partial output sequence, and the first partial output sequence for presentation to the user.

One or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform operations comprising:

• receiving a request from a user for an output sequence;
• generating, using a token processing neural network, a first partial output sequence and an external tool request sequence, wherein the external tool request sequence comprises tokens that represent a plurality of requests for one or more external tools, and wherein the plurality of requests comprises a first request and a second request;
• obtaining a first result from an external tool corresponding to the first request before obtaining a second result from an external tool corresponding to the second request;
• generating, using the token processing neural network, based on the first partial output sequence and the first result, a second partial output sequence;
• providing the second partial output sequence and the first partial output sequence for presentation to the user before obtaining the second result from the external tool corresponding to the second request;
• obtaining the second result from the external tool corresponding to the second request;
• generating, using the token processing neural network, based on the second partial output sequence and the second result, a third partial output sequence;
• and
• providing the third partial output sequence, the second partial output sequence, and the first partial output sequence for presentation to the user.