Collaborative ranking of interpretations of spoken utterances

A system comprising:

• receive audio data that captures a spoken utterance of a user, the audio data being generated by one or more microphones of a client device of the user, and the spoken utterance being directed to an automated assistant executed at least in part at the client device;
• determine, based on processing the audio data, a plurality of first-party interpretations of the spoken utterance, each of the plurality of first-party interpretations being associated with a corresponding first-party predicted value indicative of a magnitude of confidence that each of the first-party interpretations are predicted to satisfy the spoken utterance;
• identify a given third-party agent capable of satisfying the spoken utterance;
• transmit, to the given third-party agent and over one or more networks, and based on processing the audio data, one or more structured requests that, when received, causes the given third-party to determine a plurality of third-party interpretations of the spoken utterance, each of the plurality of third-party interpretations being associated with a corresponding third-party predicted value indicative of a magnitude of confidence that each of the third-party interpretations are predicted to satisfy the spoken utterance;
• receive, from the given third-party agent and over one or more of the networks, the plurality of third-party interpretations of the spoken utterance;
• select, based on the corresponding first-party predicted values and the corresponding third-party predicted values, a given interpretation of the spoken utterance from among the plurality of first-party interpretations and the plurality third-party interpretations;
• and
• generate one or more training instances to be utilized in updating a machine learning (ML) model that is trained to select given interpretations for spoken utterances based on the plurality of first-party interpretations, the corresponding first-party predicted values, the plurality of third-party interpretations, and the corresponding third-party predicted values.

A system comprising:

• at least one processor;
• and
• memory storing instructions that, when executed, cause the at least one processor to be operable to:
• obtain a plurality of first-party interpretations of a spoken utterance, wherein each of the first-party interpretations of the spoken utterance are determined by an automated assistant, and wherein the automated assistant determines the first-party interpretations based on:
• processing audio data that captures a spoken utterance of a user, the audio data being generated by one or more microphones of a client device of the user, and the spoken utterance being directed to the automated assistant executed at least in part at the client device;
• determining, based on processing the audio data, a plurality of first-party interpretations of the spoken utterance, each of the plurality of first-party interpretations being associated with a corresponding first-party predicted value indicative of a magnitude of confidence that each of the first-party interpretations are predicted to satisfy the spoken utterance;
• obtain a plurality of third-party interpretations of the spoken utterance, wherein each of the third-party interpretations of the spoken utterance are determined by one or more third-party agents, and wherein each of the one or more third-party agents determine one or more of the third-party interpretations based on:
• receiving, from the automated assistant over one or more networks, and based on processing the audio data, one or more structured requests;
• and
• determining, based on the one or more structured requests, the plurality of third-party interpretations of the spoken utterance, each of the plurality of third-party interpretations being associated with a corresponding third-party predicted value indicative of a magnitude of confidence that each of the third-party interpretations are predicted to satisfy the spoken utterance;
• select, based on the corresponding first-party predicted values and the corresponding third-party predicted values, a given interpretation of the spoken utterance from among the plurality of first-party interpretations and the plurality of third-party interpretations;
• and
• transmit the given interpretation of the spoken utterance to the automated assistant or a given third-party agent, of the one or more third-party agents.