Auto-completion for Multi-modal User Input in Assistant Systems

One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

• receive, at a client system, an initial input from a first user, wherein the initial input is in a first modality;
• determine one or more intents and one or more slots corresponding to the initial input, wherein the one or more slots are conditioned on the one or more intents;
• generate, based on the one or more intents and the one or more slots, one or more candidate continuation-inputs, where the one or more candidate continuation-inputs are in one or more candidate modalities, respectively, wherein the candidate modalities are different from the first modality, and wherein each of the candidate continuation-inputs references one or more entities represented by the one or more slots;
• and
• present, at the client system, one or more suggested inputs corresponding to one or more of the candidate continuation-inputs.

A system comprising:

• one or more processors;
• and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
• receive, at a client system, an initial input from a first user, wherein the initial input is in a first modality;
• determine one or more intents and one or more slots corresponding to the initial input, wherein the one or more slots are conditioned on the one or more intents;
• generate, based on the one or more intents and the one or more slots, one or more candidate continuation-inputs, where the one or more candidate continuation-inputs are in one or more candidate modalities, respectively, wherein the candidate modalities are different from the first modality, and wherein each of the candidate continuation-inputs references one or more entities represented by the one or more slots;
• and
• present, at the client system, one or more suggested inputs corresponding to one or more of the candidate continuation-inputs.