Multimodal responses

A multimodal client device, comprising:

• one or more microphones;
• one or more sensors that are in addition to the one or more microphones;
• one or more speakers;
• one or more displays;
• memory storing instructions;
• one or more processors executing the instructions to perform the method of:
• determining a client device action based on one or more instances of user interface input provided by a user of the multimodal client device;
• processing, using a machine learning model, (1) sensor data from the one or more sensors of the multimodal client device and (2) a multimodal response, to generate client device output,
• wherein the sensor data from the one or more sensors of the multimodal client device indicate a current client device modality of the multimodal client device,
• wherein the current client device modality is one of a plurality of discrete client device modalities available for the multimodal client device,
• wherein the sensor data is in addition to any sensor data generated by the one or more instances of user interface input,
• wherein the multimodal response includes components of output for the client device action for the plurality of discrete client device modalities,
• wherein the multimodal response is received by the multimodal client device from a remote server,
• wherein generating the client device output is done by the multimodal client device, and
• wherein the components of output include at least (a) a core message component representing information to render for each of the plurality of discrete client device modalities and (b) one or more modality dependent components each representing corresponding information to render for one or more of the plurality of discrete client device modalities;
• causing the client device output to be rendered by one or more user interface output devices of the multimodal client device;
• while at least part of the client device output is being rendered by the one or more user interface output devices of the multimodal client device:
• detecting a switch of the multimodal client device, based on alternative sensor data from the one or more sensors of the multimodal client device, from the current client device modality to a discrete new client device modality;
• in response to detecting the switch, generating alternative client device output based on processing (1) the alternative sensor data from the one or more sensors of the multimodal client device and (2) the multimodal response, using the machine learning model;
• and
• causing the alternative client device output to be rendered by the one or more user interface output devices of the multimodal client device.