Multi-user authentication on a device

A system comprising:

• one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
• receiving, by a device, an utterance from a speaker;
• classifying, by the device, the speaker of the utterance as not a known user of the device;
• providing, by the device and to a server, a query that includes authentication tokens that correspond to known users of the device, a representation of the utterance, and an indication that the speaker was classified as not a known user of the device;
• and
• receiving, by the device and from the server, a response to the query that is generated based on the authentication tokens that correspond to known users of the device, the representation of the utterance, and the indication that the speaker was classified as not a known user of the device.

A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

• receiving, by a device, an utterance from a speaker;
• classifying, by the device, the speaker of the utterance as not a known user of the device;
• providing, by the device and to a server, a query that includes authentication tokens that correspond to known users of the device, a representation of the utterance, and an indication that the speaker was classified as not a known user of the device;
• and
• receiving, by the device and from the server, a response to the query that is generated based on the authentication tokens that correspond to known users of the device, the representation of the utterance, and the indication that the speaker was classified as not a known user of the device.