Invoking automated assistant function(s) based on detected gesture and gaze

A client device comprising:

• at least one vision component;
• at least one microphone;
• one or more processors;
• memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by one or more of the processors, cause one or more of the processors to perform the following operations:
• receiving a stream of vision data that is based on output from the vision component of the client device;
• receiving a stream of audio data that is based on output from the microphone of the client device;
• processing the vision data using at least one trained machine learning model stored locally on the client device to monitor for occurrence of both:
• an invocation gesture of a user captured by the vision data, and
• a gaze of the user that is directed toward the client device;
• detecting, based on the monitoring, occurrence of both:
• the invocation gesture, and
• the gaze;
• determining that the audio data corresponds to the user that provided the invocation gesture and the gaze;
• and
• in response to detecting the occurrence of both the invocation gesture and the gaze, and contingent on determining that the audio data corresponds to the user that provided the invocation gesture and the gaze:
• transmitting, by the client device to one or more remote automated assistant components, one or both of:
• additional vision data that is based on output from the vision component, and
• at least a portion of the audio data that is based on output from the microphone of the client device.