Contextually disambiguating queries

A system, comprising:

• one or more processors;
• and
• memory storing instructions that, when executed by one or more of the processors, cause the one or more processors to perform operations comprising:
• determining, by a client device of a user, to generate a search query for the user based on an image capturing screen content displayed by the client device at a particular time and based on a voice input of the user received subsequent to the particular time;
• processing the image of the screen content displayed by the client device at the particular time to identify a particular sub-image of a plurality of disparate sub-images included in the image of the screen content displayed by the client device at the particular time;
• processing a plurality of separate portions of the particular sub-image to generate a plurality of labels that each correspond to at least one of the separate portions of the particular sub-image included in the image of the screen content displayed by the client device at the particular time;
• receiving, by the client device and subsequent to the particular time, audio data including the voice input of the user;
• selecting a particular subset of the plurality of labels based on a transcription of the voice input of the user and based on identifying a screen content type associated with a particular portion of the particular sub-image, of the plurality of separate portions of the particular sub-image, that is associated with the particular subset of the plurality of labels;
• generating the search query for the user based on the transcription of the voice input of the user and the particular selected subset of the plurality of labels that correspond to the particular portion of the particular sub-image;
• and
• providing, for display at the client device of the user, one or more search results obtained responsive to the search query that was generated for the user.

One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• determining, by a client device of a user, to generate a search query for the user based on an image capturing screen content displayed by the client device at a particular time and based on a voice input of the user received subsequent to the particular time;
• processing the image of the screen content displayed by the client device at the particular time to identify a particular sub-image of a plurality of disparate sub-images included in the image of the screen content displayed by the client device at the particular time;
• processing a plurality of separate portions of the particular sub-image to generate a plurality of labels that each correspond to at least one of the separate portions of the particular sub-image included in the image of the screen content displayed by the client device at the particular time;
• receiving, by the client device and subsequent to the particular time, audio data including the voice input of the user;
• selecting a particular subset of the plurality of labels based on a transcription of the voice input of the user and based on identifying a screen content type associated with a particular portion of the particular sub-image, of the plurality of separate portions of the particular sub-image, that is associated with the particular subset of the plurality of labels;
• generating the search query for the user based on the transcription of the voice input of the user and the particular selected subset of the plurality of labels that correspond to the particular portion of the particular sub-image;
• and
• providing, for display at the client device of the user, one or more search results obtained responsive to the search query that was generated for the user.