Multimodal search on wearable smart devices

A computing apparatus comprising:

• a non-transitory computer-readable storage medium;
• at least one processor operatively coupled to the non-transitory computer-readable storage medium;
• and
• program instructions stored on the non-transitory computer-readable storage medium that, when executed by the at least one processor, direct the computing apparatus to perform a method, the method comprising:
• obtaining, at a device, a query from a user of the device;
• identifying context associated with the query, the context comprising a set of images captured by a camera on the device, wherein the camera is positioned to capture a physical environment associated with a perspective of the user;
• generating at least one search from the query and the context, wherein generating the at least one search from the query and the context comprises:
• processing the set of images to identify an object referenced in association with the query;
• and
• generating the at least one search based on the query, the context, and the object;
• obtaining at least one result for the at least one search;
• determining a response to the query based on the at least one result;
• and
• providing the response to the user of the device.

A non-transitory computer-readable storage medium having program instructions stored there on that, when executed by at least one processor, direct the at least one processor to perform a method, the method comprising:

• obtaining, at a device, a query from a user of the device;
• identifying context associated with the query, the context comprising at least one image captured by a camera on the device, wherein the camera is positioned to capture a physical environment associated with a perspective of the user;
• generating at least one search from the query and the context, wherein generating the at least one search from the query and the context comprises:
• identifying a gesture in the at least one image;
• processing the at least one image to identify an object referenced in association with the query based on the gesture;
• and
• generating the at least one search based on the query, the context, and the object;
• obtaining at least one result for the at least one search;
• determining a response to the query based on the at least one result;
• and
• providing the response to the user of the device.