Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking

An apparatus for processing one or more images, said apparatus comprising:

• a memory;
• at least one processor coupled to the memory, wherein the at least one processor is configured to:
• obtain at least one image of a scene in a field of view;
• identify a portion of the scene based on a first user interaction;
• display augmented reality (AR) content in response to the first user interaction, the AR content comprising content associated with the identified portion of the scene;
• receiving, after displaying the AR content, an indication to initiate an online visual search of the identified portion of the scene based on a second user interaction, the second user interaction including a non-eye gaze input and occurring after the first user interaction;
• and
• cause, in response to the second user interaction, transmission of an image of the identified portion of the scene to a remote device configured to generate search results based on the image, wherein the image includes the identified portion of the scene and does not include content in the field of view outside of the identified portion of the scene.

A non-transitory computer-readable medium having instructions stored thereon that, when executed, cause at least one processor of an augmented reality (AR) device having a display to:

• obtain at least one image of a scene in a field of view of the display;
• identify a portion of the scene based on a first user interaction;
• display augmented reality (AR) content on the display in response to the first user interaction, the AR content comprising content associated with the identified portion of the scene;
• receiving, after displaying the AR content on the display, an indication to initiate an online visual search of the identified portion of the scene based on a second user interaction, the second user interaction including a non-eye gaze input and occurring after the first user interaction;
• and
• cause, in response to the second user interaction, transmission of an image of the identified portion of the scene to a remote device configured to generate search results based on the image, wherein the image includes the identified portion of the scene and does not include content in the field of view of the display outside of the identified portion of the scene.