Augmented Reality System for Interacting with Objects Represented in a Physical Document

A non-transitory computer-readable storage medium storing instructions, that when executed, causes a processor to perform steps comprising:

• receiving, from a computer system, image data of a document depicting one or more items;
• detecting a user interaction with the document in the image data;
• identifying an item in the document corresponding to the user interaction by:
• detecting a set of coordinates of the document corresponding to the user interaction;
• and
• accessing coordinate data associated with an identifier of the document, wherein the coordinate data includes mappings of coordinates of the document to items;
• detecting a gesture that occurred based on the user interaction and the item;
• correlating the gesture with the identified item;
• and
• causing the computer system to invoke a function corresponding to the gesture and identified item.

A system comprising:

• a processor;
• and
• a non-transitory computer-readable storage medium storing instructions, that when executed, causes the processor to perform steps comprising:
• receiving, from a computer system, image data of a document depicting one or more items;
• detecting a user interaction with the document in the image data;
• identifying an item in the document corresponding to the user interaction;
• detecting a gesture that occurred based on the user interaction and the item;
• correlating the gesture with the identified item;
• and
• causing the computer system to invoke a function corresponding to the gesture and identified item.