Three-dimensional object tracking to augment display area

A mobile device comprising:

• a display;
• one or more processors coupled to the display;
• a camera configured to capture images of a region external to the display;
• and
• a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
• receiving image data from the camera, the image data including data representative of a surface acting as a virtual touchscreen in the region and a real object within the region;
• analyzing the image data to determine a three-dimensional (3D) orientation and location of the real object relative to the surface and a distance of the real object from the display;
• producing an interaction event describing an interaction of the real object with the surface;
• and
• causing the display to output display elements including (i) a first display element that includes a replication of the real object in the image superimposed on the display and oriented based, at least in part, on the 3D orientation and location of the real object relative to the surface and the distance from the display, and (ii) a second display element that includes a line representative of a history of a location of a point on the real object in the region.

A mobile device comprising:

• a touchscreen display;
• a depth camera directed to a space in front of the touchscreen display, the space including a surface acting as a virtual touchscreen;
• a processor configured to receive depth data from the depth camera, the processor configured to:
• determine, based on the depth data, a three-dimensional (3D) orientation of an object relative to the surface including a distance from the surface;
• produce a virtual touchscreen event describing an interaction of the object with the surface;
• provide the virtual touchscreen event to a software application capable of supporting 3D touch interactions;
• and
• cause the touchscreen display to output display elements including (i) a first display element that includes a replication of the object superimposed on the touchscreen display and oriented based, at least in part, on the 3D orientation and location of the object relative to the surface and the distance from the touchscreen display, and (ii) a second display element that includes a line representative of a history of a location of a point on the object in the space.

A method comprising:

• receiving image data from a camera, the image data including data representative of a surface acting as a virtual touchscreen in a region in a field of view of the camera and a real object within the region;
• analyzing the image data to determine a three-dimensional (3D) orientation and location of the real object relative to the surface and a distance of the real object from a display;
• producing an interaction event describing an interaction of the real object with the surface;
• and
• causing the display to output display elements including (i) a first display element that includes a replication of the real object in the image superimposed on the display and oriented based, at least in part, on the 3D orientation and location of the real object relative to the surface and the distance from the display, and (ii) a second display element that includes a line representative of a history of a location of a point on the real object in the region.