Hand and Gaze-Based Interactions with an External Device's Display
AssigneeApple Inc.
InventorsNathan Jean Emmanuel Biette, Damien Dalla-Rosa, Vincent Chapdelaine-Couture
Life of this patent
Estimating the term…
Abstract
Various implementations disclosed herein include devices, systems, and methods in which a head-mounted device (HMD) tracks a user's gaze and hand to augment the input capabilities of a detected display of another device, e.g., a laptop, TV, automobile device, etc. In some implementations, sensor data captured via a wearable device is used to identify a position of another device's display and/or recognize gaze and/or hand activity to identify interactions associated with content displayed on that other device's display.
Claims20
A method comprising:
- at a head-mounted device (HMD) comprising a processor and one or more sensors:
- determining a three-dimensional position of a display of a second device, wherein content originating from the second device is visible to a user of the HMD;
- determining a gaze direction of the user based on sensor data obtained via the one or more sensors;
- identifying a portion of the content based on the gaze direction and the 3D position of the display;
- tracking a hand of the user based on sensor data obtained via the one or more sensors;
- and
- providing information to the second device based on identifying the portion of the content and the hand tracking, wherein the second device detects input associated with the content based on the information.
A system comprising:
- a non-transitory computer-readable storage medium;
- and
- one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
- determining a three-dimensional position of a display of a second device, wherein content originating from the second device is visible to a user of the HMD;
- determining a gaze direction of the user based on sensor data obtained via the one or more sensors;
- identifying a portion of the content based on the gaze direction and the 3D position of the display;
- tracking a hand of the user based on sensor data obtained via the one or more sensors;
- and
- providing information to the second device based on identifying the portion of the content and the hand tracking, wherein the second device detects input associated with the content based on the information.
A non-transitory computer-readable storage medium storing program instructions executable via one or more processors to perform operations comprising:
- determining a three-dimensional position of a display of a second device, wherein content originating from the second device is visible to a user of the HMD;
- determining a gaze direction of the user based on sensor data obtained via the one or more sensors;
- identifying a portion of the content based on the gaze direction and the 3D position of the display;
- tracking a hand of the user based on sensor data obtained via the one or more sensors;
- and
- providing information to the second device based on identifying the portion of the content and the hand tracking, wherein the second device detects input associated with the content based on the information.
Dependencies are inferred from the claim text.