# US 2015/0106768 A1 · Three Dimensional User Interface Effects On A Display By Using Properties Of Motion

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0106768 A1
- Document ID: US20150106768A1
- Status: Published application
- Kind: A1
- Application number: 14/571,062
- Filed: 2014-12-15
- Published: 2015-04-16
- Earliest claimed: 2010-08-04
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Mark Zimmer; Geoff Stahl; David Hayward; Frank Doepke
- Web page: https://patentgrep.com/patent/US20150106768A1

## CPC

- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/005: {Input arrangements through a video camera}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06T 15/20: Perspective computation
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The techniques disclosed herein use a compass, MEMS accelerometer, GPS module, and MEMS gyrometer to infer a frame of reference for a hand-held device. This can provide a true Frenet frame, i.e., X- and Y-vectors for the display, and also a Z-vector that points perpendicularly to the display. In fact, with various inertial clues from accelerometer, gyrometer, and other instruments that report their states in real time, it is possible to track the Frenet frame of the device in real time to provide a continuous 3D frame-of-reference. Once this continuous frame of reference is known, the position of a user's eyes may either be inferred or calculated directly by using a device's front-facing camera. With the position of the user's eyes and a continuous 3D frame-of-reference for the display, more realistic virtual 3D depictions of the objects on the device's display may be created and interacted with by the user.

## Claims

### Claim 1

A method, comprising:
generating a 2D depiction of a graphical user interface on a display of a device, wherein the graphical user interface comprises at least one graphical user interface object;
receiving positional data from one or more positional sensors disposed within the device;
determining a 3D frame of reference for the device based, at least in part, on the received positional data;
receiving optical data from one or more optical sensors disposed within the device;
determining a position of a user of the device's eyes based, at least in part, on the received optical data;
detecting an activating gesture based, at least in part, on the received positional data; and
generating a virtual 3D depiction of the graphical user interface on the display of the device in response to the detection of the activating gesture,
wherein the act of generating the virtual 3D depiction is based, at least in part, on the determined 3D frame of reference and the determined position of the user's eyes.

### Claim 2 (depends on 1)

The method of claim 1, further comprising the acts of:
detecting a deactivating gesture based, at least in part, on the received positional data; and
generating a 2D depiction of the graphical user interface in response to the detection of the deactivating gesture.

### Claim 3 (depends on 1)

The method of claim 1, wherein the activating gesture comprises a one-axis sine wave modulation of the device of a sufficient number of degrees and within a predetermined threshold amount of time.

### Claim 4 (depends on 3)

The method of claim 3, wherein the deactivating gesture comprises positional quiescence of the device for at least a predetermined threshold amount of time.

### Claim 5 (depends on 1)

The method of claim 1, wherein the act of generating a virtual 3D depiction of the graphical user interface comprises transforming a 2D depiction of at least one graphical user interface object into a 3D depiction of the graphical user interface object.

### Claim 6 (depends on 5)

The method of claim 5, wherein the 3D depiction of the graphical user interface object comprises a cube.

### Claim 7 (depends on 6)

The method of claim 6, wherein the cube has at least one rounded surface.

### Claim 8

An apparatus, comprising:
a display;
one or more optical sensors;
one or more positional sensors;
a memory; and
one or more programmable control devices communicatively coupled to the display, the optical sensors, the positional sensors, and the memory,
wherein the memory includes instructions for causing the one or more programmable control devices to:
generate a 2D depiction of a graphical user interface on the display, wherein the graphical user interface comprises at least one graphical user interface object;
receive positional data from the one or more positional sensors;
determine a 3D frame of reference for the apparatus based, at least in part, on the received positional data;
receive optical data from the one or more optical sensors;
determine a position of a user of the apparatus's eyes based, at least in part, on the received optical data;
detect an activating gesture based, at least in part, on the received positional data; and
generate a virtual 3D depiction of the graphical user interface on the display in response to the detection of the activating gesture,
wherein the instructions for causing the one or more programmable control devices to generate the virtual 3D depiction are based, at least in part, on the determined 3D frame of reference and the determined position of the user's eyes.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the memory includes further instructions for causing the one or more programmable control devices to:
detect a deactivating gesture based, at least in part, on the received positional data; and
generate a 2D depiction of the graphical user interface in response to the detection of the deactivating gesture.

### Claim 10 (depends on 8)

The apparatus of claim 8, wherein the activating gesture comprises a one-axis sine wave modulation of the apparatus of a sufficient number of degrees and within a predetermined threshold amount of time.

### Claim 11 (depends on 10)

The apparatus of claim 10, wherein the deactivating gesture comprises positional quiescence of the apparatus for at least a predetermined threshold amount of time.

### Claim 12 (depends on 8)

The apparatus of claim 8, wherein the instructions for causing the one or more programmable control devices to generate a virtual 3D depiction of the graphical user interface comprise instructions for causing the one or more programmable control devices to transform a 2D depiction of at least one graphical user interface object into a 3D depiction of the graphical user interface object.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein the 3D depiction of the graphical user interface object comprises a cube.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the cube has at least one rounded surface.

### Claim 15

A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause one or more processing units to:
generate a 2D depiction of a graphical user interface on a display of a device, wherein the graphical user interface comprises at least one graphical user interface object;
receive positional data from one or more positional sensors disposed within the device;
determine a 3D frame of reference for the device based, at least in part, on the received positional data;
receive optical data from one or more optical sensors disposed within the device;
determine a position of a user of the device's eyes based, at least in part, on the received optical data;
detect an activating gesture based, at least in part, on the received positional data; and
generate a virtual 3D depiction of the graphical user interface on the display of the device in response to the detection of the activating gesture,
wherein the instructions to cause the one or more processing units to generate the virtual 3D depiction are based, at least in part, on the determined 3D frame of reference and the determined position of the user's eyes.

### Claim 16 (depends on 15)

The non-transitory program storage device of claim 15, further comprising instructions stored thereon to cause one or more processing units to:
detect a deactivating gesture based, at least in part, on the received positional data; and
generate a 2D depiction of the graphical user interface in response to the detection of the deactivating gesture.

### Claim 17 (depends on 15)

The non-transitory program storage device of claim 15, wherein the activating gesture comprises a one-axis sine wave modulation of the device of a sufficient number of degrees and within a predetermined threshold amount of time.

### Claim 18 (depends on 17)

The non-transitory program storage device of claim 17, wherein the deactivating gesture comprises positional quiescence of the device for at least a predetermined threshold amount of time.

### Claim 19 (depends on 15)

The non-transitory program storage device of claim 15, wherein the instructions to cause the one or more processing units to generate a virtual 3D depiction of the graphical user interface further comprise instructions to cause the one or more processing units to transform a 2D depiction of at least one graphical user interface object into a 3D depiction of the graphical user interface object.

### Claim 20 (depends on 19)

The non-transitory program storage device of claim 19, wherein the 3D depiction of the graphical user interface object comprises a cube.
