# US 2011/0298798 A1 · Three-dimensional imaging and display system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2011/0298798 A1
- Document ID: US20110298798A1
- Status: Published application
- Kind: A1
- Application number: 13/212,122
- Filed: 2011-08-17
- Published: 2011-12-08
- Earliest claimed: 2005-10-21
- Assignees: Apple Inc.
- Inventors: Christoph H. KRAH
- Web page: https://patentgrep.com/patent/US20110298798A1

## CPC

- G06F 3/0325: {using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image}
- G01B 11/00: Measuring arrangements characterised by the use of optical techniques
- G01C 3/085: {with electronic parallax measurement}
- G01S 7/4817: {relating to scanning}
- G01S 7/4861: Circuits for detection, sampling, integration or read-out
- G01S 7/4865: Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak (peak detection in noise, signal conditioning G01S7/487)
- G01S 17/36: with phase comparison between the received signal and the contemporaneously transmitted signal
- G01S 17/48: Active triangulation systems, i.e. using the transmission and reflection of electromagnetic waves other than radio waves
- G01S 17/66: Tracking systems using electromagnetic waves other than radio waves
- G06F 3/005: {Input arrangements through a video camera}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/02: Input arrangements using manually operated switches, e.g. using keyboards or dials
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
- G01C 3/12: with monocular observation at a single point, e.g. coincidence type (G01C3/20 takes precedence)
- G01S 17/14: wherein a voltage or current pulse is initiated and terminated in accordance with the pulse transmission and echo reception respectively, e.g. using counters
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES

## Abstract

A three-dimensional imaging and display system is provided in which user input is optically detected in an imaging volume by measuring the path length of an amplitude modulated scanning beam as a function of the phase shift thereof. Visual image user feedback concerning the detected user input is presented.

## Claims

### Claim 1

A three-dimensional imaging and display apparatus comprising:
a processor;
a first circuit coupled to the processor, wherein the first circuit is configured to optically detect one or more user inputs in an imaging volume by measuring a path length of a modulated scanning beam as a function of a phase shift, wherein the path length of the modulated scanning beam is separately measured at a plurality of angles; and
a second circuit coupled to the processor for presenting visual image user feedback concerning the detected user input.

### Claim 2 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 wherein the first circuit is further configured to amplitude modulate the output intensity of a scanning laser beam.

### Claim 3 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 wherein the first circuit is further configured to optically detect the modulated scanning beam with a PIN diode detector.

### Claim 4 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 wherein the first circuit is further configured to calibrate the phase shift with a distance the scanning beam travels.

### Claim 5 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 wherein the first circuit is further configured to determine the path length L from the phase shift φ and the wavelength λ wherein:
L

(
V
OUT
)
=
-
c
360
·
f
MOD
·
arcsin

(
2
·
V
OUT
V
M
2
)
.

### Claim 6 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 further comprising:
a receiver coupled to the first circuit, wherein the receiver is configured to receive the modulated scanning beam after the modulated scanning beam has been reflected by the user input;
a mixer coupled to the first circuit, wherein the mixer is configured to homodyne mix a modulation frequency of the scanning beam with a signal from the receiver;
a third circuit coupled to the first circuit, wherein the third circuit is configured to isolate a phase shift from the homodyne mixed signal; and
a fourth circuit coupled to the first circuit, wherein the fourth circuit is configured to derive a distance from the phase shift, wherein the distance is a distance that the scanning beam has traveled.

### Claim 7 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 further comprising:
a third circuit coupled to the processor, wherein the third circuit is configured to display and manipulate virtual objects located in a three-dimensional imaging volume in response to virtual interactions with the virtual objects, the virtual interactions being derived from the detected user input, wherein the virtual objects are moved, scaled, or rotated in the imaging volume.

### Claim 8 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 further comprising a third circuit coupled to the processor, wherein the third circuit is configured to display externally provided images.

### Claim 9 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 wherein the first circuit is configured to perform optical detection with invisible light.

### Claim 10 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 further comprising a third circuit coupled to the processor, wherein the third circuit is configured to track one or more heads.

### Claim 11 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 further comprising:
a third circuit coupled to the processor, wherein the third circuit is configured to image a user's head in the imaging volume; and
a fourth circuit coupled to the processor, wherein the fourth circuit is configured to map an image of the user's head to at least one interface related function.

### Claim 12 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 further comprising:
a third circuit coupled to the processor, wherein the third circuit is configured to image hand gestures in the imaging volume; and
a fourth circuit coupled to the processor, wherein the fourth circuit is configured to map hand gestures to at least one interface related function.

### Claim 13 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 wherein the first circuit is further configured to measure the size of objects.

### Claim 14 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 further comprising a third circuit coupled to the processor, wherein the third circuit is configured to substitute or replace images.

### Claim 15 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 further comprising a third circuit coupled to the processor, wherein the third circuit is configured to paint and follow a moveable object as it moves in space.

### Claim 16 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 further comprising a third circuit coupled to the processor, wherein the third circuit is configured to generate an interactive virtual keyboard.

### Claim 17 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 further comprising a third circuit coupled to the processor, wherein the third circuit is configured to assign functions to objects, wherein a type of one or more objects is optically detected in the imaging volume and a predetermined function or set of functions is assigned with the one or more objects detected.

### Claim 18 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 further comprising a third circuit coupled to the processor, wherein the third circuit is configured to customize an environment in which the three-dimensional imaging and display apparatus is deployed.

### Claim 19 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 further comprising:
a third circuit coupled to the processor, wherein the third circuit is configured to detect elements of the user input with watershed segmentation;
a fourth circuit coupled to the processor, wherein the fourth circuit is configured to track individual and relative movements and positions of the elements of the user input; and
a fifth circuit coupled to the processor, wherein the fifth circuit is configured to map tracked movements and positions to predetermined functions.

### Claim 20 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 further comprising a minor operating at a resonant frequency.

### Claim 21 (depends on 1)

The three-dimensional imaging and display apparatus of claim 1 further comprising an auxiliary display configured to display one or more three-dimensional or two-dimensional images.

### Claim 22

A method for detecting and providing image feedback comprising:
optically detecting a user input by measuring a path length of a modulated scanning beam as a function of a phase shift, wherein the path length of the modulated scanning beam is separately measured at a plurality of angles; and
presenting visual images on a display area, wherein the visual images are presented in response to the user input.

### Claim 23 (depends on 22)

The method of claim 22 wherein optically detecting further comprises amplitude modulating the output intensity of a single scanning laser beam.

### Claim 24 (depends on 22)

The method of claim 22 wherein optically detecting further comprises detecting the modulated scanning beam with a PIN diode detector.

### Claim 25 (depends on 22)

The method of claim 22 wherein optically detecting further comprises calibrating the phase shift with a distance that the scanning beam travels.

### Claim 26 (depends on 22)

The method of claim 22 wherein optically detecting further comprises determining the path length L from the phase shift φ and the wavelength λ wherein:
L

(
V
OUT
)
=
-
c
360
·
f
MOD
·
arcsin

(
2
·
V
OUT
V
M
2
)
.

### Claim 27 (depends on 22)

The method of claim 22 further comprising:
receiving the modulated scanning beam on a receiver after the modulated scanning beam has been reflected by the user input;
homodyne mixing a modulation frequency of the scanning beam with a signal from the receiver;
isolating a phase shift from the homodyne mixed signal; and
deriving a distance from the phase shift, wherein the distance is a distance that the scanning beam has traveled.

### Claim 28 (depends on 22)

The method of claim 22 further comprising:
displaying virtual objects located in a three-dimensional imaging volume, wherein the virtual interactions are derived from the detected user input;
manipulating the virtual objects in response to virtual interactions with the virtual objects, wherein the manipulated objects are moved, scaled, or rotated.

### Claim 29 (depends on 22)

The method of claim 22 wherein presenting visual images further comprises displaying images that have been externally provided.

### Claim 30 (depends on 22)

The method of claim 22 wherein optically detecting further comprises detecting with invisible light.

### Claim 31 (depends on 22)

The method of claim 22 further comprising tracking one or more heads.

### Claim 32 (depends on 22)

The method of claim 22 further comprising:
imaging a user's head in the imaging volume; and
mapping an image of the user's head to one or more interface related function.

### Claim 33 (depends on 22)

The method of claim 22 further comprising:
imaging hand gestures in the imaging volume; and
mapping hand gestures to one or more interface related functions.

### Claim 34 (depends on 22)

The method of claim 22 further comprising measuring the size of the object.

### Claim 35 (depends on 22)

The method of claim 22 further comprising substituting or replacing the image presented.

### Claim 36 (depends on 22)

The method of claim 22 further comprising painting and following a movable object.

### Claim 37 (depends on 22)

The method of claim 22 further comprising generating an interactive virtual keyboard.

### Claim 38 (depends on 22)

The method of claim 22 further comprising:
optically detecting a type of one or more objects in the imaging volume; and
assigning one or more pre-determined functions to the one or more objects detected.

### Claim 39 (depends on 22)

The method of claim 22 further comprising customizing an environment for three dimensional imaging.

### Claim 40 (depends on 22)

The method of claim 22 further comprising:
detecting elements of the user input with watershed segmentation;
tracking individual and relative movements and positions of the elements of the user input; and
mapping tracked movements and positions to predetermined functions.

### Claim 41 (depends on 22)

The method of claim 22 wherein optically detecting further comprises optically detecting on one or more minors having a resonant frequency.

### Claim 42 (depends on 22)

The method of claim 22 further comprising presenting one or more images on an auxiliary display.
