# US 8,279,168 B2 · Three-dimensional virtual-touch human-machine interface system and method therefor

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 8,279,168 B2
- Document ID: US8279168B2
- Status: Granted
- Kind: B2
- Application number: 11/567,888
- Filed: 2006-12-07
- Published: 2007-06-14
- Granted: 2012-10-02
- Earliest claimed: 2005-12-09
- Assignees: Edge 3 Technologies, Inc.
- Inventors: William E. Glomski; Tarek El Dokor; Joshua T. King; James E. Holmes; Maria N. Ngomba
- Web page: https://patentgrep.com/patent/US8279168B2

## CPC

- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A three-dimensional virtual-touch human-machine interface system (20) and a method (100) of operating the system (20) are presented. The system (20) incorporates a three-dimensional time-of-flight sensor (22), a three-dimensional autostereoscopic display (24), and a computer (26) coupled to the sensor (22) and the display (24). The sensor (22) detects a user object (40) within a three-dimensional sensor space (28). The display (24) displays an image (42) within a three-dimensional display space (32). The computer (26) maps a position of the user object (40) within an interactive volumetric field (36) mutually within the sensor space (28) and the display space (32), and determines when the positions of the user object (40) and the image (42) are substantially coincident. Upon detection of coincidence, the computer (26) executes a function programmed for the image (42).

## Claims

### Claim 1

A three-dimensional virtual-touch human-machine interface system comprising:
a three-dimensional time-of-flight object sensor configured to detect a position of a user object within a first three-dimensional space, the first three-dimensional space bounded by a sensor field of view;
a three-dimensional autostereoscopic display configured to display one or more images within a second three-dimensional space, the second three-dimensional space bounded by display limits, so that a user interacting with the three-dimensional display is able to view and interact with each of the one or more images as if it were in three-dimensional space, with each of the one or more images having a breadth corresponding to a size in an x-dimension, a height corresponding to a size in a y-dimension, and a depth corresponding to a size in a z-dimension, and each of said one or more images having at least one corresponding active region being defined by x, y and z coordinates, each active region of a corresponding image being a portion of the corresponding image that is programmed to be interactive with a user object; and
a computer coupled to said sensor and said display, configured to map said position of said user object within a third three-dimensional space including x, y and z axis coordinates wherein the third three-dimensional space is common to both the first three-dimensional space and the second three-dimensional space, configured to map a position of each of said one or more active regions within said third three-dimensional space, and configured to determine when said positions of said user object and one or more of said active regions of said one or more images are substantially coincident within said third three-dimensional space, so that the computer can differentiate between two or more of said active regions differing only in z axis coordinates in said third three-dimensional space, the computer executing a function programmed for an active region upon determining that the positions of said user object and one or more of said active regions of said one or more images are substantially coincident.

### Claim 2 (depends on 1)

A system as claimed in claim 1 wherein said display is a three-dimensional autostereoscopic liquid crystal display.

### Claim 3 (depends on 1)

A system as claimed in claim 1 wherein said display is a three-dimensional autostereoscopic plasma-screen display.

### Claim 4 (depends on 1)

A system as claimed in claim 1 wherein said three-dimensional display is configured to display said one or more active regions of said one or more images in said first three-dimensional space.

### Claim 5 (depends on 1)

A system as claimed in claim 1 wherein said third three-dimensional space is mutually within said first and second three-dimensional spaces.

### Claim 6 (depends on 1)

A system as claimed in claim 1 wherein: said first three-dimensional space is bound by first limits; said second three-dimensional space is bound by second limits; said third three-dimensional space is bound by said first and second limits.

### Claim 7 (depends on 6)

A system as claimed in claim 6 wherein said third three-dimensional space is unbounded in a direction away from said display.

### Claim 8 (depends on 1)

A system as claimed in claim 1 wherein: said one or more images is one of a plurality of images; and said display is configured to display said plurality of images within said second three-dimensional space so that a user interacting with the three-dimensional display is able to view the plurality of images as if they were in three-dimensional space.

### Claim 9 (depends on 1)

A system as claimed in claim 1 wherein: said user object is one of a plurality of user objects; and said sensor is configured to detect said one user object within said first three-dimensional space.

### Claim 10

A method of determining virtual touch within a three-dimensional virtual-touch human-machine interface system, said method comprising:
displaying, through the use of a three-dimensional autostereoscopic display, an initial three-dimensional image within a three-dimensional space so that a user interacting with the three-dimensional image is able to view and interact with the image as if it was positioned in the three-dimensional space, wherein the image has a breadth corresponding to a size in an x-dimension, a height corresponding to a size in a y-dimension, and a depth corresponding to a size in a z-dimension, and the image has a corresponding active region defined by x, y and z coordinates in the three-dimensional space, the active region of the corresponding image being a portion of the corresponding image that is programmed to be interactive with a user object;
detecting a presence of a user object including x, y and z axis coordinates within the three-dimensional space through the use of a three-dimensional time-of-flight sensor;
mapping, through a computer, a position of said user object and a position of the active region within the three-dimensional space including x, y and z axis coordinates; and
determining, by said computer, if a position of said user object is substantially coincident with a position of said active region of said image along each of the x, y and z axis coordinates within said three-dimensional space so that the user is able to selectively interact with, and the computer can differentiate between, said active region corresponding to said three-dimensional image displayed by the three-dimensional autostereoscopic display, and
executing, by said computer, a function programmed for said active region when said determining activity determines that the position of said user object and the position of the active region of said three-dimensional image are substantially coincident; and
updating said three-dimensional image when said executing activity executes said function.

### Claim 11

A method as claimed in 10 additionally comprising returning to said detecting activity following said updating activity.

### Claim 12 (depends on 10)

A method as claimed in claim 10 wherein: said mapping, determining, executing, and updating activities are executed when said detecting activity detects said presence of said user object; and said detecting activity is iterated when said detecting activity fails to detect said presence of said user object.

### Claim 13 (depends on 10)

A method as claimed in claim 10 additionally comprising returning to said detecting activity when said determining activity determines said user object is not substantially coincident with said active region.

### Claim 14 (depends on 10)

A method as claimed in claim 10 additionally comprising:
ascertaining, when said determining activity determines said user object is substantially coincident with said active region, if a function programmed for said active region of said image is to be executed; and
executing said function when said ascertaining activity ascertains said function is to be executed; and updating said three-dimensional image when said executing activity executes said function.

### Claim 15 (depends on 14)

A method as claimed in claim 14 additionally comprising returning to said detecting activity when said ascertaining activity ascertains said function is not to be executed.

### Claim 16 (depends on 10)

A method as claimed in claim 10 additionally comprising establishing said position of said user object prior to said mapping activity.

### Claim 17 (depends on 10)

A method as claimed in claim 10 wherein:
said displaying activity displays said initial three-dimensional image containing at least one image wherein a plurality of active regions of said at least one image is in said three-dimensional space;
said detecting activity detects a presence of a plurality of user objects within said three-dimensional space;
said mapping activity maps said position of each of said user objects within said three-dimensional space;
said determining activity determines if each of said user objects is substantially coincident with one of said plurality of active regions; and
said executing activity executes said function when said determining activity determines each of said user objects is substantially coincident with one of said plurality of active regions.

### Claim 18

A three-dimensional virtual-touch human-machine interface system comprising:
a three-dimensional time-of-flight sensor configured to detect a position of a user object having x, y and z axis coordinates within a first three-dimensional space bound by first limits corresponding to a sensor field of view;
a three-dimensional autostereoscopic display configured to display an image within a second three-dimensional space bound by second limits corresponding to display limits, said image having at least one corresponding active region having x, y and z axis coordinates within a third three-dimensional space bound by said first and second limits, mutually within said first and second three-dimensional spaces, and unbounded in a direction away from said display so that a user interacting with the three-dimensional display is able to view and interact with the image as if it were in three-dimensional space, with the image having a breadth corresponding to a size in an x-dimension, a height corresponding to a size in a y-dimension, and a depth corresponding to a size in a z-dimension, and with each active region of a corresponding image being a portion of the corresponding image that is programmed to be interactive with a user object; and
a computer coupled to said sensor and said display, configured to map a position of said user object within said third three-dimensional space, configured to map a position of said image within said third three-dimensional space along with said at least one corresponding active regions, and configured to determine when said positions of said user object and said at least one corresponding active regions of said image are substantially coincident within said third three-dimensional space along x, y and z axis coordinates so that the computer can differentiate between two or more active regions differing only in z axis coordinates in said third three-dimensional space, the computer executing a function programmed for an active region upon determining that the positions of said user object and one or more of said active regions of said image are substantially coincident.

### Claim 19

A method of determining virtual touch within a three-dimensional virtual-touch human-machine interface system, said method comprising:
displaying, through the use of a three-dimensional autostereoscopic display, first and second three-dimensional images within a three-dimensional space so that a user interacting with the three-dimensional images is able to view and interact with the images as if they were positioned in the three-dimensional space, wherein the first and second three-dimensional images have a breadth corresponding to a size in an x-dimension, a height corresponding to a size in a v-dimension, and a depth corresponding to a size in a z-dimension, wherein each of the first and second three-dimensional images has at least one corresponding active region defined by x, y and z axis coordinates in the three-dimensional space, and wherein at least a portion of each of the active regions of the first and second three-dimensional images differ only in z axis coordinates, the active region of the corresponding first and second three-dimensional images being a portion of the corresponding image that is programmed to be interactive with a user object;
detecting a presence of a user object including x, y and z axis coordinates within the three-dimensional space through the use of a three-dimensional time-of-flight sensor;
mapping, through a computer, a position of said user object and a position of the active regions within the three-dimensional space including x, y and z axis coordinates; and
determining, by said computer, if a position of said user object is substantially coincident with a position of said active region of one of said first and second three-dimensional images along each of the x, y and z axis coordinates within said three-dimensional space so that the user is able to selectively interact with, and the computer can differentiate between, said active regions of the first and second three-dimensional images displayed by the three-dimensional autostereoscopic display, and
executing, by said computer, a function programmed for said active region when said determining activity determines that the position of said user object and the position of the active region of the first or second three-dimensional image are substantially coincident; and
updating said first or second three-dimensional image when said executing activity executes said function.
