# US 2020/0272228 A1 · Interaction system of three-dimensional space and method for operating same

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0272228 A1
- Document ID: US20200272228A1
- Status: Published application
- Kind: A1
- Application number: 15/776,960
- Filed: 2017-09-29
- Published: 2020-08-27
- Earliest claimed: 2017-03-29
- Assignees: Boe Technology Group Co., Ltd.
- Applicants: Boe Technology Group Co., Ltd.
- Inventors: Xiaoliang DING; Xue DONG; Haisheng WANG; Chunwei WU; Yingming LIU; Pengpeng WANG; Chihjen CHENG; Yanling HAN; Xueyou CAO
- Web page: https://patentgrep.com/patent/US20200272228A1

## CPC

- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 15/20: Perspective computation
- G01S 13/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S13/46)
- G01S 13/862: {Combination of radar systems with sonar systems}
- G01S 13/867: {Combination of radar systems with cameras}
- G01S 13/878: {Combination of several spaced transmitters or receivers of known location for determining the position of a transponder or a reflector (G01S13/874 takes precedence)}
- G01S 13/88: Radar or analogous systems specially adapted for specific applications (electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00)
- G01S 15/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S15/46)
- G01S 15/88: Sonar systems specially adapted for specific applications (seismic or acoustic prospecting or detecting G01V1/00)
- G06T 2207/10048: Infrared image
- G06T 2207/30201: Face
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES

## Abstract

The present disclosure provides an interaction system of three-dimensional space and a method for operating the same, comprising: determining visual field space according to an eye position of an operator and an angle at which the eye is watching, and determining absolute interaction space and relative interaction space according to respective joint positions of the operator and the visual field space, thereby achieving dividing of areas of the space; determining an operation that matches a current action and needs to be performed by the operator on a selected virtual object according to a relationship among the current action of the operator, the virtual object viewed by the operator and the absolute interaction space, such that when the operator makes the corresponding action, the operation matching the action of the operator is performed on the virtual object; outputting the generated corresponding three-dimensional image display signal to a three-dimensional image display device to complete somatosensory interaction of the three-dimensional space.

## Claims

### Claim 1

An interaction system of three-dimensional space, comprising: an eye positioning device, a space positioning device, and a master control device, wherein:
said eye positioning device is used for determining an eye position of an operator and an angle at which the eye is watching and for outputting the eye position and the angle to said master control device;
said space positioning device is used for determining respective joint positions of the operator and for outputting the respective joint positions to said master control device;
said master control device includes: a visual field space processor, an interaction space processor, an operating action processor, an interaction matching processor and an image processor;
said visual field space processor being electrically connected to said eye positioning device, for determining visual field space according to the eye position of the operator and the angle at which the eye is watching;
said interaction space processor being electrically connected to said space positioning device and said visual field space processor, respectively, for determining interaction space according to the respective joint positions of the operator and using an area in said interaction space overlapping with said visual field space as absolute interaction space and the other areas in said interaction space as relative interaction space;
said operating action processor being electrically connected to said space positioning device, for determining a current action of the operator according to changes of the respective joint positions of the operator;
said interaction matching processor being electrically connected to said visual field space processor, said interaction space processor and said operating action processor, respectively, for determining an operation that matches the current action and needs to be performed on a selected virtual object by the operator according to a relationship among the current action of the operator, the virtual object viewed by the operator and the absolute interaction space;
said image processor being electrically connected to said interaction matching processor and said visual field space processor, respectively, for generating a corresponding three-dimensional image display signal according to the visual field space and the operation that matches the current action and needs to be performed on the selected virtual object by the operator and outputting the corresponding three-dimensional image display signal to an electrically connected three-dimensional image display device.

### Claim 2 (depends on 1)

The interaction system according to claim 1, wherein said operating action processor is further used for determining a current touch control action of the operator according to a change of the joint position of a touch control part of the operator; and determining a current body action of the operator according to a change of the joint position of a non-touch control part of the operator.

### Claim 3 (depends on 1)

The interaction system according to claim 1, wherein said interaction matching processor is further used for determining that the operator performs an operation of moving the virtual object positioned in said visual field space outside the interaction space into said absolute interaction space when it is determined that the current touch control action of the operator is within the relative interaction space.

### Claim 4 (depends on 3)

The interaction system according to claim 3, wherein said visual field space processor is further used for determining watching space within said visual field space according to an angle at which the eye of the operator is watching;
said interaction matching processor is further used for determining that the operator performs an operation of moving the virtual object positioned in said watching space outside the interaction space into said absolute interaction space when it is determined that the current touch control action of the operator is within the relative interaction space.

### Claim 5 (depends on 4)

The interaction system according to claim 4, wherein said interaction space processor is further used for using an area in the absolute interaction space overlapping with the watching space as refined interaction space;
said interaction matching processor is further used for determining that the operator performs a corresponding operation on the selected virtual object positioned in said absolute interaction space when it is determined that the current touch control action of the operator is within the absolute interaction space.

### Claim 6 (depends on 5)

The interaction system according to claim 5, wherein said interaction matching processor is further used for determining that the operator performs a corresponding operation on the selected virtual object positioned in said refined interaction space when it is determined that the current touch control action of the operator is within the absolute interaction space.

### Claim 7 (depends on 2)

The interaction system according to claim 2, wherein said image processor is further used for determining a position of a three-dimensional image of said virtual object in said three-dimensional image display device according to a change of a visual angle at which the operator watches the same virtual object when the eye position of the operator changes, and for generating a corresponding three-dimensional image display signal and outputting it to the electrically connected three-dimensional image display device, wherein a positional relationship between the same virtual object and said three-dimensional image display device is relatively fixed.

### Claim 8 (depends on 7)

The interaction system according to claim 7, wherein said image processor is further used for determining the position of the three-dimensional image of said virtual object in said three-dimensional image display device according to the following formula:
c
1
=
a
1
-
(
x
2
-
x
1
)

(
a
1
-
a
2
)
2
+
(
b
1
-
b
2
)
2
d
1
d
1
=
b
1
-
(
y
2
-
y
1
)

(
a
1
-
a
2
)
2
+
(
b
1
-
b
2
)
2
d
1
c
2
=
a
2
-
(
x
2
-
x
1
)

(
a
1
-
a
2
)
2
+
(
b
1
-
b
2
)
2
d
1
d
2
=
b
2
-
(
y
2
-
y
1
)

(
a
1
-
a
2
)
2
+
(
b
1
-
b
2
)
2
d
1
wherein (x1, y1) represents a position of a pupillary distance center of the eyes of the operator before a change, (x2, y2) represents a position of the pupillary distance center of the eyes of the operator after the change, (a1, b1) represents a left-eye image position of said virtual object in said three-dimensional image display device before the eye position of the operator changes, (a2, b2) represents a right-eye image position of said virtual object in said three-dimensional image display device before the eye position of the operator changes, (c1, d1) represents a left-eye image position of said virtual object in said three-dimensional image display device after the eye position of the operator changes, (a2, b2) represents a right-eye image position of said virtual object in said three-dimensional image display device after the eye position of the operator changes, and d1 represents the pupillary distance of the eyes of the operator.

### Claim 9 (depends on 1)

The interaction system according to claim 1, wherein said eye positioning device includes: a first IR emitter, a first IR camera, and a first processing chip, wherein:
said first IR emitter is used for emitting a first IR signal to the operator;
said first IR camera is used for acquiring a first IR signal image reflected by the eyes of the operator;
said first processing chip is used for determining an eye position of the operator and an angle at which the eye is watching according to the first IR signal emitted by the first IR emitter and the first IR signal image reflected by the eyes of the operator and acquired by the first IR camera, and for outputting them to said master control device.

### Claim 10 (depends on 9)

The interaction system according to claim 9, wherein said space positioning device includes: a second IR emitter, a second IR camera, and a second processing chip, wherein:
said second IR emitter is used for emitting a second IR signal to the operator;
said second IR camera is used for acquiring a second IR signal image reflected by the respective joint positions of the operator;
said second processing chip is used for determining the respective joint positions of the operator according to the second IR signal emitted by the second IR emitter and the second IR signal image reflected by the respective joint positions of the operator and acquired by the second IR camera, and for outputting them to said master control device.

### Claim 11 (depends on 10)

The interaction system according to claim 10, wherein said first IR emitter and said second IR emitter are the same IR emitter; and said first IR camera and said second IR camera are the same IR camera.

### Claim 12 (depends on 11)

The interaction system according to claim 11, wherein said space positioning device further includes: a radar detector and/or an ultrasonic detector, wherein:
said radar detector is used for detecting, with radar, a radar signal reflected by the joint positions of a touch control part of the operator; and
said ultrasonic detector is used for detecting, with ultrasonic waves, an ultrasonic signal reflected by the joint positions of the touch control part of the operator.

### Claim 13

A method for operating an interaction system of three-dimensional space, comprising:
determining, by an eye positioning device, an eye position of an operator and an angle at which the eye is watching and outputting the eye position and the angle to a master control device;
determining, by a space positioning device, respective joint positions of the operator and outputting the respective joint positions to said master control device;
determining, by a visual field space processor in said master control device, visual field space according to the eye position of the operator and the angle at which the eye is watching;
determining, by an interaction space processor in said master control device, interaction space according to the respective joint positions of the operator and using an area in said interaction space overlapping with said visual field space as absolute interaction space and the other areas in said interaction space as relative interaction space;
determining, by an operating action processor in said master control device, a current action of the operator according to changes of the respective joint positions of the operator;
determining, by an interaction matching processor in said master control device, an operation that matches the current action and needs to be performed on a virtual object selected by the operator according to a relationship among the current action of the operator, the virtual object viewed by the operator and the absolute interaction space; and
generating, by an image processor in said master control device, a corresponding three-dimensional image display signal according to the visual field space and the operation that matches the current action and needs to be performed on the virtual object selected by the operator and outputting it to an electrically connected three-dimensional image display device.

### Claim 14

An interaction control device of three-dimensional space, comprising: a visual field space processor, an interaction space processor, an operating action processor, an interaction matching processor and an image processor;
said visual field space processor being electrically connected to an eye positioning device, for determining visual field space according to the eye position of the operator and the angle at which the eye is watching, which are received from said eye positioning device;
said interaction space processor being electrically connected to a space positioning device and said visual field space processor, respectively, for determining interaction space according to the respective joint positions of the operator which are received from said space positioning device and using an area in said interaction space overlapping with said visual field space as absolute interaction space and the other areas in said interaction space as relative interaction space;
said operating action processor being electrically connected to said space positioning device, for determining a current action of the operator according to changes of the respective joint
said interaction matching processor being electrically connected to said visual field space processor, said interaction space processor and said operating action processor, respectively, for determining an operation that matches the current action and needs to be performed on a selected virtual object by the operator according to a relationship among the current action of the operator, the virtual object viewed by the operator and the absolute interaction space;
said image processor being electrically connected to said interaction matching processor and said visual field space processor, respectively, for generating a corresponding three-dimensional image display signal according to the visual field space and the operation that matches the current action and needs to be performed on the selected virtual object by the operator and outputting the corresponding three-dimensional image display signal to an electrically connected three-dimensional image display device.

### Claim 15 (depends on 14)

The interaction control device according to claim 14, wherein said operating action processor is further used for determining a current touch control action of the operator according to a change of the joint position of a touch control part of the operator; and determining a current body action of the operator according to a change of the joint position of a non-touch control part of the operator.

### Claim 16 (depends on 15)

The interaction control device according to claim 15, wherein said image processor is further used for determining a position of a three-dimensional image of said virtual object in said three-dimensional image display device according to a change of a visual angle at which the operator watches the same virtual object when the eye position of the operator changes, and for generating a corresponding three-dimensional image display signal and outputting it to the electrically connected three-dimensional image display device, wherein a positional relationship between the same virtual object and said three-dimensional image display device is relatively fixed.

### Claim 17 (depends on 14)

The interaction control device according to claim 14, wherein said interaction matching processor is further used for determining that the operator performs an operation of moving the virtual object positioned in said visual field space outside the interaction space into said absolute interaction space when it is determined that the current touch control action of the operator is within the relative interaction space.

### Claim 18 (depends on 17)

The interaction control device according to claim 17, wherein said visual field space processor is further used for determining watching space within said visual field space according to an angle at which the eye of the operator is watching;
said interaction matching processor is further used for determining that the operator performs an operation of moving the virtual object positioned in said watching space outside the interaction space into said absolute interaction space when it is determined that the current touch control action of the operator is within the relative interaction space.

### Claim 19 (depends on 18)

The interaction control device according to claim 18, wherein said interaction space processor is further used for using an area in the absolute interaction space overlapping with the watching space as refined interaction space;
said interaction matching processor is further used for determining that the operator performs a corresponding operation on the selected virtual object positioned in said absolute interaction space when it is determined that the current touch control action of the operator is within the absolute interaction space.

### Claim 20 (depends on 19)

The interaction control device according to claim 19, wherein said interaction matching processor is further used for determining that the operator performs a corresponding operation on the selected virtual object positioned in said refined interaction space when it is determined that the current touch control action of the operator is within the absolute interaction space.
