# US 2018/0025520 A1 · Binocular see-through AR head-mounted display device and information displaying method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0025520 A1
- Document ID: US20180025520A1
- Status: Published application
- Kind: A1
- Application number: 15/544,963
- Filed: 2015-08-07
- Published: 2018-01-25
- Earliest claimed: 2015-01-21
- Assignees: Chengdu Idealsee Technology Co., Ltd.
- Applicants: Chengdu Idealsee Technology Co., Ltd
- Inventors: Qinhua Huang; Haitao Song; Xinyu Li
- Web page: https://patentgrep.com/patent/US20180025520A1

## CPC

- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/017: {Head mounted}
- G02B 27/0172: {characterised by optical features}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 20/20: in augmented reality scenes
- H04N 13/344: with head-mounted left-right displays
- H04N 13/383: for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
- G02B 2027/0127: {comprising devices increasing the depth of field}
- G02B 2027/0132: {comprising binocular systems}
- G02B 2027/014: {comprising information/image processing systems}
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Disclosed are a binocular see-through AR head-mounted device and an information displaying method thereof. Sight mapping relationship η is preset in the head-mounted device, human eye spatial sight information data of a user is tracked and calculated by a sight tracking system, virtual information that needs to be displayed is displayed on the left and right lines of sight of the human eyes on the basis of a binocular see-through AR head-mounted device virtual image imaging principle and a human eye binocular vision principle, thus implementing accurate overlap of the virtual information to the proximity of the position of the fixation point of the human eyes, allowing a high degree of integration of the virtual information with the environment, and implementing enhanced virtual reality in the true sense. The present invention provides a simple solution, requires only the sight tracking system to complete the process, obviates the need for excessive hardware facilities, and is inexpensive.

## Claims

### Claim at position 1

1: An information displaying method of a binocular see-through AR (Augmented Reality) head-mounted device, wherein a sight mapping relationship η is preset in the head-mounted device; η is the mapping relationship between line-of-sight information data in eye-centered coordinate and coordinates of a left and a right pixel pair of an image display source of the head-mounted device; the information displaying method comprises steps of:
capturing the line-of-sight information data in eye-centered coordinate when a user stares at a target object;
calculating the coordinates of the left and right pixel pair of the target object according to η and the line-of-sight information data in eye-centered coordinate of the target object; and
displaying a information source image of virtual information to be displayed onto an image display source on left and right respectively, wherein the coordinates of the left and right pixel pair of the target object is a reference position.

### Claim at position 2 (depends on 1)

2: The method, as recited in claim 1, wherein displaying the information source image of virtual information to be displayed onto the image display source on the left and the right respectively, wherein the coordinates of the left and the right pixel pair of the target object is a reference position, comprises steps of:
displaying the information source image of virtual information to be displayed onto the image display source on left and right respectively, wherein the coordinates of the left and right pixel pair of the target object is a center; or
displaying the information source image of virtual information to be displayed onto the image display source on left and right respectively, which is at a shifted position of the coordinates of the left and right pixel pair of the target object.

### Claim at position 3 (depends on 2)

3: The method, as recited in claim 2, wherein when the user initially uses the head-mounted device and/or while each use of the head-mounted device, η is modified.

### Claim at position 4 (depends on 3)

4: The method, as recited in claim 3, wherein modification method of η comprises steps of:
controlling the image display source to display each pixel pair of a left and a right modified bitmap in sequence to project a virtual bitmap corresponding to each pixel pair in front of the user eyes;
capturing the line-of-sight information data in eye-centered coordinate while the user stares at every virtual bitmap to calculate the line-of-sight information data in eye-centered coordinate corresponding to the virtual bitmap; and
modifying η by using the line-of-sight information data in eye-centered coordinate corresponding to all the virtual bitmap.

### Claim at position 5 (depends on 4)

5: The method, as recited in claim 4, wherein the left and the right modified bitmap is a bitmap with r rows and s columns, wherein r and s are positive integers which are greater than 1 and less than 10.

### Claim at position 6 (depends on 3)

6: The method, as recited in claim 3, wherein the modification of η comprises steps of:
controlling the image display source to display multiple sets of a left and a right modified image in sequence to project a virtual overlap image corresponding to every set of the modified image in front of the user eyes;
capturing line-of-sight information data in eye-centered coordinate while the user stares at every virtual overlap image to obtain the line-of-sight information data in eye-centered coordinate corresponding to the virtual overlap image; and
modifying η by using the line-of-sight information data in eye-centered coordinate corresponding to all of the virtual overlap images.

### Claim at position 7

-9. (canceled)

### Claim at position 8

10: A binocular see-through AR head-mounted display device comprises an image display source, a line-of-sight tracking system and a data processing module, wherein the data processing module stores a line-of-sight mapping relationship η; η is a mapping relationship between line-of-sight information data in eye-centered coordinate and coordinates of a left and a right pixel pair of the image display source on the head-mounted device, wherein
the line-of-sight tracking system is for capturing line-of-sight information data in eye-centered coordinate while a user stares at a target object;
the data processing module is for obtaining coordinates of the left and the right pixel pair of the target object according to η and the line-of-sight information data in eye-centered coordinate of the target object;
the image display source is for displaying an information source image of virtual information to be displayed on left and right respectively, wherein the coordinates of the left and the right pixel pair of the target object is a reference position.

### Claim at position 9 (depends on 10)

11: The head-mounted display device, as recited in claim 10, wherein the image display source is for:
displaying the information source image of the virtual information to be displayed on the left and the right of the image display source respectively, wherein the coordinates of the left and the right pixel pair of the target object as a center; or
displaying the information source image of the virtual information to be displayed on the left and the right of the image display source respectively, which is at a shifted position of the coordinates of the left and the right pixel pair of the target object.

### Claim at position 10 (depends on 11)

12: The head-mounted display device, as recited in claim 11, comprises an adapter module which is for modifying η when the user initially use the head-mounted device and/or while each use of the head-mounted device.

### Claim at position 11 (depends on 12)

13: The head-mounted display device, as recited in claim 12, wherein the adapter module is for:
controlling the image display source to display each pixel pair in a left and a right modified bitmap in sequence to project a virtual bitmap corresponding to each pixel pair in front of the user eyes;
controlling the line-of-sight tracking system to capture the line-of-sight information data in eye-centered coordinate when the user stares at each virtual bitmap to obtain the line-of-sight information data in eye-centered coordinate corresponding to the virtual bitmap;
controlling the data processing module to modify η by using the line-of-sight information data in eye-centered coordinate corresponding to all the virtual bitmaps.

### Claim at position 12 (depends on 13)

14: The head-mounted display device, as recited in claim 13, wherein the left and the right modified bitmap is a bitmap with r rows and s columns, wherein the r and s are positive integers which are greater than 1 and less than 10.

### Claim at position 13 (depends on 12)

15: The head-mounted display device, as recited in claim 12, wherein the adapter module is for:
controlling the image display source to display multiple sets of modified images on the left and the right in sequence to project a virtual overlapped image corresponding to each set of the modified image in front of the user eyes;
controlling the line-of-sight tracking system to capture the line-of-sight information data in eye-centered coordinate while the user stares at each virtual overlapped image to obtain line-of-sight information data in eye-centered coordinate corresponding to the virtual overlapped image;
controlling the data processing module to modify η by using the human space line-of-sight information data corresponding to all the virtual overlapped image.

### Claim 16

(canceled)

### Claim at position 15 (depends on 1)

17: The method, as recited in claim 1 wherein an offline constructing method of η comprises steps of:
capturing multiple sets of the left and the right bitmaps;
testing a test user in multiple test users by using all the left and right bitmaps in sequence to obtain test data of the test user; and
constructing η according to the test data of all the test users;
wherein, testing the test user by using all the left and the right bitmap to calculate the test data of the test user comprises steps of:
controlling the image display source to display each pixel pair of the left and the right bitmap in sequence for each set of the left and the right bitmap of the multiple sets of the left and the right bitmaps in sequence to project the virtual bitmap corresponding to the coordinates of the pixel pair in front of the test user eyes;
capturing the line-of-sight information data in eye-centered coordinate while the user stares at the virtual bitmap to obtain the line-of-sight information data in eye-centered coordinate corresponding to the pixel pair; and
determining the test data of the test user and building a mapping relationship between the test data comprising the coordinates of each pixel pair of all the left and the right bitmap and the corresponding line-of-sight information data in eye-centered coordinate.

### Claim at position 16 (depends on 2)

18: The method, as recited in claim 2, wherein an offline constructing method of η comprises steps of:
capturing multiple sets of the left and the right bitmaps;
testing a test user in multiple test users by using all the left and right bitmaps in sequence to obtain test data of the test user; and
constructing η according to the test data of all the test users;
wherein, testing the test user by using all the left and the right bitmap to calculate the test data of the test user comprises steps of:
controlling the image display source to display each pixel pair of the left and the right bitmap in sequence for each set of the left and the right bitmap of the multiple sets of the left and the right bitmaps in sequence to project the virtual bitmap corresponding to the coordinates of the pixel pair in front of the test user eyes;
capturing the line-of-sight information data in eye-centered coordinate while the user stares at the virtual bitmap to obtain the line-of-sight information data in eye-centered coordinate corresponding to the pixel pair; and
determining the test data of the test user and building a mapping relationship between the test data comprising the coordinates of each pixel pair of all the left and the right bitmap and the corresponding line-of-sight information data in eye-centered coordinate.

### Claim at position 17 (depends on 17)

19: The method, as recited in claim 17, wherein capturing the multiple sets of the left and the right bitmaps comprise steps of:
capturing an initial left and right bitmap; and
obtaining the multiple sets of the left and the right bitmaps according to the initial left and right bitmap and shorten a center to center distance of the calculated multiple sets of the left and the right bitmaps in sequence.

### Claim at position 18 (depends on 1)

20: The method, as recited in claim 1, wherein η fulfils:
(ELnm1,ERnm2)θ(SLnm1,SRnm2);
wherein (ELnm1,ERnm2) is the line-of-sight information data in eye-centered coordinate; (SLnm1,SRnm2) are the coordinates of the left and the right pixel pair of the image display source on the head-mounted device, wherein nε[1,N], m1ε[1,M] m2ε[1,M]; N and M are a vertical resolution and a horizontal resolution of a left and a right image display source of the head-mounted device respectively; ƒ is a fitting function.

### Claim at position 19 (depends on 2)

21: The method, as recited in claim 2, wherein η fulfils:
(ELnm1,ERnm2)ƒ(SLnm1,SRnm2);
wherein (ELnm1,ERnm2) is the line-of-sight information data in eye-centered coordinate; (SLnm1,SRnm2) are the coordinates of the left and the right pixel pair of the image display source on the head-mounted device, wherein nε[1,N], m1ε[1,M], m2ε[1,M]; N and M are a vertical resolution and a horizontal resolution of a left and a right image display source of the head-mounted device respectively; ƒ is a fitting function.

### Claim at position 20 (depends on 5)

22: The method, as recited in claim 5, wherein η fulfils:
(ELnm1,ERnm2)ƒ(SLnm1,SRnm2);
wherein (ELnm1,ERnm2) is the line-of-sight information data in eye-centered coordinate; (SLnm1,SRnm2) are the coordinates of the left and the right pixel pair of the image display source on the head-mounted device, wherein nε[1,N], m1ε[1,M], m2ε[1,M]; N and M are a vertical resolution and a horizontal resolution of a left and a right image display source of the head-mounted device respectively; ƒ is a fitting function.

### Claim at position 21 (depends on 6)

23: The method, as recited in claim 6, wherein η fulfils:
(ELnm1,ERnm2)ƒ(SLnm1,SRnm2);
wherein (ELnm1,ERnm2) is the line-of-sight information data in eye-centered coordinate; (SLnm1, SRnm2) are the coordinates of the left and the right pixel pair of the image display source on the head-mounted device, wherein nε[1,N], m1ε[1,M], m2ε[1,M]; N and M are a vertical resolution and a horizontal resolution of a left and a right image display source of the head-mounted device respectively; ƒ is a fitting function.

### Claim at position 22 (depends on 10)

24: The head-mounted device, as recited in claim 10, wherein η fulfills:
(ELnm1,ERnm2)ƒ(SLnm1,SRnm2);
wherein (ELnm1,ERnm2) is the line-of-sight information data in eye-centered coordinate; (SLnm1,SRnm2) are coordinates of the left and the right pixel pair of the image display source on the head-mounted device; wherein nε[1,N], m1ε[1,M], m2ε[1,M]; N and M are a vertical resolution and a horizontal resolution of a left and a right image display source on the head-mounted device respectively; ƒ is a fitting function.