# US 11,307,423 B2 · Image processing method for augmented reality device, augmented reality device and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,307,423 B2
- Document ID: US11307423B2
- Status: Granted
- Kind: B2
- Application number: 16/903,400
- Filed: 2020-06-17
- Published: 2020-12-31
- Granted: 2022-04-19
- Earliest claimed: 2019-06-28
- Assignees: Beijing Boe Optoelectronics Technology Co., Ltd.; Boe Technology Group Co., Ltd.
- Applicants: Beijing Boe Optoelectronics Technology Co., Ltd.; BOE Technology Group Co., Ltd.
- Inventors: Liyang Cui; Jiankang Sun; Lili Chen; Hao Zhang
- Web page: https://patentgrep.com/patent/US11307423B2

## CPC

- G02B 27/0176: {characterised by mechanical features}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G02B 27/0081: {with means for altering, e.g. enlarging, the entrance or exit pupil}
- G02B 27/0172: {characterised by optical features}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/579: from motion
- G02B 2027/0134: {of stereoscopic type}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06T 2207/20101: Interactive definition of point of interest, landmark or seed
- G06T 2207/20104: Interactive definition of region of interest [ROI]
- G06T 2207/30041: Eye; Retina; Ophthalmic
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An image processing method for an augmented reality device, an augmented reality device and a storage medium are provided. The image processing method for the augmented reality device includes: tracking lines of sight of a left eye and a right eye to determine a position of a binocular watching range in a three-dimensional space; shooting an image of an external environment where the augmented reality (AR) device is located, and acquiring environment map point cloud data based on a Simultaneous Localization and Mapping (SLAM) algorithm; and updating the environment map point cloud data within the binocular watching range in response to detecting an update of the image of the external environment.

## Claims

### Claim 1

An image processing method for an augmented reality device, comprising:
tracking lines of sight of a left eye and a right eye to determine a position of a binocular watching range in a three-dimensional space;
shooting an image of an external environment where the augmented reality (AR) device is located, and acquiring environment map point cloud data based on a Simultaneous Localization and Mapping (SLAM) algorithm; and
updating the environment map point cloud data within the binocular watching range in response to detecting an update of the image the external environment;
wherein the tracking the lines of sight of the left eye and the right eye to determine the position of the binocular watching range in the three-dimensional space comprises:
capturing an image of a left eye and an image of a right eye, determining a center position of a left pupil according to the image of the left eye, and determining a center position of a right pupil according to the image of the right eye;
inputting pupil center position data of both eyes into a sight watching model to determine a position of an intersection point of the lines of sight of both eyes on a virtual imaging plane; and
determining a midpoint of a line connecting a left exit pupil center and a right exit pupil center of a binocular optical imaging module of the AR device, constructing a sight cone by taking the line connecting the midpoint and the intersection point as a central axis, and taking a space within the sight cone as the binocular watching range; wherein a vertex of the sight cone is the midpoint.

### Claim 2 (depends on 1)

The image processing method for the augmented reality device according to claim 1, wherein,
the sight watching model is used to describe a mapping relationship between coordinates of the centers of the pupils of both eyes in a coordinate system and coordinates of watching points of the lines of sight of both eyes on left and right imaging planes of the binocular optical imaging module of the AR device in the coordinate system.

### Claim 3 (depends on 2)

The image processing method for the augmented reality device according to claim 2, wherein,
the inputting the pupil center position data of both eyes into the sight watching model to determine the position of the intersection point of the lines of sight of both eyes on the virtual imaging plane comprises:
inputting the pupil center position data of both eyes into the sight watching model, and calculating a coordinate of a left watching point on a left imaging plane and a coordinate of a right watching point on a right imaging plane of the binocular optical imaging module of the AR device; and
determining a coordinate of the intersection point of a left watching line and a right watching line on the virtual imaging plane; wherein the left watching line is determined by the left exit pupil center point and the left watching point on the left imaging plane, and the right watching line is determined by the right exit pupil center point and the right watching point on the right imaging plane.

### Claim 4 (depends on 2)

The image processing method for the augmented reality device according to claim 2, wherein, the sight watching model comprises a polynomial model, and the polynomial model is expressed by a following formula:
{
g
x
=
a
0
+
a
1
⁢
x
+
a
2
⁢
y
+
a
3
⁢
x
⁢
y
+
a
4
⁢
x
2
+
a
5
⁢
y
2
g
y
=
b
0
+
b
1
⁢
x
+
b
2
⁢
y
+
b
3
⁢
x
⁢
y
+
b
4
⁢
x
2
+
b
5
⁢
y
2
wherein (gx, gy) is the coordinate of a watching point on an imaging plane of the binocular optical imaging module of the AR device, (x,y) is the coordinate of an eye pupil center, and a0, a1, a2, a3, a4, a5,b0, b1, b2, b3, b4, b5 are parameters of the model.

### Claim 5 (depends on 2)

The image processing method for the augmented reality device according to claim 2, wherein, before tracking the lines of sight of the left and right eyes, the method further comprises:
selecting X watching points on virtual imaging planes of the binocular optical imaging module of the AR device, wherein a distance between a virtual imaging plane where an i-th watching point is located and an origin of coordinates in the coordinate system is Di, wherein i is a positive integer greater than or equal to 1 and less than or equal to X;
determining X groups of imaging coordinates of the X watching points on the left and right imaging planes of the binocular optical imaging module according to the coordinates of the X watching points and the left and right exit pupil center points of the X watching points on the binocular optical imaging module;
capturing human eye images when human eyes watch the X watching points, and acquiring X groups of coordinates of left and right pupil center points when human eyes watch the X watching points; and
inputting the X groups of imaging coordinates and the X groups of coordinates of left and right pupil center points into the sight watching model to acquire parameter values of the sight watching model.

### Claim 6 (depends on 1)

The image processing method for the augmented reality device according to claim 1, wherein, an initial value of an apex angle φ of the sight cone is greater than or equal to 15 degrees and less than or equal to 30 degrees.

### Claim 7 (depends on 1)

The image processing method for the augmented reality device according to claim 1, wherein,
the detecting an update of the image of the external environment comprises:
capturing the image of the external environment through a binocular camera in real time, performing spatial localization and environment mapping based on the SLAM algorithm; and
when a front-end localization thread of the SLAM algorithm outputs a new key frame for environment mapping to a back-end mapping thread, converting the new key frame into a three-dimensional point cloud, taking the three-dimensional point cloud as an input point cloud to match the binocular watching range, and taking the input point cloud located within the binocular watching range as an update point cloud.

### Claim 8 (depends on 1)

The image processing method for the augmented reality device according to claim 1, wherein,
the updating the environment map point cloud data within the binocular watching range comprises:
calculating an average distance d2 between the midpoint of the line connecting the center points of the left and right exit pupils and each point in an update point cloud;
calculating an average distance d3 between the midpoint of the line connecting the center points of the left and right exit pupils and each point in a target point cloud; wherein, the target point cloud is an initial point cloud falling within the binocular watching range, and the initial point cloud is a three-dimensional point cloud generated according to an old image of an environment map; and
comparing the sizes of d2 and d3, and when the sizes of d2 and d3 satisfy a condition for updating the environment map, replacing the target point cloud with the update point cloud.

### Claim 9 (depends on 8)

The image processing method for the augmented reality device according to claim 8, wherein, the condition for updating the environment map comprises: d3<d2·s, wherein s is a coefficient less than 1.

### Claim 10 (depends on 8)

The image processing method for the augmented reality device according to claim 8, wherein, after replacing the target point cloud with the update point cloud, the method further comprises:
updating the sight cone; and
recalculating the average distance d2 between the midpoint of the line connecting the center points of the left and right exit pupils and the update point cloud and the average distance d3 between the midpoint of the line connecting the center points of the left and right exit pupils and the target point cloud; comparing the sizes of d2 and d3, and when the sizes of d2 and d3 satisfy the condition for updating the environment map, replacing the target point cloud with the update point cloud.

### Claim 11 (depends on 10)

The image processing method for the augmented reality device according to claim 10, wherein, the updating the sight cone comprises: setting an apex angle φ of the sight cone to be n times of an initial value, wherein n is greater than 1 and less than or equal to 4.

### Claim 12

An augmented reality device, comprising:
an environment image capturing camera module, configured to capture an environment image;
a human eye image capturing camera module, configured to capture a human eye image;
a memory configured to store executable instructions; and
a processor, configured to implement following steps when executing the executable instructions stored in the memory:
tracking lines of sight of a left eye and a right eye to determine a position of a binocular watching range in a three-dimensional space;
acquiring an image of an external environment where the augmented reality (AR) device is located, and acquiring environment map point cloud data based on a Simultaneous Localization and Mapping (SLAM) algorithm; and
updating the environment map point cloud data within the binocular watching range in response to detecting an update of the image of the external environment;
wherein the tracking the lines of sight of the left eye and the right eye to determine the position of the binocular watching range in the three-dimensional space comprises:
capturing an image of a left eye and an image of a right eye, determining a center position of a left pupil according to the image of the left eye, and determining a center position of a right pupil according to the image of the right eye;
inputting pupil center position data of both eyes into a sight watching model to determine a position of an intersection point of the lines of sight of both eyes on a virtual imaging plane; and
determining a midpoint of a line connecting a left exit pupil center and a right exit pupil center of a binocular optical imaging module of the AR device, constructing a sight cone by taking the line connecting the midpoint and the intersection point as a central axis, and taking a space within the sight cone as the binocular watching range; wherein a vertex of the sight cone is the midpoint;
wherein the sight watching model is used to describe a mapping relationship between coordinates of the centers of the pupils of both eyes in a coordinate system and coordinates of watching points of the lines of sight of both eyes on left and right imaging planes of the binocular optical imaging module of the AR device in the coordinate system.

### Claim 13 (depends on 12)

The augmented reality device according to claim 12, wherein,
the inputting the pupil center position data of both eyes into the sight watching model to determine the position of the intersection point of the lines of sight of both eyes on the virtual imaging plane comprises:
inputting the pupil center position data of both eyes into the sight watching model, and calculating a coordinate of a left watching point on a left imaging plane and a coordinate of a right watching point on a right imaging plane of the binocular optical imaging module of the AR device; and
determining a coordinate of the intersection point of a left watching line and a right watching line on the virtual imaging plane; wherein the left watching line is determined by the left exit pupil center point and the left watching point on the left imaging plane, and the right watching line is determined by the right exit pupil center point and the right watching point on the right imaging plane.

### Claim 14 (depends on 13)

The augmented reality device according to claim 13, wherein,
the processor is configured to further implement following steps when executing the executable instructions stored in the memory:
selecting X watching points on virtual imaging plane of the binocular optical imaging module of the AR device, wherein a distance between a virtual imaging plane where an i-th watching point is located and an origin of coordinates in the coordinate system is Di, wherein i is a positive integer greater than or equal to 1 and less than or equal to X;
determining X groups of imaging coordinates of the X watching points on the left and right imaging planes of the binocular optical imaging module according to the coordinates of the X watching points and the left and right exit pupil center points of the X watching points on the binocular optical imaging module;
capturing human eye images when human eyes watch the X watching points, and acquiring X groups of coordinates of left and right pupil center points when human eyes watch the X watching points; and
inputting the X groups of imaging coordinates and the X groups of coordinates of left and right pupil center points into the sight watching model to acquire parameter values of the sight watching model.

### Claim 15 (depends on 12)

The augmented reality device according to claim 12, wherein,
the detecting an update of the image of the external environment comprises:
capturing the image of the external environment through a binocular camera in real time, performing spatial localization and environment mapping based on the SLAM algorithm; and
when a front-end localization thread of the SLAM algorithm outputs a new key frame for environment mapping to a back-end mapping thread, converting the new key frame into a three-dimensional point cloud, taking the three-dimensional point cloud as an input point cloud to match the binocular watching range, and taking the input point cloud located within the binocular watching range as an update point cloud.

### Claim 16 (depends on 12)

The augmented reality device according to claim 12, wherein,
the updating of the environment map point cloud data within the binocular watching range comprises:
calculating an average distance d2 between the midpoint of the line connecting the center points of the left and right exit pupils and each point in an update point cloud;
calculating an average distance d3 between the midpoint of the line connecting the center points of the left and right exit pupils and each point in a target point cloud; wherein, the target point cloud is an initial point cloud falling within the binocular watching range, and the initial point cloud is a three-dimensional point cloud generated according to an old image of an environment map; and
comparing the sizes of d2 and d3, and when the sizes of d2 and d3 satisfy a condition for updating the environment map, replacing the target point cloud with the update point cloud.

### Claim 17 (depends on 16)

The augmented reality device according to claim 16, wherein, the condition for updating the environment map comprises: d3<d2·s, wherein s is a coefficient less than 1.

### Claim 18

A non-transitory computer readable storage medium storing executable instructions, wherein when the executable instructions are executed by a processor, the following steps are implemented:
tracking lines of sight of a left eye and a right eye to determine a position of a binocular watching range in a three-dimensional space;
acquiring an image of an external environment where the augmented reality (AR) device is located, and acquiring environment map point cloud data based on a Simultaneous Localization and Mapping (SLAM) algorithm; and
updating the environment map point cloud data within the binocular watching range in response to detecting an update of the image of the external environment;
wherein the tracking the lines of sight of the left eye and the right eye to determine the position of the binocular watching range in the three-dimensional space comprises:
capturing an image of a left eye and an image of a right eye, determining a center position of a left pupil according to the image of the left eye, and determining a center position of a right pupil according to the image of the right eye;
inputting pupil center position data of both eyes into a sight watching model to determine a position of an intersection point of the lines of sight of both eyes on a virtual imaging plane; and
determining a midpoint of a line connecting a left exit pupil center and a right exit pupil center of a binocular optical imaging module of the AR device, constructing a sight cone by taking the line connecting the midpoint and the intersection point as a central axis, and taking a space within the sight cone as the binocular watching range; wherein a vertex of the sight cone is the midpoint.