# US 2016/0259406 A1 · Interactive projection display

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0259406 A1
- Document ID: US20160259406A1
- Status: Published application
- Kind: A1
- Application number: 15/028,019
- Filed: 2014-10-09
- Published: 2016-09-08
- Earliest claimed: 2013-10-10
- Assignees: Beijing Zhigu Rui Tuo Tech Co., Ltd
- Applicants: Beijing Zhigu Rui Tuo Tech Co., Ltd
- Inventors: Lin Du
- Web page: https://patentgrep.com/patent/US20160259406A1

## CPC

- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G02B 3/14: of variable focal length
- G02B 7/287: {including a sight line detecting device}
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/017: {Head mounted}
- G02B 27/0179: {Display position adjusting means not related to the information to be displayed}
- G02B 27/10: Beam splitting or combining systems (polarising systems G02B27/28; mixing and splitting light signals using optical waveguides G02B6/28; {beam shaping, e.g. changing the cross-sectional area, by dividing or superposing multiple light beams G02B27/0905})
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 9/3185: {Geometric adjustment, e.g. keystone or convergence (optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14)}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Disclosed are an interactive projection display method and an interactive projection display system. The method comprises: obtaining to-be-displayed location information corresponding to a coordination device; obtaining display information corres-ponding to the coordination device; generating, according to the display information, virtual display content corresponding to the coordination device; and projecting the virtual display content to a location corresponding to the to-be-displayed location information at a fundus of a user. According to the method and the system, virtual display content, used to interact with a user, of a coordination device is projected near the coordination device, to cause interaction between the user and the coordination device to be more convenient.

## Claims

### Claim 1

A method, comprising:
obtaining, by a device comprising a processor, to-be-displayed location information corresponding to a coordination device;
obtaining display information corresponding to the coordination device;
generating, according to the display information, virtual display content corresponding to the coordination device; and
projecting the virtual display content to a location corresponding to the to-be-displayed location information at a fundus of a user.

### Claim 2 (depends on 1)

The method of claim 1, wherein the obtaining the to-be-displayed location information comprises:
obtaining device location information of the coordination device; and
determining the to-be-displayed location information according to the device location information of the coordination device.

### Claim 3 (depends on 2)

The method of claim 2, wherein the obtaining the device location information comprises:
receiving the device location information from the coordination device.

### Claim 4 (depends on 2)

The method of claim 2, wherein the obtaining the device location information comprises:
receiving the device location information from an external device.

### Claim 5 (depends on 2)

The method of claim 2, wherein the obtaining the device location information comprises:
detecting the device location information.

### Claim 6 (depends on 5)

The method of claim 5, wherein the detecting the device location information comprises:
detecting a first location of a gaze point of the user; and
determining the device location information according to the first location of the gaze point.

### Claim 7 (depends on 6)

The method of claim 6, wherein the detecting the first location of the gaze point of the user comprises:
acquiring at least one image at the fundus of an eye of the user at a second location;
adjusting, to acquire an image of the at least one image that satisfies at least one defined clarity criterion, at least one imaging parameter of an optical path between the second location to acquire the at least one image and the eye; and
analyzing the at least one image to obtain the at least one imaging parameter and at least one optical parameter of the eye corresponding to the image that satisfies the at least one defined clarity criterion, and acquiring the first location of the gaze point of the user relative to the user.

### Claim 8 (depends on 7)

The method of claim 7, wherein the adjusting the at least one parameter comprises:
adjusting a focal length of at least one optical device on the optical path or a third location of the at least one optical device on the optical path.

### Claim 9 (depends on 7)

The method of claim 7, wherein the adjusting the at least one parameter further comprises:
transferring, separately corresponding to different locations of a pupil of the eye in response to different optical axis directions of the eye, the at least one image presented at the fundus to the second location.

### Claim 10 (depends on 7)

The method of claim 7, wherein the detecting the first location of the gaze point of the user further comprises:
projecting a light spot pattern to the fundus of the eye.

### Claim 11 (depends on 5)

The method of claim 5, wherein the detecting the device location information comprises:
detecting interaction information of the user; and
determining the device location information according to the interaction information.

### Claim 12 (depends on 2)

The method of claim 2, wherein the determining the to-be-displayed location information comprises:
determining the to-be-displayed location information according to the device location information and a preset rule.

### Claim 13 (depends on 1)

The method of claim 1, wherein the obtaining the to-be-displayed location information comprises:
obtaining the to-be-displayed location information through interaction with the user.

### Claim 14 (depends on 1)

The method of claim 1, wherein the obtaining the display information comprises:
obtaining the display information from the coordination device.

### Claim 15 (depends on 1)

The method of claim 1, wherein the obtaining the display information comprises:
obtaining the display information from an external device.

### Claim 16 (depends on 1)

The method of claim 1, wherein the generating the virtual display content comprises:
generating the virtual display content according to the to-be-displayed location information and the display information.

### Claim 17 (depends on 16)

The method of claim 16, wherein the virtual display content comprises perspective information corresponding to the to-be-displayed location information.

### Claim 18 (depends on 16)

The method of claim 16, wherein the generating the virtual display content comprises:
generating the virtual display content separately corresponding to two eyes of the user.

### Claim 19 (depends on 1)

The method of claim 1, wherein the projecting the virtual display content comprises:
projecting the virtual display content to the fundus; and
adjusting at least one projection imaging parameter of an optical path between a projection location and the fundus according to the to-be-displayed location information.

### Claim 20 (depends on 19)

The method of claim 19, wherein the adjusting the at least one projection imaging parameter comprises:
adjusting at least one imaging parameter of at least one other optical device on the another optical path or another location of the at least one another optical device on the another optical path.

### Claim 21 (depends on 19)

The method of claim 19, wherein the projecting the virtual display content further comprises:
projecting, separately corresponding to different locations of a pupil in response to different optical axis directions of an eye related to the fundus, the virtual display content to the fundus.

### Claim 22 (depends on 21)

The method of claim 21, wherein the projecting the virtual display content comprises:
performing, on the virtual display content, counter-deformation processing corresponding to the different locations of the pupil.

### Claim 23 (depends on 19)

The method of claim 19, wherein the projecting the virtual display content comprises:
controlling a size of the virtual display content projected to the fundus.

### Claim 24 (depends on 1)

The method of claim 1, further comprising:
obtaining a trigger signal and triggering the obtaining of the to-be-displayed location information and the obtaining of the display information.

### Claim 25 (depends on 1)

The method of claim 1, wherein the coordination device is a wearable device.

### Claim 26

A system, comprising:
a memory that stores executable components; and
a processor, coupled to the memory, that executes the executable components to perform operations of the system, the executable components comprising:
a location information obtaining apparatus configured to obtain to-be-displayed location information corresponding to a coordination device;
a display information obtaining apparatus configured to obtain display information corresponding to the coordination device;
a content generating apparatus configured to generate, according to the display information, virtual display content corresponding to the coordination device; and
a projection apparatus configured to project the virtual display content to a location corresponding to the to-be-displayed location information at a fundus of a user.

### Claim 27 (depends on 26)

The system of claim 26, wherein the location information obtaining apparatus comprises:
a device location obtaining module configured to obtain device location information of the coordination device; and
a to-be-displayed location determining module configured to determine the to-be-displayed location information according to the device location information.

### Claim 28 (depends on 27)

The system of claim 27, wherein the device location obtaining module comprises:
a communications submodule configured to receive the device location information.

### Claim 29 (depends on 28)

The system of claim 28, wherein the communications submodule is further configured to:
receive the device location information from the coordination device.

### Claim 30 (depends on 28)

The system of claim 28, wherein the communications submodule is further configured to:
receive the device location information from an external device.

### Claim 31 (depends on 27)

The system of claim 27, wherein the device location obtaining module comprises:
a detection submodule configured to detect the device location information.

### Claim 32 (depends on 31)

The system of claim 31, wherein the detection submodule comprises:
a gaze point detection unit configured to detect a first location of a gaze point of the user; and
a device location determining unit configured to determine the device location information according to the first location of the gaze point.

### Claim 33 (depends on 32)

The system of claim 32, wherein the gaze point detection unit comprises:
an image acquisition subunit configured to acquire at least one image at the fundus of an eye of the user at a second location;
an adjustable imaging subunit configured to adjust at least one imaging parameter of an optical path between the image acquisition subunit and the eye to acquire an image of the at least one image that satisfies at least one defined clarity criterion; and
an image processing subunit configured to analyze the at least one image to obtain the at least one imaging parameter and at least one optical parameter of the eye corresponding to the image that satisfies the at least one defined clarity criterion, and acquire the first location of the gaze point of the user relative to the user.

### Claim 34 (depends on 33)

The system of claim 33, wherein the adjustable imaging subunit comprises:
an adjustable lens device, wherein at least one imaging parameter of the adjustable lens device being adjustable or a third location of the adjustable lens device on the optical path being adjustable.

### Claim 35 (depends on 33)

The system of claim 33, wherein the adjustable imaging subunit further comprises:
a curved beam splitting device configured to transfer, separately corresponding to different locations of a pupil of the eye in response to different optical axis directions of the eye, the at least one image presented at the fundus to the image acquisition subunit.

### Claim 36 (depends on 33)

The system of claim 33, wherein the gaze point detection unit further comprises:
a projection subunit configured to project a light spot pattern to the fundus of the eye.

### Claim 37 (depends on 31)

The system of claim 31, wherein the detection submodule further comprises:
an interaction detection unit configured to detect interaction information of the user; and
a device location determining unit configured to determine the device location information according to the interaction information of the user.

### Claim 38 (depends on 27)

The system of claim 27, wherein the to-be-displayed location determining module is further configured to determine the to-be-displayed location information according to the device location information and a preset rule.

### Claim 39 (depends on 26)

The system of claim 26, wherein the device location information obtaining apparatus comprises:
an interaction obtaining module configured to obtain the to-be-displayed location information through interaction with the user.

### Claim 40 (depends on 26)

The system of claim 26, wherein the display information obtaining apparatus comprises:
a communications module configured to obtain the display information corresponding to the coordination device from the coordination device.

### Claim 41 (depends on 26)

The system of claim 26, wherein the display information obtaining apparatus comprises:
a communications module configured to obtain the display information corresponding to the coordination device from an external device.

### Claim 42 (depends on 26)

The system of claim 26, wherein the content generating apparatus is further configured to generate the virtual display content according to the to-be-displayed location information and the display information.

### Claim 43 (depends on 42)

The system of claim 42, wherein the content generating apparatus is further configured to generate the virtual display content comprising perspective information corresponding to the to-be-displayed location information.

### Claim 44 (depends on 42)

The system of claim 42, wherein the content generating apparatus is further configured to generate the virtual display content separately corresponding to two eyes of the user.

### Claim 45 (depends on 26)

The system of claim 26, wherein the projection apparatus comprises:
a projection module configured to project the virtual display content to the fundus; and
an imaging adjustment module configured to adjust at least one projection imaging parameter of an optical path between the projection module and the fundus according to the to-be-displayed location information.

### Claim 46 (depends on 45)

The system of claim 45, wherein the imaging adjustment module comprises:
an adjustable lens device, wherein at least one imaging parameter of the adjustable lens device is adjustable or another location of the adjustable lens device on the optical path being adjustable.

### Claim 47 (depends on 45)

The system of claim 45, wherein the imaging adjustment module comprises:
a curved beam splitting unit configured to project, separately corresponding to different locations of a pupil in response to different optical axis directions of an eye related to the fundus, the virtual display content to the fundus.

### Claim 48 (depends on 47)

The system of claim 47, wherein the projection apparatus comprises:
a counter-deformation processing module configured to perform, on the virtual display content, counter-deformation processing corresponding to the different locations of the pupil.

### Claim 49 (depends on 45)

The system of claim 45, wherein the projection apparatus comprises:
a projection control module configured to control a size of the virtual display content projected to the fundus.

### Claim 50 (depends on 26)

The system of claim 26, wherein the executable components further comprise:
a triggering apparatus configured to obtain a trigger signal and trigger the location obtaining apparatus and the display information obtaining apparatus.

### Claim 51 (depends on 33)

The system of claim 33, wherein functions of the gaze point detection unit and the projection apparatus are implemented by a same device.

### Claim 52 (depends on 26)

The system of claim 26, wherein the coordination device is a wearable device.

### Claim 53 (depends on 26)

The system of claim 26, wherein the system is a wearable device used near an eye of the user.

### Claim 54

A computer readable storage device, comprising at least one executable instruction, which, in response to execution, causes a system comprising a processor to perform operations, comprising:
receiving to-be-displayed location information corresponding to a coordination device and display information corresponding to the coordination device;
generating, according to the display information, virtual display content corresponding to the coordination device; and
projecting the virtual display content to a location corresponding to the to-be-displayed location information at a fundus of a user.
