# US 2019/0361236 A1 · Imaging Method for Modular MR Device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0361236 A1
- Document ID: US20190361236A1
- Status: Published application
- Kind: A1
- Application number: 16/477,527
- Filed: 2018-06-01
- Published: 2019-11-28
- Earliest claimed: 2017-06-02
- Assignees: Holo Interactive US, Inc.
- Applicants: Fuzhou Lightflow Technology Co., Ltd.
- Inventors: Botao Hu
- Web page: https://patentgrep.com/patent/US20190361236A1

## CPC

- G02B 27/017: {Head mounted}
- A61B 5/6803: {Head-worn items, e.g. helmets, masks, headphones or goggles}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- H04M 1/724097: {Worn on the head}
- H04M 1/72439: for image or video messaging
- A61B 2503/12: Healthy persons not otherwise provided for, e.g. subjects of a marketing survey
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06F 3/012: {Head tracking input arrangements}
- H04M 2250/12: including a sensor for measuring a physical value, e.g. temperature or motion
- H04M 2250/52: including functional features of a camera
- Y02D 30/70: in wireless communication networks
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)
- Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE

## Abstract

An imaging method is provided for a modular mixed reality (MR) device having an MR calculation module, an MR optical path module and an MR posture module. The MR calculation module is configured to adjust display content according to data from the MR posture module. The MR optical path module comprises a virtual-image optical path and a mixed optical path. A semi-transparent semi-reflective mirror is provided in the mixed optical path. One surface of the mirror is a real-image introduction surface facing a real environment, while the other is a virtual-image introduction surface facing the virtual-image optical path. Virtual-image light is reflected by the virtual-image introduction surface onto an observation end and mixed with real environment light transmitted to the observation end by the real-image introduction surface to form a mixed reality image.

## Claims

### Claim 1

An imaging method for modular mixed reality (MR) device, wherein the MR device comprising:
an MR calculation module, an MR optical path module, and an MR posture module;
the MR calculation module comprises a display assembly;
the MR posture module comprises an imaging assembly and an IMU assembly, and the imaging assembly is configured to collect images in a preset angular direction of the display assembly, and the IMU assembly is configured to collect posture data of the MR device;
the MR calculation module is connected to the MR posture module, adjusting display content of the display assembly according to the image data and posture data acquired by the MR posture module; the MR optical path module comprises a virtual-image optical path and a mixed optical path, the virtual-image optical path being connected to the display assembly; the input end of the mixed optical path is connected to the virtual-image optical path, while the output end of the mixed optical path is an observation end;
a semi-transparent semi-reflective mirror is configured in the mixed optical path; one surface of the semi-transparent semi-reflective mirror is configured as a real-image introduction surface and the other surface of the mirror is configured as a virtual-image introduction surface; the real-image introduction surface faces a real environment, while the virtual-image introduction surface faces the virtual-image optical path;
display content of the display assembly is processed and transmitted by means of the virtual-image optical path to form a virtual image;
virtual-image light is reflected by means of the virtual-image introduction surface onto the observation end, and real environment light is transmitted to the observation end by means of the real-image introduction surface, being mixed with the virtual image to form a mixed reality image.

### Claim 2 (depends on 1)

The imaging method for modular mixed reality (MR) device as claimed in claim 1, wherein:
the MR calculation module is a main smart phone;
the display assembly is a display module of the main smart phone;
the IMU assembly comprises a magnetometer, a gyroscope and an accelerometer; the IMU assembly has a main IMU assembly and an auxiliary IMU assembly; the main IMU assembly collects the posture data of the display assembly; the main IMU assembly is arranged in the main smart phone; the auxiliary IMU assembly is arranged in one or more control devices wirelessly connected to the main smart phone; the auxiliary IMU assembly collects the posture data or position data of the one or more control devices; the posture data includes posture angle, angular rate or acceleration data;
the imaging assembly comprises a main imaging assembly and an auxiliary imaging assembly; the main imaging assembly is a rear camera of the main smart phone, and the auxiliary imaging assembly is a camera arranged at the control device.

### Claim 3 (depends on 2)

The imaging method for modular mixed reality (MR) device as claimed in claim 2, wherein:
the MR optical path module is a passive MR head-mounted mechanism;
the main smart phone is fixed at the MR head-mounted mechanism;
the main imaging assembly is the rear camera of the main smart phone; and
the control device is either a game console handle, a wearable device that can be worn on the hand or the foot, a sensor and a control device that are fixed at the MR head-mounted mechanism, or an auxiliary phone that is held by a user or is tied to the limbs of the user.

### Claim 4 (depends on 3)

The imaging method for modular mixed reality (MR) device as claimed in claim 3, wherein:
the virtual-image optical path of the MR head-mounted mechanism comprises a resting plate, a total reflection mirror and a field lens; the field lens is combined by two Fresnel lenses;
the main smart phone is placed horizontally on the resting plate; the main smart phone displays the VR split-screen mode image in the form of horizontal double-split screen when the MR head-mounted mechanism is in operation; the image light of double-split screen is reflected by means of the total reflection mirror onto the two Fresnel lenses, which then refract image light of double-split screen, so that image light forms two virtual-image light paths with preset field angle; virtual-image light is reflected by means of the virtual-image introduction surface onto an observation end, and real environment light is transmitted to the observation end by means of the real-image introduction surface, being mixed with the virtual image light to form a mixed reality image.

### Claim 5 (depends on 4)

The imaging method for modular mixed reality (MR) device as claimed in claim 4, wherein:
the orientation of the rear camera of the main smart phone is the orientation of the MR head-mounted mechanism;
the posture data of the display assembly is the posture data of the main smart phone;
the IMU assembly at the main smart phone collects the posture data of the main smart phone;
when the MR head-mounted mechanism is in operation, the rear camera of the main smart phone collects feature points of the real scene at the initial orientation of the MR head-mounted mechanism, and successively collects images as posture images while the MR head-mounted mechanism is in operation; the MR calculation module adjusts the images on the double-split screen according to the variation of the feature points at the posture image and the variation of the posture data of the main smart phone.

### Claim 6 (depends on 5)

The imaging method for modular mixed reality (MR) device as claimed in claim 5, wherein:
the image displayed in the form of horizontal double-split screen comprises a virtual character and a control identifier; the MR calculation module generates the control identifier according to the posture data and the position data of the control device uploaded by the auxiliary IMU assembly; the control identifier moves with the movement of the control device; the virtual character can interact with the control identifier.

### Claim 7 (depends on 6)

The imaging method for modular mixed reality (MR) device as claimed in claim 6, wherein:
the main smart phone is connected to an external device by network;
the virtual character and the control identifier included in the image that is displayed in the form of horizontal double-split screen is a part of the mixed reality image; the virtual character corresponds to the external device, and when the virtual character interacts with the control identifier, the external device performs respective implementation according to the interaction content.

### Claim 8 (depends on 7)

The imaging method for modular mixed reality (MR) device as claimed in claim 7, wherein the imaging method sequentially includes the steps of:
A1, the user fixing the main smart phone preinstalled with the MR application to the resting plate of the MR head-mounted mechanism and holding the auxiliary phone which is also a smart phone preloaded the MR application;
A2, the user wearing the MR head-mounted mechanism, and bringing the eyes close to the observation end so as to observe the mixed reality image;
A3, activating the MR application of the main smart phone and setting it as a display end; the main smart phone displaying the image in the form of transverse double-split screen, image light of double-split screen is reflected by means of the total reflection mirror onto the two Fresnel lenses, two Fresnel lenses refract image light of double-split screen, so that image light forms two virtual-image light paths with preset field angle; virtual-image light is reflected by means of the virtual-image introduction surface onto an observation end, and real environment light transmitted to the observation end by means of the real-image introduction surface, being mixed with the virtual image light to form a mixed reality image;
A4, the rear camera of the main smart phone collecting feature points of the real scene at the initial orientation of the MR head-mounted mechanism, and successively collecting images as posture images while the MR head-mounted mechanism is in operation; the MR calculation module adjusting the images on the double-split screen according to the variation of the feature points at the posture image and the variation of the posture data of the main smart phone;
A5, the user lifting the auxiliary mobile phone to a specific point of the mixed reality image, and activating the MR application on the auxiliary phone and setting it as a control end; the auxiliary IMU assembly on the auxiliary phone collecting the posture data and the position data of the auxiliary phone; the control end uploading the posture data and position data of the auxiliary phone to the display end which is connected to the control end by wirelessly manners;
A6, the MR calculation module generating the control identifier on the mixed reality image according to the posture data and position data of the auxiliary phone, wherein the control identifier moves with the movement of the auxiliary phone; when the control identifier on the mixed reality image is in contact with or adjacent to the virtual character, the virtual character interacts with the control identifier;
A7, the virtual character corresponds to the external device, and when the virtual character is interacting with the control identifier, the external device performs corresponding implementation according to the interaction content.

### Claim 9 (depends on 8)

The imaging method for modular mixed reality (MR) device as claimed in claim 8, wherein the main smart phone and the auxiliary phone generate and share unified spatial locating data by monocular visual inertial odometer method, the monocular visual inertial odometer method comprising the steps of:
B1, the main smart phone and the auxiliary phone collecting images by means of cameras to generate posture image respectively; the main smart phone and the auxiliary phone collecting posture data by means of built-in IMU assembly respectively; the main smart phone and the auxiliary phone associate the posture image with the posture data respectively, forming a respective spatial image association data; the main smart phone and the auxiliary phone aggregate respective spatial image related data through network connection to generate a unified spatial image related database in the main smart phone and the auxiliary phone;
B2, the main smart phone and the auxiliary phone successively collecting posture images and posture data during the movement, and adding the collected posture images and posture data to the spatial image related database respectively to associate;
B3, during the movement, the main smart phone and the auxiliary phone comparing the collected posture images and the posture data with that data in the spatial image related database to obtain the exact location of the phone in the current space and predict the trajectory and posture change of the phone;
B4, the main smart phone and the auxiliary phone reading the spatial image related database during the movement, and comparing the collected posture images with the posture images and the posture data in the past N time frames collected from the same coordinate and same posture, updating the spatial image related database when there is difference between the collected posture images and the posture images and the posture data in the past N time frames;
B5, in steps B3 and B4, the main smart phone and the auxiliary phone comparing and verifying the data with a preset tolerance threshold to improve the efficiency and robustness of the spatial location.

### Claim 10 (depends on 5)

The imaging method for modular mixed reality (MR) device as claimed in claim 5, wherein the MR head-mounted mechanism is made up of a sheet, which is provided with an A-folding section, a B-folding section and a C-folding section along the length; the A-folding section is fixed with the semi-transparent and semi-reflective mirror and the field lens; the B-folding section is fixed with the total reflective mirror; the C-folding section is provided with resting plate; the resting plate is provided with a observation hole for collecting external images by the rear camera of the main smart phone; the method for preparing the MR head-mounted mechanism comprises the steps of:
B1, folding the A-folding section and the B-folding section to a rhombic column, so that the lens is located at the connection line of the rhombic vertex; one side surface of the rhombic column is open and is a incident surface of image light, and the other three side surfaces are closed and form a observation hole wall, a semi-transparent semi-reflective mirror wall and a total reflective mirror wall respectively; the incident surface of image light faces the total reflective mirror wall; the total reflective mirror wall is provided with the total reflective mirror; the observation hole locates at the observation hole wall; the side wall of the rhombic column facing the observation hole is the semi-transparent semi-reflective mirror; the semi-transparent semi-reflective mirror locates at the semi-transparent semi-reflective mirror wall;
B2, expanding a light shield at the A-folding section and inserting the light shield into the observation hole wall;
B3, expanding the C-folding section and putting the main smart phone having rear camera on the resting plate, so that the rear camera aligns to the observation hole of the resting plate; folding the C-folding section then to the incident surface of light image of the rhombic column; the observation end includes the observation hole, at which the mixed reality image formed by mixing the screen image of the phone with the external image can be seen when the main smart phone displays the VR split screen mode image in the form of transverse double-split screen.

### Claim 11 (depends on 1)

The imaging method for modular mixed reality (MR) device as claimed in claim 1, wherein the MR calculation module is a main smart phone; the display assembly is a display module of the main smart phone; the IMU assembly comprises a magnetometer, a gyroscope and an accelerometer; the IMU assembly has a main IMU assembly and zero or more auxiliary IMU assemblies; the main IMU assembly collects the posture data of the display assembly and the main IMU assembly is arranged in the main smart phone; the auxiliary IMU assembly is arranged in one or more control devices wirelessly connected to the main smart phone; the auxiliary IMU assembly collects the posture data or position data of the one or more control devices; the posture data includes posture angle, angular rate or acceleration data; the imaging assembly comprises a main imaging assembly and a auxiliary imaging assembly; the main imaging assembly is a rear camera of the main smart phone, and the auxiliary imaging assembly is a camera at the control device and is optional.

### Claim 12 (depends on 11)

The imaging method for modular mixed reality (MR) device as claimed in claim 11, wherein the MR optical path module is a passive MR head-mounted mechanism; the main smart phone is fixed at the MR head-mounted mechanism; the main imaging assembly is the rear camera of the main smart phone; the control device is either a game console handle, a wearable device that can be worn on the hand or the foot, a sensor and a control device that are fixed at the MR head-mounted mechanism, or an auxiliary phone that is held by the user or is tied to the limbs of the user.

### Claim 13 (depends on 12)

The imaging method for modular mixed reality (MR) device as claimed in claim 12, wherein the virtual-image optical path of the MR head-mounted mechanism comprises a resting plate, a total reflection mirror and a field lens; the field lens is combined by two Fresnel lenses; the main smart phone is placed horizontally on the resting plate; the main smart phone displays the VR split-screen mode image in the form of double-split screen in the transverse direction when the MR head-mounted mechanism is in operation; image light of double-split screen is reflected by means of the total reflection mirror onto the two Fresnel lenses, which then refract image light of double-split screen, so that image light forms two virtual-image light paths with preset field angle; virtual-image light is reflected by means of the virtual-image introduction surface onto an observation end, and real environment light is transmitted to the observation end by means of the real-image introduction surface, being mixed with the virtual image light to form a mixed reality image.

### Claim 14 (depends on 13)

The imaging method for modular mixed reality (MR) device as claimed in claim 13, wherein the orientation of the rear camera of the main smart phone is the orientation of the MR head-mounted mechanism; the posture data of the display assembly is the posture data of the main smart phone; the IMU assembly at the main smart phone collects the posture data of the main smart phone; when the MR head-mounted mechanism is in operation, the rear camera of the main smart phone collects feature points of the real scene at the initial orientation of the MR head-mounted mechanism, and successively collects images as posture images of feature points while the MR head-mounted mechanism is in operation; the MR calculation module calculates the space position of the main smart phone according to the variation of the feature points at the posture image and the variation of the posture data of the main smart phone, and adjusts the images on the double-split screen based on the space position.

### Claim 15 (depends on 14)

The imaging method for modular mixed reality (MR) device as claimed in claim 14, wherein the image displayed in the form of transverse double-split screen comprises a virtual character and a control identifier; the MR calculation module generates the space position of the control identifier according to the posture data and position data of the control device uploaded by the auxiliary IMU assembly or the auxiliary imaging assembly; the control identifier moves with the movement of the control device; the virtual character can interact with the control identifier.

### Claim 16 (depends on 15)

The imaging method for modular mixed reality (MR) device as claimed in claim 15, wherein the main smart phone is connected to an external device by network; the virtual character and the control identifier included in the image that is displayed in the form of horizontal double-split screen is a part of the mixed reality image; the virtual character corresponds to the external device, and when the virtual character interacts with the control identifier, the external device performs respective implementation according to the interaction content.

### Claim 17 (depends on 16)

The imaging method for modular mixed reality (MR) device as claimed in claim 16, wherein the imaging method sequentially includes the steps of:
A1, the user fixing the main smart phone preinstalled with the MR application to the resting plate of the MR head-mounted mechanism and holding the auxiliary phone which is also a smart phone preloaded the MR application;
A2, the user wearing the MR head-mounted mechanism, and bringing the eyes close to the observation end so as to observe the mixed reality image;
A3, activating the MR application of the main smart phone and setting it as a display end; the main smart phone displaying the image in the form of horizontal double-split screen, image light of double-split screen is reflected by means of the total reflection mirror onto the two Fresnel lenses, two Fresnel lenses refract image light of double-split screen, so that image light forms two virtual-image light paths with preset field angle; virtual-image light is reflected by means of the virtual-image introduction surface onto an observation end, and real environment light transmitted to the observation end by means of the real-image introduction surface, being mixed with the virtual image light to form a mixed reality image;
A4, the rear camera of the main smart phone collecting feature points of the real scene at the initial orientation of the MR head-mounted mechanism, and successively collecting images as posture images of feature points while the MR head-mounted mechanism is in operation; the MR calculation module calculating the space position of the main smart phone according to the variation of the feature points at the posture image and the variation of the posture data of the main smart phone, and adjusting the images on the double-split screen based on the space position;
A5, the user lifting the control device to a specific point of the mixed reality image, if the control device is a smart phone, activating the MR application on the auxiliary phone and setting it as a control end; the auxiliary IMU assembly on the control device collecting the posture data and position data of the auxiliary phone; the control end uploading the posture data and position data of the control device to the display end which is connected to the control end wirelessly;
A6, the MR calculation module generating the control identifier on the mixed reality image according to the posture data and position data of the auxiliary phone, wherein the control identifier moves with the movement of the auxiliary phone; when the control identifier on the mixed reality image is in contact with or adjacent to the virtual character, the virtual character interacts with the control identifier;
A7, the virtual character corresponds to the external device, and when the virtual character is interacting with the control identifier, the external device performs corresponding implementation according to the interaction content.

### Claim 18 (depends on 14)

The imaging method for modular mixed reality (MR) device as claimed in claim 14, wherein the MR head-mounted mechanism is made up of a sheet, which is provided with an A-folding section, a B-folding section and a C-folding section along the length; the A-folding section is fixed with the semi-transparent and semi-reflective mirror and the field lens; the B-folding section is fixed with the total reflective mirror; the C-folding section is provided with resting plate; the resting plate is provided with a observation hole for collecting external images by the rear camera of the main smart phone.

### Claim 19 (depends on 18)

The imaging method for modular mixed reality (MR) device as claimed in claim 18, wherein the method for preparing the MR head-mounted mechanism comprises the steps of:
B1, folding the A-folding section and the B-folding section to a rhombic column, so that the lens is located at the connection line of the rhombic vertex; one side surface of the rhombic column is open and is a incident surface of image light, and the other three side surfaces are closed and form a observation hole wall, a semi-transparent semi-reflective mirror wall and a total reflective mirror wall respectively; the incident surface of image light faces the total reflective mirror wall; the total reflective mirror wall is provided with the total reflective mirror; the observation hole locates at the observation hole wall; the side wall of the rhombic column facing the observation hole is the semi-transparent semi-reflective mirror; the semi-transparent semi-reflective mirror locates at the semi-transparent semi-reflective mirror wall;
B2, expanding a light shield at the A-folding section and inserting the light shield into the observation hole wall;
B3, expanding the C-folding section and putting the main smart phone having rear camera on the resting plate, so that the rear camera aligns to the observation hole of the resting plate; folding the C-folding section then to the incident surface of light image of the rhombic column; the observation end includes the observation hole, at which the mixed reality image formed by mixing the screen image of the phone with the external image can be seen when the main smart phone displays the VR split screen mode image in the form of transverse double-split screen.
