# US 2023/0267692 A1 · Mixed reality processing system and mixed reality processing method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267692 A1
- Document ID: US20230267692A1
- Status: Published application
- Kind: A1
- Application number: 17/881,783
- Filed: 2022-08-05
- Published: 2023-08-24
- Earliest claimed: 2022-02-18
- Assignees: HTC Corporation
- Applicants: HTC Corporation
- Inventors: Shi-Hao DONG; Yi-Chen LIN
- Web page: https://patentgrep.com/patent/US20230267692A1

## CPC

- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G02B 27/0172: {characterised by optical features}
- G06T 7/11: Region-based segmentation
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06T 2207/30196: Human being; Person
- G06T 2207/30244: Camera pose
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A mixed reality processing method includes the following steps. A virtual scene is displayed by a head-mounted display. The virtual world coordinates of a physical object are obtained using a first SLAM map. A tracker uses a second SLAM map to obtain the real world coordinates of the physical object. The virtual world coordinates and the real world coordinates are merged. The virtual scene and the physical object are combined by a processor to generate a mixed reality image.

## Claims

### Claim 1

A mixed reality processing system, comprising:
a camera, configured to capture a two-dimensional image including a physical object;
a head-mounted display (HMD), configured to display a virtual scene, and obtain virtual world coordinates of the physical object in the virtual world through a first simultaneous localization and mapping map (SLAM map); wherein the virtual world coordinates are used to generate the virtual scene based on real world coordinates;
a tracker, configured to differentiate the physical object from the two-dimensional image, and obtain the real world coordinates of the physical object in the real world through a simultaneous localization and mapping map; and
a processor, configured to merge the virtual world coordinates and the real world coordinates, and merge the virtual scene and the physical object to generate a mixed reality image.

### Claim 2 (depends on 1)

The mixed reality processing system of claim 1, wherein the physical object is a human body, the tracker inputs the two-dimensional image into a segmentation model, and the segmentation model outputs a human body block, and the human body block is a part of the two-dimensional image.

### Claim 3 (depends on 2)

The mixed reality processing system of claim 2, wherein the tracker inputs the two-dimensional image into a skeleton model, and the skeleton model outputs a plurality of human skeleton points, the tracker generates a three-dimensional pose according to the human skeleton points, and the three-dimensional pose is configured to adjust a capture range of the human body block.

### Claim 4 (depends on 2)

The mixed reality processing system of claim 2, wherein the processor is located in the tracker or in an external computer, and the processor is configured to generate merging coordinates after respectively coinciding the origins and coordinate axes of the virtual world coordinates and the real world coordinates;
wherein the processor superimposes the human body block from the tracker on the virtual scene from the head-mounted display according to the merging coordinates to generate the mixed reality image.

### Claim 5 (depends on 1)

The mixed reality processing system of claim 1, wherein the tracker and the camera are located outside the head-mounted display, and outside-in tracking technology is applied to track the position of the head-mounted display.

### Claim 6

A mixed reality processing method, comprising:
capturing a two-dimensional image including a physical object using a camera;
using a head-mounted display (HMD) to display a virtual scene and obtain virtual world coordinates of the physical object in the virtual world through a first simultaneous localization and mapping map (SLAM map), wherein the virtual world coordinates used to generate the virtual scene are based on real world coordinates;
using a tracker to differentiate the physical object from the two-dimensional image and obtain the real world coordinates of the physical object in the real world through a simultaneous localization and mapping map; and
using a processor to merge the virtual world coordinates and the real world coordinates and merge the virtual scene and the physical object to generate a mixed reality image.

### Claim 7 (depends on 6)

The mixed reality processing method of claim 6, wherein the physical object is a human body, the mixed reality processing method further comprising:
the tracker inputs the two-dimensional image into a segmentation model, and the segmentation model outputs a human body block;
wherein the human body block is a part of the two-dimensional image.

### Claim 8 (depends on 7)

The mixed reality processing method of claim 7, further comprising:
using the tracker to input the two-dimensional image into a skeleton model;
wherein the skeleton model outputs a plurality of human skeleton points, the tracker generates a three-dimensional pose according to the human skeleton points, the three-dimensional pose is configured to adjust the capture range of the human body block.

### Claim 9 (depends on 7)

The mixed reality processing method of claim 7, wherein the processor is located in the tracker or in an external computer, and the mixed reality processing method further comprises:
the processor generates merging coordinates after respectively coinciding the origins and coordinate axes of the virtual world coordinates and the real world coordinates;
using the processor to generate the mixed reality image by superimposing the human body block from the tracker on the virtual scene from the head-mounted display according to the merging coordinates.

### Claim 10 (depends on 6)

The mixed reality processing method of claim 6, wherein the tracker and the camera are located outside the head-mounted display, and outside-in tracking technology is applied to track the position of the head-mounted display.
