# US 2021/0174599 A1 · Mixed reality system, program, mobile terminal device, and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174599 A1
- Document ID: US20210174599A1
- Status: Published application
- Kind: A1
- Application number: 17/181,456
- Filed: 2021-02-22
- Published: 2021-06-10
- Earliest claimed: 2018-08-24
- Assignees: Cygames, Inc.
- Applicants: Cygames, Inc.
- Inventors: Shuichi Kurabayashi
- Web page: https://patentgrep.com/patent/US20210174599A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/25: Output arrangements for video game devices
- A63F 13/525: Changing parameters of virtual cameras
- A63F 13/98: Accessories, i.e. detachable arrangements optional for the use of the video game device, e.g. grip supports of game controllers
- G06F 3/01: Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence)
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06T 7/73: using feature-based methods
- G06T 2207/30204: Marker
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

One or more embodiments of the invention is a mixed reality system for displaying a mixed-reality image on a display, the system including a mobile terminal device having the display and a photographing device. The system includes a plurality of feature point sets arranged in the prescribed real space. The mobile terminal device recognizes each of the feature point sets photographed by the photographing device, determines a viewpoint position of a virtual camera, in a virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, based on arrangement position and posture information for each of the feature point sets and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, and generates a mixed-reality image in which an image of the virtual object is superimposed on a photographed image of the real space.

## Claims

### Claim 1

A mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system comprising a mobile terminal device having the display and a photographing device that photographs the real space, wherein:
the mixed reality system
includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and
the mobile terminal device
stores data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object, stores arrangement position and posture information in the virtual space for each of the feature point sets arranged in the prescribed real space,
recognizes each of the feature point sets photographed by the photographing device,
determines a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, on the basis of the arrangement position and posture information for each of the feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and
on the basis of the data of the first virtual object, the data of the second virtual object, and the viewpoint position, generates a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the real space.

### Claim 2 (depends on 1)

A mixed reality system according to claim 1, wherein, in the case where the first virtual object and the second virtual object overlap each other in a field-of-view range of the virtual camera when the mixed-reality image is generated, if a depth distance of the second virtual object from the viewpoint position is larger than that of the first virtual object, the mobile terminal device generates a mixed-reality image in which the image of the second virtual object with overlapping parts removed is superimposed.

### Claim 3 (depends on 1)

A mixed reality system according to claim 1, wherein
the mixed reality system includes five or more of said feature point sets, and
the mobile terminal device, at prescribed time intervals, recognizes each of the feature point sets photographed by the photographing device, determines the viewpoint position, and generates the mixed-reality image.

### Claim 4 (depends on 1)

A mixed reality system according to claim 1, wherein the mixed reality system includes four of said feature point sets arranged in the prescribed real space, and the individual feature point sets have positional relationships corresponding to four vertex positions of a virtual rectangle in the real space and are arranged away from each other by a predefined distance in a vertical direction and a predefined distance in a horizontal direction.

### Claim 5 (depends on 4)

A mixed reality system according to claim 4, wherein one or a plurality of the feature point sets are securely attached to each of a plurality of rod-like bodies installed in a horizontally-isolated manner.

### Claim 6 (depends on 1)

A mixed reality system according to claim 1, wherein each of the feature point sets is securely attached to a transmission member formed of a light transmitting material.

### Claim 7 (depends on 1)

A mixed reality system according to claim 1, wherein each of the feature point sets is securely attached by using a wire.

### Claim 8 (depends on 1)

A mixed reality system according to claim 1, wherein the feature point sets are posters, paintings, pictures, ArUco markers, AR markers, two-dimensional codes, or data matrices.

### Claim 9

A non-transitory computer readable medium storing a program executed by a mobile terminal device in a mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system comprising the mobile terminal device having the display and a photographing device that photographs the real space, wherein:
the mixed reality system
includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and
the program causes the mobile terminal device to execute
a step of recognizing each of the feature point sets photographed by the photographing device,
a step of determining a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, on the basis of arrangement position and posture information in the virtual space in which each of the feature point sets is arranged, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and
a step of generating a mixed-reality image in which an image of a second virtual object according to the viewpoint position is superimposed on the photographed image of the real space, on the basis of data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, data of the second virtual object, in the virtual space, that does not correspond to the real object, and the viewpoint position.

### Claim 10

A mobile terminal device in a mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system comprising the mobile terminal device having the display and a photographing device that photographs the real space, wherein:
the mixed reality system
includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and
the mobile terminal device
stores data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object, stores arrangement position and posture information in the virtual space for each of the feature point sets arranged in the prescribed real space,
recognizes each of the feature point sets photographed by the photographing device,
determines a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, on the basis of the arrangement position and posture information for each of the feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and
on the basis of the data of the first virtual object, the data of the second virtual object, and the viewpoint position, generates a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the real space.

### Claim 11

A method executed by a mobile terminal device in a mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system comprising the mobile terminal device having the display and a photographing device that photographs the real space, wherein:
the mixed reality system
includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and
the method includes
a step of recognizing each of the feature point sets photographed by the photographing device,
a step of determining a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, on the basis of arrangement position and posture information in the virtual space in which each of the feature point sets is arranged, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and
a step of generating a mixed-reality image in which an image of a second virtual object according to the viewpoint position is superimposed on the photographed image of the real space, on the basis of data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, data of the second virtual object, in the virtual space, that does not correspond to the real object, and the viewpoint position.

### Claim 12

A mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system comprising a server and a mobile terminal device having the display and a photographing device that photographs the real space, wherein:
the mixed reality system
includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships;
the server
stores data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object, stores arrangement position and posture information in the virtual space for each of the feature point sets arranged in the prescribed real space; and
the mobile terminal device
recognizes each of the feature point sets photographed by the photographing device,
determines a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, on the basis of the arrangement position and posture information for each of the feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and
on the basis of the data of the first virtual object, the data of the second virtual object, and the viewpoint position, generates a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the real space.

### Claim 13

A mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system comprising a mobile terminal device having the display and a photographing device that photographs the real space, wherein:
the mixed reality system
includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and
the mobile terminal device
stores data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object, stores arrangement position and posture information in the virtual space for each of the feature point sets arranged in the prescribed real space,
recognizes each of the feature point sets photographed by the photographing device,
determines position and posture information of the first virtual object on the basis of the arrangement position and posture information for each of the feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and
on the basis of the data of the first virtual object and the data of the second virtual object, generates a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the real space.

### Claim 14

A non-transitory computer readable medium storing a program executed by a mobile terminal device in a mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system comprising the mobile terminal device having the display and a photographing device that photographs the real space, wherein:
the mixed reality system
includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and
the program causes the mobile terminal device to execute
a step of recognizing each of the feature point sets photographed by the photographing device,
a step of determining position and posture information of a first virtual object, obtained in advance, that corresponds to a real object present in the prescribed real space and that defines the virtual space, on the basis of arrangement position and posture information in the virtual space in which each of the feature point sets is arranged, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and
a step of generating a mixed-reality image in which an image of a second virtual object according to a viewpoint position is superimposed on the photographed image of the real space, on the basis of data of the first virtual object and data of the second virtual object, in the virtual space, that does not correspond to the real object.
