# US 12,223,607 B2 · Mixed reality system, program, mobile terminal device, and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,607 B2
- Document ID: US12223607B2
- Status: Granted
- Kind: B2
- Application number: 18/501,702
- Filed: 2023-11-03
- Published: 2024-02-29
- Granted: 2025-02-11
- Earliest claimed: 2018-08-24
- Assignees: Cygames, Inc.
- Applicants: Cygames, Inc.
- Inventors: Shuichi Kurabayashi
- Web page: https://patentgrep.com/patent/US12223607B2

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/25: Output arrangements for video game devices
- A63F 13/525: Changing parameters of virtual cameras
- A63F 13/98: Accessories, i.e. detachable arrangements optional for the use of the video game device, e.g. grip supports of game controllers
- G06F 3/01: Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence)
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06T 7/73: using feature-based methods
- G06T 2207/30204: Marker
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

One or more embodiments of the invention is a mixed reality system for displaying a mixed-reality image on a display, the system including a mobile terminal device having the display and a photographing device. The system includes a plurality of feature point sets arranged in the prescribed real space. The mobile terminal device recognizes each of the feature point sets photographed by the photographing device, determines a viewpoint position of a virtual camera, in a virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, based on arrangement position and posture information for each of the feature point sets and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, and generates a mixed-reality image in which an image of the virtual object is superimposed on a photographed image of the real space.

## Claims

### Claim 1

A mixed reality system for displaying, on a display for displaying a first virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the first virtual object arranged in a virtual space corresponding to the prescribed real space is superimposed on a photographed image of the prescribed real space, the mixed reality system comprising a mobile terminal device having the display and a photographing device that photographs the prescribed real space, wherein:
the mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the plurality of feature point sets including identifiable information that allows identification of each of the plurality of feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and
the mobile terminal device is configured to:
store data, obtained in advance, of the first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object,
store arrangement position and posture information in the virtual space for each of the plurality of feature point sets arranged in the prescribed real space,
recognize each of the plurality of feature point sets photographed by the photographing device,
determine a plurality of projections using a homography matrix, a template image of a feature point set among the plurality of feature point sets, and a plurality of local features in the photographed image,
wherein the plurality of projections are selected from a group consisting of one or more rotation projections, one or more enlargement projections, one or more reduction projections, and one or more deformation projections,
determine a relative position of the mobile terminal device using the plurality of projections,
determine a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the prescribed real space, based on the arrangement position and posture information for each of the plurality of feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and the relative position and posture information of the mobile terminal device with respect to each of the plurality of feature point sets, the relative position and posture information being determined from shapes and sizes of the plurality of feature point sets, and
determine a first depth distance between the viewpoint position of the virtual camera and the first virtual object using the plurality of feature point sets,
determine a second depth distance between the viewpoint position of the virtual camera and the second virtual object using the plurality of feature point sets,
determine whether the first virtual object is closer to the viewpoint position than the second virtual object based on the first depth distance and the second depth distance,
generate, in response to determining that the second virtual object is closer than the first virtual object and based on the first depth distance of the first virtual object, the second depth distance of the second virtual object, and the viewpoint position, a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the prescribed real space,
wherein a first portion of the second virtual object is superimposed on the first virtual object in the mixed-reality image in response to determining that the second virtual object is closer to the viewpoint position than the first virtual object,
wherein a second portion of the first virtual object is not displayed in the mixed-reality image based on the first depth distance of the second portion of the first virtual object being located behind the real object in the prescribed real space,
wherein the plurality of feature point sets comprise five or more of feature point sets that are AR markers, and
wherein the mobile terminal device, at prescribed time intervals, recognizes each of the five or more feature point sets taken by the photographing device to determine the viewpoint position for generating the mixed-reality image.

### Claim 2 (depends on 1)

A mixed reality system according to claim 1, wherein, in response to the first virtual object and the second virtual object overlapping each other in a field-of-view range of the virtual camera when the mixed-reality image is generated, if a second depth distance of the second virtual object from the viewpoint position is larger than that of the first virtual object, the mobile terminal device generates a mixed-reality image in which the image of the second virtual object with overlapping parts removed is superimposed.

### Claim 3 (depends on 1)

A mixed reality system according to claim 1, wherein the mixed reality system includes four of the plurality of feature point sets arranged in the prescribed real space, and individual feature point sets among the plurality of feature point sets have positional relationships corresponding to four vertex positions of a virtual rectangle in the prescribed real space and are arranged away from each other by a predefined distance in a vertical direction and a predefined distance in a horizontal direction.

### Claim 4 (depends on 3)

A mixed reality system according to claim 3, wherein at least one of the plurality of feature point sets are securely attached to each of a plurality of rod-like bodies installed in a horizontally-isolated manner.

### Claim 5 (depends on 1)

A mixed reality system according to claim 1, wherein each of the plurality of feature point sets is securely attached to a transmission member formed of a light transmitting material.

### Claim 6 (depends on 1)

A mixed reality system according to claim 1, wherein each of the plurality of feature point sets is securely attached by using a wire.

### Claim 7

A non-transitory computer readable medium storing a program executed by a mobile terminal device in a mixed reality system for displaying, on a display for displaying a first virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the first virtual object arranged in a virtual space corresponding to the prescribed real space is superimposed on a photographed image of the prescribed real space, the mixed reality system comprising the mobile terminal device having the display and a photographing device that photographs the prescribed real space, wherein:
the mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the plurality of feature point sets including identifiable information that allows identification of each of the plurality of feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and
the program causes the mobile terminal device to execute:
a step of recognizing each of the plurality of feature point sets photographed by the photographing device,
a step of determining a plurality of projections using a homography matrix, a template image of a feature point set among the plurality of feature point sets, and a plurality of local features in the photographed image,
wherein the plurality of projections are selected from a group consisting of one or more rotation projections, one or more enlargement projections, one or more reduction projections, and one or more deformation projections,
a step of determining a relative position of the mobile terminal device using the plurality of projections,
a step of determining a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the prescribed real space, based on arrangement position and posture information in the virtual space in which each of the plurality of feature point sets is arranged, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the plurality of feature point sets, the relative position and posture information being determined from shapes and sizes of the plurality of feature point sets,
a step of determining a first depth distance between the viewpoint position of the virtual camera and the first virtual object using the plurality of feature point sets,
a step of determining a second depth distance between the viewpoint position of the virtual camera and a second virtual object using the plurality of feature point sets,
a step of determining whether the first virtual object is closer to the viewpoint position than the second virtual object based on the first depth distance and the second depth distance, and
a step of generating, in response to determining that the second virtual object is closer than the first virtual object and based on the first depth distance of the first virtual object, the second depth distance of the second virtual object, and the viewpoint position, a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the prescribed real space, based on data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, data of the second virtual object, in the virtual space, that does not correspond to the real object, and the viewpoint position,
wherein a first portion of the second virtual object is superimposed on the first virtual object in the mixed-reality image in response to determining that the second virtual object is closer to the viewpoint position than the first virtual object,
wherein a second portion of the first virtual object is not displayed in the mixed-reality image based on the first depth distance of the second portion of the first virtual object being located behind the real object in the prescribed real space,
wherein the plurality of feature point sets comprise five or more of feature point sets that are AR markers, and
wherein the mobile terminal device, at prescribed time intervals, recognizes each of the five or more feature point sets taken by the photographing device to determine the viewpoint position for generating the mixed-reality image.

### Claim 8

A mobile terminal device in a mixed reality system for displaying, on a display for displaying a first virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the first virtual object arranged in a virtual space corresponding to the prescribed real space is superimposed on a photographed image of the prescribed real space, the mixed reality system comprising the mobile terminal device having the display and a photographing device that photographs the prescribed real space, wherein:
the mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the plurality of feature point sets including identifiable information that allows identification of each of the plurality of feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and
the mobile terminal device is configured to:
store data, obtained in advance, of the first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object, store arrangement position and posture information in the virtual space for each of the plurality of feature point sets arranged in the prescribed real space,
recognize each of the plurality of feature point sets photographed by the photographing device,
determine a plurality of projections using a homography matrix, a template image of a feature point set among the plurality of feature point sets, and a plurality of local features in the photographed image,
wherein the plurality of projections are selected from a group consisting of one or more rotation projections, one or more enlargement projections, one or more reduction projections, and one or more deformation projections,
determine a relative position of the mobile terminal device using the plurality of projections,
determine a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the prescribed real space, based on the arrangement position and posture information for each of the plurality of feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and the relative position and posture information of the mobile terminal device with respect to each of the plurality of feature point sets, the relative position and posture information being determined from shapes and sizes of the plurality of feature point sets,
determine a first depth distance between the viewpoint position of the virtual camera and the first virtual object using the plurality of feature point sets,
determine a second depth distance between the viewpoint position of the virtual camera and the second virtual object using the plurality of feature point sets,
determine whether the first virtual object is closer to the viewpoint position than the second virtual object based on the first depth distance and the second depth distance, and
generate, in response to determining that the second virtual object is closer than the first virtual object and based on the first depth distance of the first virtual object, the second depth distance of the second virtual object, and the viewpoint position, a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the prescribed real space,
wherein a first portion of the second virtual object is superimposed on the first virtual object in the mixed-reality image in response to determining that the second virtual object is closer to the viewpoint position than the first virtual object,
wherein a second portion of the first virtual object is not displayed in the mixed-reality image based on the depth distance of the second portion of the first virtual object being located behind the real object in the prescribed real space,
wherein the plurality of feature point sets comprise five or more of feature point sets that are AR markers, and
wherein the mobile terminal device, at prescribed time intervals, recognizes each of the five or more feature point sets taken by the photographing device to determine the viewpoint position for generating the mixed-reality image.

### Claim 9

A method executed by a mobile terminal device in a mixed reality system for displaying, on a display for displaying a first virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the first virtual object arranged in a virtual space corresponding to the prescribed real space is superimposed on a photographed image of the prescribed real space, the mixed reality system comprising the mobile terminal device having the display and a photographing device that photographs the prescribed real space, wherein:
the mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the plurality of feature point sets including identifiable information that allows identification of each of the plurality of feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and
the method comprises:
a step of recognizing each of the plurality of feature point sets photographed by the photographing device,
a step of determining a plurality of projections using a homography matrix, a template image of a feature point set among the plurality of feature point sets, and a plurality of local features in the photographed image,
wherein the plurality of projections are selected from a group consisting of one or more rotation projections, one or more enlargement projections, one or more reduction projections, and one or more deformation projections,
a step of determining a relative position of the mobile terminal device using the plurality of projections,
a step of determining a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the prescribed real space, based on arrangement position and posture information in the virtual space in which each of the plurality of feature point sets is arranged, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and the relative position and posture information of the mobile terminal device with respect to each of the plurality of feature point sets, the relative position and posture information being determined from shapes and sizes of the plurality of feature point sets,
a step of determining a depth distance between the viewpoint position of the virtual camera and the first virtual object using the plurality of feature point sets,
a step of determining a second depth distance between the viewpoint position of the virtual camera and a second virtual object using the plurality of feature point sets,
a step of determining whether the first virtual object is closer to the viewpoint position than the second virtual object based on the first depth distance and the second depth distance, and
a step of generating, in response to determining that the second virtual object is closer than the first virtual object and based on the first depth distance of the first virtual object, the second depth distance of the second virtual object, and the viewpoint position, a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the prescribed real space, based on data, obtained in advance, of the first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, data of the second virtual object, in the virtual space, that does not correspond to the real object, and the viewpoint position,
wherein a first portion of the second virtual object is superimposed on the first virtual object in the mixed-reality image in response to determining that the second virtual object is closer to the viewpoint position than the first virtual object,
wherein a second portion of the first virtual object is not displayed in the mixed-reality image based on the first depth distance of the second portion of the first virtual object being located behind the real object in the prescribed real space,
wherein the plurality of feature point sets comprise five or more of feature point sets that are AR markers, and
wherein the mobile terminal device, at prescribed time intervals, recognizes each of the five or more feature point sets taken by the photographing device to determine the viewpoint position for generating the mixed-reality image.

### Claim 10

A mixed reality system for displaying, on a display for displaying a first virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the first virtual object arranged in a virtual space corresponding to the prescribed real space is superimposed on a photographed image of the prescribed real space, the mixed reality system comprising a server and a mobile terminal device having the display and a photographing device that photographs the prescribed real space, wherein:
the mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the plurality of feature point sets including identifiable information that allows identification of each of the plurality of feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships;
the server is configured to:
store data, obtained in advance, of the first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object,
store arrangement position and posture information in the virtual space for each of the plurality of feature point sets arranged in the prescribed real space; and
the mobile terminal device is configured to:
recognize each of the plurality of feature point sets photographed by the photographing device,
determine a plurality of projections using a homography matrix, a template image of a feature point set among the plurality of feature point sets, and a plurality of local features in the photographed image,
wherein the plurality of projections are selected from a group consisting of one or more rotation projections, one or more enlargement projections, one or more reduction projections, and one or more deformation projections,
determine a relative position of the mobile terminal device using the plurality of projections,
determine a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the prescribed real space, based on the arrangement position and posture information for each of the plurality of feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and the relative position and posture information of the mobile terminal device with respect to each of the plurality of feature point sets, the relative position and posture information being determined from shapes and sizes of the plurality of feature point sets,
determine a first depth distance between the viewpoint position of the virtual camera and the first virtual object using the plurality of feature point sets,
determine a second depth distance between the viewpoint position of the virtual camera and the second virtual object using the plurality of feature point sets,
determine whether the first virtual object is closer to the viewpoint position than the second virtual object based on the first depth distance and the second depth distance, and
generate, in response to determining that the second virtual object is closer than the first virtual object and based on the first depth distance of the first virtual object, the second depth distance of the second virtual object, and the viewpoint position, a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the prescribed real space,
wherein a first portion of the second virtual object is superimposed on the first virtual object in the mixed-reality image in response to determining that the second virtual object is closer to the viewpoint position than the first virtual object,
wherein a second portion of the first virtual object is not displayed in the mixed-reality image based on the first depth distance of the second portion of the first virtual object being located behind the real object in the prescribed real space,
wherein the plurality of feature point sets comprise five or more of feature point sets that are AR markers, and
wherein the mobile terminal device, at prescribed time intervals, recognizes each of the five or more feature point sets taken by the photographing device to determine the viewpoint position for generating the mixed-reality image.

### Claim 11

A mixed reality system for displaying, on a display for displaying a first virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the first virtual object arranged in a virtual space corresponding to the prescribed real space is superimposed on a photographed image of the prescribed real space, the mixed reality system comprising a mobile terminal device having the display and a photographing device that photographs the prescribed real space, wherein:
the mixed reality system
includes a plurality of feature point sets arranged in the prescribed real space, the plurality of feature point sets including identifiable information that allows identification of each of the plurality of feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and
the mobile terminal device
stores data, obtained in advance, of the first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object,
stores arrangement position and posture information in the virtual space for each of the plurality of feature point sets arranged in the prescribed real space,
recognizes each of the plurality of feature point sets photographed by the photographing device,
determines a plurality of projections using a homography matrix, a template image of a feature point set among the plurality of feature point sets, and a plurality of local features in the photographed image,
wherein the plurality of projections are selected from a group consisting of one or more rotation projections, one or more enlargement projections, one or more reduction projections, and one or more deformation projections,
determines a relative position of the mobile terminal device using the plurality of projections,
determines position and posture information of the first virtual object based on the arrangement position and posture information for each of the plurality of feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and the relative position and posture information of the mobile terminal device with respect to each of the plurality of feature point sets, the relative position and posture information being determined from shapes and sizes of the plurality of feature point sets,
determines a first depth distance between a viewpoint position of a virtual camera and the first virtual object using the plurality of feature point sets,
determines a second depth distance between the viewpoint position of the virtual camera and the second virtual object using the plurality of feature point sets,
determines whether the first virtual object is closer to the viewpoint position than the second virtual object based on the first depth distance and the second depth distance, and
generates, in response to determining that the second virtual object is closer than the first virtual object and based on the first depth distance of the first virtual object, the second depth distance of the second virtual object, and the viewpoint position, a mixed-reality image in which an image of the second virtual object according to a viewpoint position is superimposed on the photographed image of the prescribed real space,
wherein a first portion of the second virtual object is superimposed on the first virtual object in the mixed-reality image in response to determining that the second virtual object is closer to the viewpoint position than the first virtual object,
wherein a second portion of the first virtual object is not displayed in the mixed-reality image based on the first depth distance of the second portion of the first virtual object being located behind the real object in the prescribed real space,
wherein the plurality of feature point sets comprise five or more of feature point sets that are AR markers, and
wherein the mobile terminal device, at prescribed time intervals, recognizes each of the five or more feature point sets taken by the photographing device to determine the viewpoint position for generating the mixed-reality image.

### Claim 12

A non-transitory computer readable medium storing a program executed by a mobile terminal device in a mixed reality system for displaying, on a display for displaying a first virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the first virtual object arranged in a virtual space corresponding to the prescribed real space is superimposed on a photographed image of the prescribed real space, the mixed reality system comprising the mobile terminal device having the display and a photographing device that photographs the prescribed real space, wherein:
the mixed reality system
includes a plurality of feature point sets arranged in the prescribed real space, the plurality of feature point sets including identifiable information that allows identification of each of the plurality of feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and
the program causes the mobile terminal device to execute
a step of recognizing each of the plurality of feature point sets photographed by the photographing device,
a step of determining a plurality of projections using a homography matrix, a template image of a feature point set among the plurality of feature point sets, and a plurality of local features in the photographed image,
wherein the plurality of projections are selected from a group consisting of one or more rotation projections, one or more enlargement projections, one or more reduction projections, and one or more deformation projections,
a step of determining a relative position of the mobile terminal device using the plurality of projections,
a step of determining position and posture information of the first virtual object, obtained in advance, that corresponds to a real object present in the prescribed real space and that defines the virtual space, based on arrangement position and posture information in the virtual space in which each of the plurality of feature point sets is arranged, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and the relative position and posture information of the mobile terminal device with respect to each of the plurality of feature point sets, the relative position and posture information being determined from shapes and sizes of the plurality of feature point sets,
a step of determining a first depth distance between a viewpoint position of a virtual camera and the first virtual object using the plurality of feature point sets,
determine a second depth distance between the viewpoint position of the virtual camera and a second virtual object using the plurality of feature point sets,
determine whether the first virtual object is closer to the viewpoint position than the second virtual object based on the first depth distance and the second depth distance, and
a step of generating, in response to determining that the second virtual object is closer than the first virtual object and based on the first depth distance of the first virtual object, the second depth distance of the second virtual object, and the viewpoint position, a mixed-reality image in which an image of a second virtual object according to a viewpoint position is superimposed on the photographed image of the prescribed real space,
wherein a first portion of the second virtual object is superimposed on the first virtual object in the mixed-reality image in response to determining that the second virtual object is closer to the viewpoint position than the first virtual object,
wherein a second portion of the first virtual object is not displayed in the mixed-reality image based on the first depth distance of the second portion of the first virtual object being located behind the real object in the prescribed real space,
wherein the plurality of feature point sets comprise five or more of feature point sets that are AR markers, and
wherein the mobile terminal device, at prescribed time intervals, recognizes each of the five or more feature point sets taken by the photographing device to determine the viewpoint position for generating the mixed-reality image.