Mixed reality system, program, mobile terminal device, and method

A non-transitory computer readable medium storing a program executed by a mobile terminal device in a mixed reality system for displaying, on a display for displaying a first virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the first virtual object arranged in a virtual space corresponding to the prescribed real space is superimposed on a photographed image of the prescribed real space, the mixed reality system comprising the mobile terminal device having the display and a photographing device that photographs the prescribed real space, wherein:

• the mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the plurality of feature point sets including identifiable information that allows identification of each of the plurality of feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships;
• and
• the program causes the mobile terminal device to execute:
• a step of recognizing each of the plurality of feature point sets photographed by the photographing device,
• a step of determining a plurality of projections using a homography matrix, a template image of a feature point set among the plurality of feature point sets, and a plurality of local features in the photographed image,
• wherein the plurality of projections are selected from a group consisting of one or more rotation projections, one or more enlargement projections, one or more reduction projections, and one or more deformation projections,
• a step of determining a relative position of the mobile terminal device using the plurality of projections,
• a step of determining a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the prescribed real space, based on arrangement position and posture information in the virtual space in which each of the plurality of feature point sets is arranged, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the plurality of feature point sets, the relative position and posture information being determined from shapes and sizes of the plurality of feature point sets,
• a step of determining a first depth distance between the viewpoint position of the virtual camera and the first virtual object using the plurality of feature point sets,
• a step of determining a second depth distance between the viewpoint position of the virtual camera and a second virtual object using the plurality of feature point sets,
• a step of determining whether the first virtual object is closer to the viewpoint position than the second virtual object based on the first depth distance and the second depth distance, and
• a step of generating, in response to determining that the second virtual object is closer than the first virtual object and based on the first depth distance of the first virtual object, the second depth distance of the second virtual object, and the viewpoint position, a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the prescribed real space, based on data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, data of the second virtual object, in the virtual space, that does not correspond to the real object, and the viewpoint position,
• wherein a first portion of the second virtual object is superimposed on the first virtual object in the mixed-reality image in response to determining that the second virtual object is closer to the viewpoint position than the first virtual object,
• wherein a second portion of the first virtual object is not displayed in the mixed-reality image based on the first depth distance of the second portion of the first virtual object being located behind the real object in the prescribed real space,
• wherein the plurality of feature point sets comprise five or more of feature point sets that are AR markers, and
• wherein the mobile terminal device, at prescribed time intervals, recognizes each of the five or more feature point sets taken by the photographing device to determine the viewpoint position for generating the mixed-reality image.

A mobile terminal device in a mixed reality system for displaying, on a display for displaying a first virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the first virtual object arranged in a virtual space corresponding to the prescribed real space is superimposed on a photographed image of the prescribed real space, the mixed reality system comprising the mobile terminal device having the display and a photographing device that photographs the prescribed real space, wherein:

• the mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the plurality of feature point sets including identifiable information that allows identification of each of the plurality of feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships;
• and
• the mobile terminal device is configured to:
• store data, obtained in advance, of the first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object, store arrangement position and posture information in the virtual space for each of the plurality of feature point sets arranged in the prescribed real space,
• recognize each of the plurality of feature point sets photographed by the photographing device,
• determine a plurality of projections using a homography matrix, a template image of a feature point set among the plurality of feature point sets, and a plurality of local features in the photographed image,
• wherein the plurality of projections are selected from a group consisting of one or more rotation projections, one or more enlargement projections, one or more reduction projections, and one or more deformation projections,
• determine a relative position of the mobile terminal device using the plurality of projections,
• determine a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the prescribed real space, based on the arrangement position and posture information for each of the plurality of feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and the relative position and posture information of the mobile terminal device with respect to each of the plurality of feature point sets, the relative position and posture information being determined from shapes and sizes of the plurality of feature point sets,
• determine a first depth distance between the viewpoint position of the virtual camera and the first virtual object using the plurality of feature point sets,
• determine a second depth distance between the viewpoint position of the virtual camera and the second virtual object using the plurality of feature point sets,
• determine whether the first virtual object is closer to the viewpoint position than the second virtual object based on the first depth distance and the second depth distance, and
• generate, in response to determining that the second virtual object is closer than the first virtual object and based on the first depth distance of the first virtual object, the second depth distance of the second virtual object, and the viewpoint position, a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the prescribed real space,
• wherein a first portion of the second virtual object is superimposed on the first virtual object in the mixed-reality image in response to determining that the second virtual object is closer to the viewpoint position than the first virtual object,
• wherein a second portion of the first virtual object is not displayed in the mixed-reality image based on the depth distance of the second portion of the first virtual object being located behind the real object in the prescribed real space,
• wherein the plurality of feature point sets comprise five or more of feature point sets that are AR markers, and
• wherein the mobile terminal device, at prescribed time intervals, recognizes each of the five or more feature point sets taken by the photographing device to determine the viewpoint position for generating the mixed-reality image.

A method executed by a mobile terminal device in a mixed reality system for displaying, on a display for displaying a first virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the first virtual object arranged in a virtual space corresponding to the prescribed real space is superimposed on a photographed image of the prescribed real space, the mixed reality system comprising the mobile terminal device having the display and a photographing device that photographs the prescribed real space, wherein:

• the mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the plurality of feature point sets including identifiable information that allows identification of each of the plurality of feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships;
• and
• the method comprises:
• a step of recognizing each of the plurality of feature point sets photographed by the photographing device,
• a step of determining a plurality of projections using a homography matrix, a template image of a feature point set among the plurality of feature point sets, and a plurality of local features in the photographed image,
• wherein the plurality of projections are selected from a group consisting of one or more rotation projections, one or more enlargement projections, one or more reduction projections, and one or more deformation projections,
• a step of determining a relative position of the mobile terminal device using the plurality of projections,
• a step of determining a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the prescribed real space, based on arrangement position and posture information in the virtual space in which each of the plurality of feature point sets is arranged, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and the relative position and posture information of the mobile terminal device with respect to each of the plurality of feature point sets, the relative position and posture information being determined from shapes and sizes of the plurality of feature point sets,
• a step of determining a depth distance between the viewpoint position of the virtual camera and the first virtual object using the plurality of feature point sets,
• a step of determining a second depth distance between the viewpoint position of the virtual camera and a second virtual object using the plurality of feature point sets,
• a step of determining whether the first virtual object is closer to the viewpoint position than the second virtual object based on the first depth distance and the second depth distance, and
• a step of generating, in response to determining that the second virtual object is closer than the first virtual object and based on the first depth distance of the first virtual object, the second depth distance of the second virtual object, and the viewpoint position, a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the prescribed real space, based on data, obtained in advance, of the first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, data of the second virtual object, in the virtual space, that does not correspond to the real object, and the viewpoint position,
• wherein a first portion of the second virtual object is superimposed on the first virtual object in the mixed-reality image in response to determining that the second virtual object is closer to the viewpoint position than the first virtual object,
• wherein a second portion of the first virtual object is not displayed in the mixed-reality image based on the first depth distance of the second portion of the first virtual object being located behind the real object in the prescribed real space,
• wherein the plurality of feature point sets comprise five or more of feature point sets that are AR markers, and
• wherein the mobile terminal device, at prescribed time intervals, recognizes each of the five or more feature point sets taken by the photographing device to determine the viewpoint position for generating the mixed-reality image.

A mixed reality system for displaying, on a display for displaying a first virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the first virtual object arranged in a virtual space corresponding to the prescribed real space is superimposed on a photographed image of the prescribed real space, the mixed reality system comprising a server and a mobile terminal device having the display and a photographing device that photographs the prescribed real space, wherein:

• the mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the plurality of feature point sets including identifiable information that allows identification of each of the plurality of feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships;
• the server is configured to:
• store data, obtained in advance, of the first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object,
• store arrangement position and posture information in the virtual space for each of the plurality of feature point sets arranged in the prescribed real space;
• and
• the mobile terminal device is configured to:
• recognize each of the plurality of feature point sets photographed by the photographing device,
• determine a plurality of projections using a homography matrix, a template image of a feature point set among the plurality of feature point sets, and a plurality of local features in the photographed image,
• wherein the plurality of projections are selected from a group consisting of one or more rotation projections, one or more enlargement projections, one or more reduction projections, and one or more deformation projections,
• determine a relative position of the mobile terminal device using the plurality of projections,
• determine a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the prescribed real space, based on the arrangement position and posture information for each of the plurality of feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and the relative position and posture information of the mobile terminal device with respect to each of the plurality of feature point sets, the relative position and posture information being determined from shapes and sizes of the plurality of feature point sets,
• determine a first depth distance between the viewpoint position of the virtual camera and the first virtual object using the plurality of feature point sets,
• determine a second depth distance between the viewpoint position of the virtual camera and the second virtual object using the plurality of feature point sets,
• determine whether the first virtual object is closer to the viewpoint position than the second virtual object based on the first depth distance and the second depth distance, and
• generate, in response to determining that the second virtual object is closer than the first virtual object and based on the first depth distance of the first virtual object, the second depth distance of the second virtual object, and the viewpoint position, a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the prescribed real space,
• wherein a first portion of the second virtual object is superimposed on the first virtual object in the mixed-reality image in response to determining that the second virtual object is closer to the viewpoint position than the first virtual object,
• wherein a second portion of the first virtual object is not displayed in the mixed-reality image based on the first depth distance of the second portion of the first virtual object being located behind the real object in the prescribed real space,
• wherein the plurality of feature point sets comprise five or more of feature point sets that are AR markers, and
• wherein the mobile terminal device, at prescribed time intervals, recognizes each of the five or more feature point sets taken by the photographing device to determine the viewpoint position for generating the mixed-reality image.

A mixed reality system for displaying, on a display for displaying a first virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the first virtual object arranged in a virtual space corresponding to the prescribed real space is superimposed on a photographed image of the prescribed real space, the mixed reality system comprising a mobile terminal device having the display and a photographing device that photographs the prescribed real space, wherein:

• the mixed reality system
• includes a plurality of feature point sets arranged in the prescribed real space, the plurality of feature point sets including identifiable information that allows identification of each of the plurality of feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships;
• and
• the mobile terminal device
• stores data, obtained in advance, of the first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object,
• stores arrangement position and posture information in the virtual space for each of the plurality of feature point sets arranged in the prescribed real space,
• recognizes each of the plurality of feature point sets photographed by the photographing device,
• determines a plurality of projections using a homography matrix, a template image of a feature point set among the plurality of feature point sets, and a plurality of local features in the photographed image,
• wherein the plurality of projections are selected from a group consisting of one or more rotation projections, one or more enlargement projections, one or more reduction projections, and one or more deformation projections,
• determines a relative position of the mobile terminal device using the plurality of projections,
• determines position and posture information of the first virtual object based on the arrangement position and posture information for each of the plurality of feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and the relative position and posture information of the mobile terminal device with respect to each of the plurality of feature point sets, the relative position and posture information being determined from shapes and sizes of the plurality of feature point sets,
• determines a first depth distance between a viewpoint position of a virtual camera and the first virtual object using the plurality of feature point sets,
• determines a second depth distance between the viewpoint position of the virtual camera and the second virtual object using the plurality of feature point sets,
• determines whether the first virtual object is closer to the viewpoint position than the second virtual object based on the first depth distance and the second depth distance, and
• generates, in response to determining that the second virtual object is closer than the first virtual object and based on the first depth distance of the first virtual object, the second depth distance of the second virtual object, and the viewpoint position, a mixed-reality image in which an image of the second virtual object according to a viewpoint position is superimposed on the photographed image of the prescribed real space,
• wherein a first portion of the second virtual object is superimposed on the first virtual object in the mixed-reality image in response to determining that the second virtual object is closer to the viewpoint position than the first virtual object,
• wherein a second portion of the first virtual object is not displayed in the mixed-reality image based on the first depth distance of the second portion of the first virtual object being located behind the real object in the prescribed real space,
• wherein the plurality of feature point sets comprise five or more of feature point sets that are AR markers, and
• wherein the mobile terminal device, at prescribed time intervals, recognizes each of the five or more feature point sets taken by the photographing device to determine the viewpoint position for generating the mixed-reality image.

A non-transitory computer readable medium storing a program executed by a mobile terminal device in a mixed reality system for displaying, on a display for displaying a first virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the first virtual object arranged in a virtual space corresponding to the prescribed real space is superimposed on a photographed image of the prescribed real space, the mixed reality system comprising the mobile terminal device having the display and a photographing device that photographs the prescribed real space, wherein:

• the mixed reality system
• includes a plurality of feature point sets arranged in the prescribed real space, the plurality of feature point sets including identifiable information that allows identification of each of the plurality of feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships;
• and
• the program causes the mobile terminal device to execute
• a step of recognizing each of the plurality of feature point sets photographed by the photographing device,
• a step of determining a plurality of projections using a homography matrix, a template image of a feature point set among the plurality of feature point sets, and a plurality of local features in the photographed image,
• wherein the plurality of projections are selected from a group consisting of one or more rotation projections, one or more enlargement projections, one or more reduction projections, and one or more deformation projections,
• a step of determining a relative position of the mobile terminal device using the plurality of projections,
• a step of determining position and posture information of the first virtual object, obtained in advance, that corresponds to a real object present in the prescribed real space and that defines the virtual space, based on arrangement position and posture information in the virtual space in which each of the plurality of feature point sets is arranged, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and the relative position and posture information of the mobile terminal device with respect to each of the plurality of feature point sets, the relative position and posture information being determined from shapes and sizes of the plurality of feature point sets,
• a step of determining a first depth distance between a viewpoint position of a virtual camera and the first virtual object using the plurality of feature point sets,
• determine a second depth distance between the viewpoint position of the virtual camera and a second virtual object using the plurality of feature point sets,
• determine whether the first virtual object is closer to the viewpoint position than the second virtual object based on the first depth distance and the second depth distance, and
• a step of generating, in response to determining that the second virtual object is closer than the first virtual object and based on the first depth distance of the first virtual object, the second depth distance of the second virtual object, and the viewpoint position, a mixed-reality image in which an image of a second virtual object according to a viewpoint position is superimposed on the photographed image of the prescribed real space,
• wherein a first portion of the second virtual object is superimposed on the first virtual object in the mixed-reality image in response to determining that the second virtual object is closer to the viewpoint position than the first virtual object,
• wherein a second portion of the first virtual object is not displayed in the mixed-reality image based on the first depth distance of the second portion of the first virtual object being located behind the real object in the prescribed real space,
• wherein the plurality of feature point sets comprise five or more of feature point sets that are AR markers, and
• wherein the mobile terminal device, at prescribed time intervals, recognizes each of the five or more feature point sets taken by the photographing device to determine the viewpoint position for generating the mixed-reality image.