Inconsistency detecting system, mixed-reality system, program, and inconsistency detecting method

A mixed-reality system that includes a server as well as a portable display device having a transmissive display unit for displaying a virtual object to a user and also having a photographing unit for photographing a real space and that renders the virtual object on the display unit in a predetermined real space, whereby the virtual object is visually recognized by the user as being superimposed on the real space through the display unit,

• wherein each of the following part is included in the server or the display device:
• a three-dimensional-apace-data storage part configured to store three-dimensional space data including point cloud data of a real object located in the predetermined real space, acquired in advance, the point cloud data each having three-dimensional position information;
• a user-environment determining part configured to determine, on the basis of data obtained from a sensor included in the display device and the three-dimensional space data, a user environment including the position of the display device and also including a visual-field region that is visually recognized by the user through the display unit;
• a rendering part configured to render the virtual object on the display unit on the basis of the user environment;
• a first point-cloud-data generating part configured to generate first point cloud data, which is point cloud data of the real object and the virtual object, from a combined image obtained by displaying the virtual object that is rendered by the rendering part, in a superimposed fashion, on a real-space image that is visually recognized by the user through the display unit, the real-space image being generated from the photographed real-space image;
• a second point-cloud-data generating part configured to generate second point cloud data by using point cloud data in the visual-field region determined by the user-environment determining part, stored by the three-dimensional-space-data storage part, as well as point cloud data of the virtual object that is rendered by the rendering part;
• and
• an inconsistency detecting part configured to detect an inconsistency on the basis of a result of comparison between the first point cloud data and the second point cloud data.

A non-transitory computer readable medium storing a program for detecting an inconsistency in a mixed-reality system that includes a portable display device having a transmissive display unit for displaying a virtual object to a user and also having a photographing unit for photographing a real space and that renders the virtual object on the display unit in a predetermined real space, whereby the virtual object is visually recognized by the user as being superimposed on the real space through the display unit,

• wherein the mixed-reality system includes:
• a three-dimensional-space-data storage part configured to store three-dimensional space data including point cloud data of a real object located in the predetermined real space, acquired in advance, the point cloud data each having three-dimensional position information;
• a user-environment determining part configured to determine, on the basis of data obtained from a sensor included in the display device and the three-dimensional space data, a user environment including the position of the display device and also including a visual-field region that is visually recognized by the user through the display unit;
• and
• a rendering part configured to render the virtual object on the display unit on the basis of the user environment, and
• wherein the program causes the display device to execute:
• a step of generating first point cloud data, which is point cloud data of the real object and the virtual object, from a combined image obtained by displaying the virtual object that is rendered by the rendering part, in a superimposed fashion, on a real-space image that is visually recognized by the user through the display unit, the real-space image being generated from the photographed real-space image;
• a step of generating second point cloud data by using point cloud data in the visual-field region determined by the user-environment determining part, stored by the three-dimensional-space-data storage part, as well as point cloud data of the virtual object that is rendered by the rendering part;
• and
• a step of detecting an inconsistency on the basis of a result of comparison between the first point cloud data and the second point cloud data.

A method of detecting an inconsistency in a mixed-reality system that includes a portable display device having a transmissive display unit for displaying a virtual object to a user and also having a photographing unit for photographing a real space and that renders the virtual object on the display unit in a predetermined real space, whereby the virtual object is visually recognized by the user as being superimposed on the real space through the display unit,

• wherein the mixed-reality system includes:
• a three-dimensional-space-data storage part configured to store three-dimensional space data including point cloud data of a real object located in the predetermined real space, acquired in advance, the point cloud data each having three-dimensional position information;
• a user-environment determining part configured to determine, on the basis of data obtained from a sensor included in the display device and the three-dimensional space data, a user environment including the position of the display device and also including a visual-field region that is visually recognized by the user through the display unit;
• and
• a rendering part configured to render the virtual object on the display unit on the basis of the user environment, and
• the method comprising:
• a step of generating first point cloud data, which is point cloud data of the real object and the virtual object, from a combined image obtained by displaying the virtual object that is rendered by the rendering part, in a superimposed fashion, on a real-space image that is visually recognized by the user through the display unit, the real-space image being generated from the photographed real-space image;
• a step of generating second point cloud data by using point cloud data in the visual-field region determined by the user-environment determining part, stored by the three-dimensional-space-data storage part, as well as point cloud data of the virtual object that is rendered by the rendering part;
• and
• a step of detecting an inconsistency on the basis of a result of comparison between the first point cloud data and the second point cloud data.