Information processing device, information processing system, and non-transitory computer-readable storage medium for storing program

An information processing system comprising:

• a detection device configured to detect an orientation of a work operator present at a work site;
• and
• a information processing device including:
• a network interface circuit configured to communicate with a mobile terminal movable along with a work operator in a work site;
• a memory configured to store data illustrating a three-dimensional model reproducing an environment of the work site, the three-dimensional model including a plurality of feature points, wherein each of the feature points includes three-dimensional coordinates, feature quantity, and color information;
• and
• a processor coupled to the memory and configured to perform authoring processing in response to an operation by a person different from the work operator, the authoring processing including:
• obtaining, via the network interface circuit, an image photographed by a camera configured to photograph a region of the work site in a line-of-sight direction of the work operator;
• obtaining an estimated location of the work operator and an estimated orientation of the work operator from the mobile terminal via the network interface circuit;
• obtaining a first partial region from among the work site reproduced in the three-dimensional model by using the estimated location and the estimated orientation, the first partial region being a region at the estimated location and having a given size centered on the estimated orientation;
• in response to the obtaining of the first partial region, searching a second partial region from among the first partial region by comparing the plurality of feature points corresponding to the first partial region and predetermined feature points of a work target associated with the work site, the second partial region being a region including the work target in the work site;
• and
• in response to detecting of the second partial region in the searching of the second partial region, registering, in the memory, position information indicating a position of the work target by using the three-dimensional coordinates corresponding to the second partial region, the position information being information configured to be used to display, according to the position of the work target, image information on an image in which the work site is photographed,
• wherein the obtaining of the first partial region is configured to, in response to the obtained image including a scene in which the work operator points out a direction of a position of the work target by an arm portion of the work operator in the work site,
• obtain a pointing direction corresponding to the direction pointed by the arm portion of the work operator by matching a plurality of feature points obtained from the image and a predetermined three-dimensional model corresponding to the arm portion, the arm portion of the work operator including a region from an elbow to a fingertip of the work operator, and
• obtain the first partial region by using an average vector, the average vector being a vector obtained by averaging the orientation of the work operator and the pointing direction obtained based on a result of the matching.

A non-transitory computer-readable storage medium for storing a program which causes a processor to perform authoring processing in response to an operation by a first operator, the authoring processing comprising:

• obtaining data illustrating a three-dimensional model reproducing an environment of a work site, the three-dimensional model including a plurality of feature points, wherein each of the feature points includes three-dimensional coordinates, feature quantity, and color information;
• communicating with a mobile terminal through a network interface to obtain an estimated location of a second operator and an estimated orientation of the second operator from the mobile terminal via the network interface circuit, the mobile terminal being movable along with the second operator in the work site, the second operator being a person other than the first operator;
• obtaining, via the network interface circuit, an image photographed by a camera configured to photograph a region of the work site in a line-of-sight direction of the second operator;
• obtaining a first partial region from among the work site reproduced in the three-dimensional model by using the estimated location and the estimated orientation, the first partial region being a region at the estimated location and having a given size centered on the estimated orientation;
• in response to the obtaining of the first partial region, searching a second partial region from among the first partial region by comparing the plurality of feature points corresponding to the first partial region and predetermined feature points of a work target associated with the work site, the second partial region being a region including the work target in the work site;
• and
• in response to detecting of the second partial region in the searching of the second partial region, registering, in the memory, position information indicating a position of the work target by using the three-dimensional coordinates corresponding to the second partial region, the position information being information configured to be used to display, according to the position of the work target, image information on an image in which the work site is photographed,
• wherein the obtaining of the first partial region is configured to, in response to the obtained image including a scene in which the second operator points out a direction of a position of the work target by an arm portion of the second operator in the work site,
• obtain a pointing direction corresponding to the direction pointed by the arm portion of the second operator by matching a plurality of feature points obtained from the image and a predetermined three-dimensional model corresponding to the arm portion, the arm portion of the second operator including a region from an elbow to a fingertip of the second operator, and
• obtain the first partial region by using an average vector, the average vector being a vector obtained by averaging the orientation of the work operator and the pointing direction obtained based on a result of the matching.