Target digital twin model generation system, control system for robot, virtual shop generation system, target digital twin model generation method, control method for robot, and virtual shop generation method

A robot control system comprising:

• a sensor that detects information regarding appearance of a handling target of a robot;
• and
• a robot controller that controls the robot so as to perform handling of the handling target on a basis of a detection result of the information regarding the appearance of the handling target and handling information of target master data acquired from a target digital twin model generation system,
• the target digital twin model generation system comprising:
• a light output configured to output planar light having a predetermined pattern and light, which includes visible light and invisible light to a target, having a plurality of wavelengths, from a plurality of illumination positions that surround the target at different timings;
• an imager configured to individually capture images of the target irradiated with the planar light and the target sequentially irradiated with the light having the plurality of wavelengths, at a plurality of imaging positions corresponding to the plurality of illumination positions in synchronization with timings at which the planar light and the light having the plurality of wavelengths are respectively output;
• and
• a controller configured to control the light output and the imager, wherein the controller acquires three-dimensional data indicating a three-dimensional shape over an entire circumference of a surface of the target on a basis of an imaging result of the target irradiated with the planar light, acquires two-dimensional data indicating a two-dimensional appearance over the entire circumference of the target viewed from the plurality of imaging positions on a basis of imaging results of the target sequentially irradiated with the light having the plurality of wavelengths, and generates a target digital twin model that reproduces appearance of the target in a computer-readable form by associating the three-dimensional data with the two-dimensional data,
• wherein the controller acquires texture data indicating texture of the surface of the target as the two-dimensional data on a basis of an imaging result of the target irradiated with the visible light and acquires optical absorption property data in which an optical absorption region on the surface of the target is visualized as the two-dimensional data on a basis of an imaging result of the target irradiated with the invisible light,
• wherein the controller corrects the three-dimensional data based on a generator and a discriminator such that a difference discriminated by the discriminator becomes smaller, and generates the target digital twin model by associating the corrected three-dimensional data with the two-dimensional data, the generator generating a first appearance image indicating appearance of the target at a certain viewpoint from a model generated by pasting the texture data to the three-dimensional data, the discriminator discriminating the difference between the first appearance image generated by the generator and a second appearance image which is generated from the imaging result of the target irradiated with the visible light and which indicates appearance of the target at a same viewpoint as a viewpoint of the first appearance image,
• wherein the controller recognizes additional information added to the surface of the target as characters or figures on a basis of the two-dimensional appearance of the target acquired from the target digital twin model and generates target master data as a comprehensive database regarding the target in which the additional information is registered along with the three-dimensional data and the two-dimensional data,
• wherein the controller acquires a product code and product information associated with the target on a basis of the additional information and further registers the acquired product code and product information in the target master data,
• wherein the controller learns handling information for implementing optimal handling of the target including an optimal gripping position of the target by performing a simulation of handling of the target by a robot and further registers the learned handling information in the target master data.

A virtual shop generation system comprising:

• an acquirer configured to acquire target master data from a target digital twin model generation system;
• and
• a virtual shop generator configured to generate a virtual shop that virtually reproduces a store in which products associated with product codes and product information registered in the target master data acquired by the acquirer are arbitrarily displayed,
• wherein the target digital twin model generation system comprises:
• a light output configured to output planar light having a predetermined pattern and light, which includes visible light and invisible light to a target, having a plurality of wavelengths, from a plurality of illumination positions that surround the target at different timings;
• an imager configured to individually capture images of the target irradiated with the planar light and the target sequentially irradiated with the light having the plurality of wavelengths, at a plurality of imaging positions corresponding to the plurality of illumination positions in synchronization with timings at which the planar light and the light having the plurality of wavelengths are respectively output;
• and
• a controller configured to control the light output and the imager, wherein the controller acquires three-dimensional data indicating a three-dimensional shape over an entire circumference of a surface of the target on a basis of an imaging result of the target irradiated with the planar light, acquires two-dimensional data indicating a two-dimensional appearance over the entire circumference of the target viewed from the plurality of imaging positions on a basis of imaging results of the target sequentially irradiated with the light having the plurality of wavelengths, and generates a target digital twin model that reproduces appearance of the target in a computer-readable form by associating the three-dimensional data with the two-dimensional data,
• wherein the controller acquires texture data indicating texture of the surface of the target as the two-dimensional data on a basis of an imaging result of the target irradiated with the visible light and acquires optical absorption property data in which an optical absorption region on the surface of the target is visualized as the two-dimensional data on a basis of an imaging result of the target irradiated with the invisible light,
• wherein the controller corrects the three-dimensional data based on a generator and a discriminator such that a difference discriminated by the discriminator becomes smaller, and generates the target digital twin model by associating the corrected three-dimensional data with the two-dimensional data, the generator generating a first appearance image indicating appearance of the target at a certain viewpoint from a model generated by pasting the texture data to the three-dimensional data, the discriminator discriminating the difference between the first appearance image generated by the generator and a second appearance image which is generated from the imaging result of the target irradiated with the visible light and which indicates appearance of the target at a same viewpoint as a viewpoint of the first appearance image,
• wherein the controller recognizes additional information added to the surface of the target as characters or figures on a basis of the two-dimensional appearance of the target acquired from the target digital twin model and generates target master data as a comprehensive database regarding the target in which the additional information is registered along with the three-dimensional data and the two-dimensional data,
• wherein the controller acquires a product code and product information associated with the target on a basis of the additional information and further registers the acquired product code and product information in the target master data, and
• wherein the controller learns handling information for implementing optimal handling of the target including an optimal gripping position of the target by performing a simulation of handling of the target by a robot and further registers the learned handling information in the target master data.