Remote control manipulator system and remote control assistance system

A remote control manipulator system comprising:

• a manipulator controlled by an operator remotely to handle an object;
• an operation instruction inputter through which the operator inputs an operation instruction instructing an operation of the manipulator to move or to be still;
• a control signal generator to generate a control signal controlling the manipulator from the operation instruction;
• a right camera and a left camera mounted on the manipulator, the right camera and the left camera having a predetermined positional relationship;
• a stereoscopic display device to display a right-eye image captured by the right camera and a left-eye image captured by the left camera to be viewed stereoscopically;
• a position and posture sensor to detect position and posture information representing a position and a posture of the manipulator;
• a ranging sensor being mounted on the manipulator and to measure ranging data including distances and directions to the object and a surrounding environment being a thing being present around the object;
• a structure data storage to store manipulator structure data representing a structure of the manipulator, camera structure data representing structures of the right camera and the left camera, and ranging sensor structure data representing a structure of the ranging sensor;
• a manipulator model generator to generate a manipulator model being a three-dimensional model of the manipulator including the right camera, the left camera, and the ranging sensor by referring to the manipulator structure data, the camera structure data, the ranging sensor structure data, and the position and posture information;
• an image processing model generator to perform image processing on the right-eye image and the left-eye image based on positions and capturing directions of the right camera and the left camera included in the manipulator model to generate a surrounding environment model being a three-dimensional model of the surrounding environment;
• a ranging data processing model generator to process the ranging data to generate the surrounding environment model based on a position and a ranging direction of the ranging sensor included in the manipulator model;
• a model integrator to generate an integrated three-dimensional model being a three-dimensional model obtained by integrating the manipulator model and at least one of a camera surrounding environment model and a sensor surrounding environment model, the camera surrounding environment model being the surrounding environment model generated by the image processing model generator, the sensor surrounding environment model being the surrounding environment model generated by the ranging data processing model generator;
• a model image generator to generate a model image being an image of the integrated three-dimensional model viewed from a viewpoint designated by the operator;
• and
• a model image display device to display the model image and configure to be able to be viewed by the operator switching between the stereoscopic display device and the model image display device through moving the head of the operator.

A remote control manipulator system comprising:

• a manipulator controlled remotely by an operator to handle an object;
• an operation instruction inputter through which the operator inputs an operation instruction instructing an operation of the manipulator to move or to be still;
• a control signal generator to generate a control signal controlling the manipulator from the operation instruction;
• a right camera and a left camera mounted on the manipulator, the right camera and the left camera having a predetermined positional relationship;
• a stereoscopic display device to display a right-eye image captured by the right camera and a left-eye image captured by the left camera to be viewed stereoscopically;
• a ranging sensor being mounted on the manipulator and to measure ranging data including distances and directions to the object and a surrounding environment being a thing being present around the object;
• a position and posture sensor to detect position and posture information representing a position and a posture of the manipulator;
• a structure data storage to store manipulator structure data representing a structure of the manipulator and ranging sensor structure data representing a structure of the ranging sensor;
• a manipulator model generator to generate a manipulator model being a three-dimensional model of the manipulator including the ranging sensor by referring to the manipulator structure data, the ranging sensor structure data, and the position and posture information;
• a ranging data processing model generator to process the ranging data to generate a surrounding environment model being a three-dimensional model of the surrounding environment by referring to a position and a ranging direction of the ranging sensor included in the manipulator model;
• a model integrator to generate an integrated three-dimensional model being a three-dimensional model obtained by integrating the surrounding environment model and the manipulator model;
• a model image generator to generate a model image being an image of the integrated three-dimensional model viewed from a viewpoint designated by the operator;
• and
• a model image display device to display the model image and configure to be able to be viewed by the operator switching between the stereoscopic display device and the model image display device through moving the head of the operator.

A remote control assistance system comprising:

• a stereoscopic display device to display a right-eye image and a left-eye image captured by a right camera and a left camera to be viewed stereoscopically, the right camera and the left camera being mounted on a manipulator controlled by an operator remotely to handle an object and having a predetermined positional relationship;
• a structure data storage to store manipulator structure data representing a structure of the manipulator and camera structure data representing structures of the right camera and the left camera;
• a manipulator model generator to generate a manipulator model being a three-dimensional model of the manipulator including the right camera and the left camera by referring to the manipulator structure data, the camera structure data, and position and posture information representing a position and a posture of the manipulator;
• an image processing model generator to perform image processing on the right-eye image and the left-eye image based on positions and capturing directions of the right camera and the left camera included in the manipulator model to generate a surrounding environment model being a three-dimensional model of the object and a surrounding environment being a thing being present around the object;
• a model integrator to generate an integrated three-dimensional model being a three-dimensional model obtained by integrating the surrounding environment model and the manipulator model;
• a model image generator to generate a model image being an image of the integrated three-dimensional model viewed from a viewpoint designated by the operator;
• and
• a model image display device to display the model image and configure to be able to be viewed by the operator switching between the stereoscopic display device and the model image display device through moving the head of the operator.

A remote control assistance system comprising:

• a stereoscopic display device to display a right-eye image captured by a right camera and a left-eye image captured by a left camera to be viewed stereoscopically, the right camera and the left camera being mounted on a manipulator controlled by an operator remotely to handle an object, and having a predetermined positional relationship;
• a ranging sensor being mounted on the manipulator and to measure ranging data including distances and directions to the object and a surrounding environment being a thing being present around the object;
• a structure data storage to store manipulator structure data representing a structure of the manipulator, camera structure data representing structures of the right camera and the left camera, and ranging sensor structure data representing a structure of the ranging sensor;
• a manipulator model generator to generate a manipulator model being a three-dimensional model of the manipulator including the right camera, the left camera, and the ranging sensor by referring to the manipulator structure data, the camera structure data, the ranging sensor structure data, and position and posture information representing a position and a posture of the manipulator;
• an image processing model generator to perform image processing on the right-eye image and the left-eye image based on positions and capturing directions of the right camera and the left camera included in the manipulator model to generate a surrounding environment model being a three-dimensional model of the surrounding environment;
• a ranging data processing model generator to process the ranging data to generate the surrounding environment model based on a position and a ranging direction of the ranging sensor included in the manipulator model;
• a model integrator to generate an integrated three-dimensional model being a three-dimensional model obtained by integrating the manipulator model and at least one of a camera surrounding environment model and a sensor surrounding environment model, the camera surrounding environment model being the surrounding environment model generated by the image processing model generator, the sensor surrounding environment model being the surrounding environment model generated by the ranging data processing model generator;
• a model image generator to generate a model image being an image of the integrated three-dimensional model viewed from a viewpoint designated by the operator;
• and
• a model image display device to display the model image and configure to be able to be viewed by the operator switching between the stereoscopic display device and the model image display device through moving the head of the operator.

A remote control assistance system comprising:

• a stereoscopic display device to display a right-eye image captured by a right camera and a left-eye image captured by a left camera to be viewed stereoscopically, the right camera and the left camera being mounted on a manipulator controlled by an operator remotely to handle an object, and having a predetermined positional relationship;
• a ranging sensor being mounted on the manipulator and to measure ranging data including distances and directions to the object and a surrounding environment being a thing being present around the object;
• a structure data storage to store manipulator structure data representing a structure of the manipulator and ranging sensor structure data representing a structure of the ranging sensor;
• a manipulator model generator to generate a manipulator model being a three-dimensional model of the manipulator including the ranging sensor by referring to the manipulator structure data, the ranging sensor structure data, and position and posture information representing a position and a posture of the manipulator;
• a ranging data processing model generator to process the ranging data to generate a surrounding environment model being a three-dimensional model of the surrounding environment based on a position and a ranging direction of the ranging sensor included in the manipulator model;
• a model integrator to generate an integrated three-dimensional model being a three-dimensional model obtained by integrating the surrounding environment model and the manipulator model;
• a model image generator to generate a model image being an image of the integrated three-dimensional model viewed from a viewpoint designated by the operator;
• and
• a model image display device to display the model image and configure to be able to be viewed by the operator switching between the stereoscopic display device and the model image display device through moving the head of the operator.