Image processing device, system and method based on position detection

An image processing system comprising:

• an imaging device configured to shoot a moving image of a subject equipped with a marker;
• and
• an image processing device configured to generate a display image that changes according to motion of the subject based on a position of the marker in an image frame of the shot moving image, the image processing device includes:
• an image acquirer configured to acquire stereo images from an imaging device comprising a plurality of cameras;
• a position information acquirer configured to acquire positions of objects in a field of view of the imaging device based on shot images of the objects stored in an image storage, wherein said position information acquirer comprises:
• a depth image acquirer configured to acquire a depth image, wherein the position of objects in the depth direction is calculated from the parallax between them, and distances of the respective objects from the cameras in the depth direction are mapped on a two-dimensional coordinate of the image plane and represented as pixel values;
• and
• a marker detector that detects a marker held by a user in a marker detection image;
• a target region detector comprising:
• a first object detector configured to detect a region of an image of a first predetermined object from the shot image or the depth image based on a shape or a feature;
• and
• a second object detector configured to detect a region of an image of a second object based on the detected region of the image of the first predetermined object detected by the first object detector and the pixel value in the depth image;
• a clipping region decider configured to decide a clipping region comprising the first object, the second object, and the marker, wherein said first object and second object comprise the head and body of the user, respectively;
• and
• a display image generator configured to generate a display image by fusing the clipped region with another image,
• wherein a plurality of time delayed images of the user, defined by the first and second objects, and the marker are shot by the imaging device, and the clipping region of each of the time delayed images are successively inserted in a virtual world to create a moving image therein, said virtual world comprising other images and objects that appear to react to the user in the virtual world, thereby allowing interaction between the user and objects in the virtual world,
• wherein the marker is shot by the imaging device n times every time a user image is shot m times, where m and n are natural numbers that indicate a degree of importance of information obtained from the respective images, and
• wherein the plurality of cameras include a first camera and a second camera, wherein the second camera skips shooting an image at a time when the first camera shoots a marker detection image.

An image processing method by which an image processing device generates a display image by using a shot image obtained by shooting by a camera, the image processing method comprising:

• acquiring stereo image data from an imaging device comprising a plurality of cameras and storing the data of the stereo images in a storage device;
• acquiring data of a depth image, wherein the position of objects in the depth direction is calculated from the parallax between them, and distances of the respective objects from the cameras in the depth direction are mapped on a two-dimensional coordinate of the image plane and represented as pixel values;
• reading out the data of the stereo image or the depth image from the storage device and detecting a region of an image of a first predetermined object from the read image based on a shape or a feature;
• detecting a marker held by a user in a marker detection image;
• reading out the data of the depth image from the storage device and detecting a region of an image of a second object from the depth image based on the detected region of the image of the first predetermined object from the read image and the pixel value in the depth image;
• deciding a clipping region including at least the region of the image of the first object, the region of the image of the second object, and the marker, wherein said first object and second object comprise the head and body of the user, respectively;
• and
• reading out the data of the stereo image from the storage device and fusing the clipped region with another image to generate the display image and output the display image to a display device,
• wherein a plurality of time delayed images of the user, defined by the first and second objects, and the marker are shot by the imaging device, and the clipping region of each of the time delayed images are successively inserted in a virtual world to create a moving image therein, said virtual world comprising other images and objects that appear to react to the user in the virtual world, thereby allowing interaction between the user and objects in the virtual world,
• wherein the marker is shot by the imaging device n times every time a user image is shot m times, where m and n are natural numbers that indicate a degree of importance of information obtained from the respective images, and
• wherein the plurality of cameras include a first camera and a second camera, wherein the second camera skips shooting an image at a time when the first camera shoots a marker detection image.

A non-transitory computer-readable recording medium in which a computer program is recorded, the computer program being for a computer, comprising:

• acquiring data of a stereo image from an imaging device comprising a plurality of cameras;
• acquiring a depth image, wherein the position of objects in the depth direction is calculated from the parallax between them, and distances of the respective objects from the cameras in the depth direction are mapped on a two-dimensional coordinate of the image plane and represented as pixel values;
• detecting a region of an image of a first predetermined object from the stereo image or the depth image based on a shape or a feature;
• detecting a region of an image of a second object based on the detected region of the image of the first object from the stereo image or the depth image and the pixel value in the depth image;
• detecting a marker held by a user in a marker detection image;
• deciding a clipping region including at least the region of the image of the first object, the region of the image of the second object, and the marker, wherein said first object and second object comprise the head and body of the user, respectively;
• and
• generating a display image by fusing the clipped region with another image,
• wherein a plurality of time delayed images of the user, defined by the first and second objects, and the marker are shot by the imaging device, and the clipping region of each of the time delayed images are successively inserted in a virtual world to create a moving image therein, said virtual world comprising other images and objects that appear to react to the user in the virtual world, thereby allowing interaction between the user and objects in the virtual world,
• wherein the marker is shot by the imaging device n times every time a user image is shot m times, where m and n are natural numbers that indicate a degree of importance of information obtained from the respective images, and
• wherein the plurality of cameras include a first camera and a second camera, wherein the second camera skips shooting an image at a time when the first camera shoots a marker detection image.