US 10,855,965 B1Granted

Dynamic multi-view rendering for autostereoscopic displays by generating reduced number of views for less-critical segments based on saliency/depth/eye gaze map

AssigneeHong Kong Applied Science and Technology Research Institute Company Limited

InventorsYuzhong Jiao, Man Chi Chan, Ping Chan Mok

Life of this patent

Estimating the term…

Loading patent…

Abstract

A segmented 3D multi-view image generator generates fewer multi-view view images for partitions having less salient features. Saliency values are calculated based on a depth map and image processing of the input image. The saliency values are compared to thresholds to map pixel locations to first, second, and third partitions. First, second, and third segmented images are created from the input image using a partition map. A multi-view generator uses the depth map and viewer eye locations to generates 28 view images from the first segmented image, 14 unique view images from the second segmented image that are replicated to 28 view images, and 7 unique view images from the third segmented image that are replicated to 28 view images. The view images for each segment are interlaced to generated interlaced segmented images that are then integrated together into a single 3D image that drives a 28-view autostereoscopic display.

Claims20

Dependent claims

A reduced-load multi-view 3D image generator comprising:

an input that receives an input image and a depth map for the input image;
a partitioner that generates a partition map, the partition map having K partition indicators for pixel locations in the input image, a partition indicator for indicating that a pixel at a pixel location is assigned to one of K partitions;
a segmenter that merges the partition map with the input image to generate a K segmented images;
a multi-view generator that generates multiple view images for display on an autostereoscopic display to viewers at different viewpoints to provide a 3D effect, wherein pixels are shifted by a different parallax value for different multiple view images to create the 3D effect for the different viewing eye positions;
wherein the multi-view generator shifts pixels in the first of the K segmented images to generate M first segmented view images, and shifts pixels in a second of the K segmented images to generate N second segmented view images, and shifts pixels in a third of the K segmented images to generate P third segmented view images when K is at least 3;
wherein K, M, N, and P are whole numbers, and M is greater than N, and N is greater than P, and K is at least 2;
a view expander that generates M-N additional second segmented view images from the N second segmented view images to generate M second segmented view images;
wherein the view expander further generates M-P additional third segmented view images from the P third segmented view images to generate M third segmented view images when K is at least 2;
a view interlacer and integrator that interlaces pixels from multiple view images for multiple segments into an interlaced image in an interlacing pattern that allows the autostereoscopic display to display the multiple view images to the different viewing eye positions;
wherein the view interlacer and integrator integrating and interlacing pixels from the M first segmented view images, from the M second segmented view images, and from the M third segmented view images into an interlaced display image for display on the autostereoscopic display,
whereby the multi-view generator generates M-N and M-P fewer segmented view images to reduce computational load.

8 more elements

A segmented multi-view image generator comprising:

an image input for receiving an input image and a depth map, the depth map indicating depths or disparities of pixel locations in the input image;
a partitioner that generates a partition map from the depth map, the partition map having partition indicators for the pixel locations in the input image, the partition indicators indicating a partition for the pixel location, the partition being a first partition or a second partition;
a segmenter that generates a first segmented image by removing from the input image pixels in pixel locations that do not have the partition indicator indicating the first partition to generate the first segmented image, the segmenter also generating a first segmented depth map by removing from the depth map depth values for pixel locations that do not have the partition indicator indicating the first partition to generate the first segmented depth map;
wherein the segmenter also generates a second segmented image by removing from the input image pixels in pixel locations that do not have the partition indicator indicating the second partition to generate the second segmented image, the segmenter also generating a second segmented depth map by removing from the depth map depth values for pixel locations that do not have the partition indicator indicating the second partition to generate the second segmented depth map;
a multi-view generator that generates a first plurality of first segmented view images by shifting pixels in the first segmented image by a shift amount determined from a depth value for the pixel location from the first segmented depth map, the multi-view generator also generating a second plurality of second segmented view images by shifting pixels in the second segmented image by a shift amount determined from the depth value for the pixel location from the second segmented depth map;
wherein each first segmented view image in the first plurality of first segmented view images has a different value of a view identifier;
wherein the first plurality of first segmented view images has M first segmented view images and has M values of the view identifier, wherein M is a whole number of at least 4;
wherein each second segmented view image in the second plurality of second segmented view images has a different value of the view identifier;
wherein the second plurality of second segmented view images has N second segmented view images and has N values of the view identifier, wherein N is a whole number that is less than M;
wherein the multi-view generator generates a reduced number view images for the second segmented view images;
a view expander that receives the second plurality of second segmented view images and generates M−N additional second segmented view images from the second plurality of second segmented view images to generate an expanded second plurality of second segmented view images having M values of the view identifier;
a view interlacer that generates an interlaced first segmented image from the first plurality of first segmented view images by placing pixels from the first segmented view image into pixel locations in the interlaced first segmented image that are visible to a viewer identified by the view identifier for the first segmented view image, and not visible to other viewers identified by other view identifiers when the interlaced first segmented image is displayed on an autostereoscopic display, wherein pixels from all M first segmented view images are interlaced together to form the interlaced first segmented image that is visible to M different viewers at M different view locations when displayed on the autostereoscopic display;
wherein the view interlacer also generates an interlaced second segmented image from the second plurality of second segmented view images by placing pixels from the second segmented view image into pixel locations in the interlaced second segmented image that are visible to a viewer identified by the view identifier for the second segmented view image, and not visible to other viewers identified by other view identifiers when the interlaced second segmented image is displayed on an autostereoscopic display, wherein pixels from all M second segmented view images in the expanded second plurality are interlaced together to form the interlaced second segmented image that is visible to M different viewers at M different view locations when displayed on the autostereoscopic display;
and
a segment integrator that superimposes the interlaced first segmented image and the interlaced second segmented image to generate an interlaced image for display on the autostereoscopic display;
wherein the multi-view generator generates a reduced number of view images for the second segmented view images than for the first segmented view images to reduce computational load.

13 more elements

A segmented multi-view rendering method comprising:

generating a partition map having at least two partitions;
for each partition, generate a segmented image for that partition that has pixels only at pixel locations indicated for that partition in the partition map, the pixels being copied from pixel locations in an input image;
receiving a depth map having depth values for pixel locations in the input image, wherein the depth values indicate a disparity or a depth of a pixel at the pixel location;
generating a full set of M view images from the segmented image for a first partition;
generating a reduced set of N view images from the segmented image for a second partition;
wherein M and N are whole numbers, and M is greater than N;
wherein the M view images are generated using a parallax shift that is a function of the depth values and a viewpoint, wherein the viewpoint is different for each of the M view images;
generating M−N replicated view images from the reduced set of N view images to generate an expanded set of view images for the second partition;
interlacing pixels from the full set of M view images for the first partition to generate an interlaced first image;
interlacing pixels from the expanded set of view images for the second partition to generate an interlaced second image;
integrating the interlaced first image and the interlaced second image to generate a 3D display image;
and
sending the 3D display image to an autostereoscopic display that generates M view images to M viewpoints from the 3D display image.

10 more elements

Dependencies are inferred from the claim text.