2D-to-3D video frame conversion

A method of processing 2D video images from a video stream for converting the 2D video images to 3D images, the method comprising:

• providing a synthetic 3D reference database comprising computer generated 2D images and corresponding depth information for the 2D images;
• submitting input video frames to the synthetic 3D reference database;
• matching at least a portion of the input video frame with at least a portion of a 2D image in the synthetic 3D reference database;
• selecting the corresponding depth information for that 2D image, wherein the depth information is a depth gradient;
• applying the selected depth information to the matched input video frame portion to generate a 2D plus depth information frame portion;
• reconstructing depth information using a Poisson reconstruction, according to:
• (
• ∂
• 2
• ∂
• x
• 2
• +
• ∂
• 2
• ∂
• y
• 2
• )
• ⁢
• D
• =
• ∇
• ·
• G
• ,
• where G=(Gx, Gy) is the depth gradient, D is the depth information and ∇ G is the divergence of G:
• ∇
• ·
• G
• =
• (
• ∂
• G
• x
• ∂
• x
• +
• ∂
• G
• y
• ∂
• y
• )
• .
• and
• refining depth gradients using:
• G
• x
• =
• G
• x
• ×
• max
• ⁡
• (
• 1
• -
• e
• (
• 1
• -
• 1
• α
• ⁢
• 
• G
• x
• 
• )
• ,
• 0
• )
• G
• y
• =
• G
• y
• ×
• max
• (
• 1
• -
• e
• (
• 1
• -
• 1
• α
• ⁢
• 
• G
• y
• 
• )
• ,
• 0
• )
• .

A method of processing 2D video images from a video stream for converting the 2D video images to 3D images, the method comprising:

• providing a synthetic 3D reference database comprising computer generated 2D images and corresponding depth information for the 2D images;
• submitting input video frames to the synthetic 3D reference database;
• matching at least a portion of the input video frame with at least a portion of a 2D image in the synthetic 3D reference database;
• selecting the corresponding depth information for that 2D image, wherein the depth information is a depth gradient;
• applying the selected depth information to the matched input video frame portion to generate a 2D plus depth information frame portion;
• reconstructing depth information using a Poisson reconstruction, according to:
• (
• ∂
• 2
• ∂
• x
• 2
• +
• ∂
• 2
• ∂
• y
• 2
• )
• ⁢
• D
• =
• ∇
• ·
• G
• ,
• where G=(Gx, Gy) is the depth gradient, D is the depth information and ∇ G is the divergence of G:
• ∇
• ·
• G
• =
• (
• ∂
• G
• x
• ∂
• x
• +
• ∂
• G
• y
• ∂
• y
• )
• .
• and
• forcing the higher-order depth derivatives to be zero, comprising setting, in the continuous domain:
• (
• ∂
• 4
• ∂
• x
• 4
• +
• ∂
• 4
• ∂
• y
• 4
• )
• ⁢
• D
• =
• 0.

A system to process 2D video images from a video stream for converting the 2D video images to 3D images, the system comprising:

• a synthetic 3D reference database comprising computer generated 2D images and corresponding depth information for the 2D images;
• a computer programmed to execute instructions comprising:
• submitting input video frames to the synthetic 3D reference database;
• matching at least a portion of an input video frame with at least a portion of a 2D image in the synthetic 3D reference database;
• selecting the corresponding depth information for that 2D image, wherein the depth information is a depth gradient;
• applying the selected depth information to the matched input video frame portion to generate a 2D plus depth information frame portion;
• identifying objects in the input video frame;
• determining object masks for the identified objects;
• and
• estimating the depth information using the determined object masks and the matched input video frame, allowing depth discontinuities at object boundaries by modifying the Poisson equation there.