# US 2014/0198176 A1 · Systems and methods for generating a depth map and converting two-dimensional data to stereoscopic data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0198176 A1
- Document ID: US20140198176A1
- Status: Published application
- Kind: A1
- Application number: 14/217,600
- Filed: 2014-03-18
- Published: 2014-07-17
- Earliest claimed: 2011-06-28
- Assignees: Cyberlink Corp.
- Applicants: Cyberlink Corp.
- Inventors: Hao-Ping Hung
- Web page: https://patentgrep.com/patent/US20140198176A1

## CPC

- H04N 13/128: Adjusting depth or disparity
- H04N 13/261: with monoscopic-to-stereoscopic image conversion
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Systems and methods for performing multimedia conversion are disclosed. In a multimedia conversion system for converting two-dimensional (2D) multimedia data to stereoscopic multimedia data, an absolute difference between pixel pairs is determined for each scan line in a frame of the 2D multimedia data. The absolute difference values are accumulated, and a depth map is derived based on the accumulated difference values. The depth map is applied to the frame in the 2D multimedia data.

## Claims

### Claim 1

A method implemented in a multimedia conversion system for converting two-dimensional (2D) multimedia data to stereoscopic multimedia data, comprising:
determining vertical characteristics of pixels in a frame of the 2D multimedia data, wherein determining vertical characteristics of the pixels comprises for each scan line in the frame, determining a difference between pixel pairs comprising a pixel in a current scan line and a pixel in a second scan line;
calculating a depth for each pixel in the 2D multimedia data according to a depth of a vertical pixel in the second scan line and the determined vertical characteristics of the pixel; and
generating a complementary stereoscopic image pair according to the depth of each pixel.

### Claim 2 (depends on 1)

The method of claim 1, wherein the second scan line is a neighboring scan line of the current scan line.

### Claim 3 (depends on 1)

The method of claim 1, further comprising prior to the steps of determining, calculating, and generating, performing the step of removing any letterboxing effect present in the 2D multimedia data.

### Claim 4 (depends on 1)

The method of claim 1, wherein the difference between pixel pairs is an absolute difference.

### Claim 5 (depends on 1)

The method of claim 1, wherein the difference between pixel pairs is normalized according to an absolute difference.

### Claim 6 (depends on 1)

The method of claim 1, wherein the vertical characteristics comprise at least one of: a scan line absolute difference; a difference between pixels in the second scan line; and an absolute difference between pixels in the second scan line.

### Claim 7 (depends on 6)

The method of claim 6, wherein the scan line absolute difference is calculated according to the following:
D
1

[
y
]
=
∑
x
=
1
W


abs

(
p

(
x
,
y
)
-
p

(
x
,
y
-
k
)
)
,
wherein D1[y] represents the scan line absolute difference; abs( ) is an absolute function; p(x, y) represents a pixel value at position (x, y); W is an image width of the frame of the 2D multimedia data; and k is a constant.

### Claim 8 (depends on 6)

The method of claim 6, wherein the difference between pixels in the second scan line is calculated according to the following:
D
2

[
y
]
=
∑
x
=
1
W


p

(
x
,
y
)
-
p

(
x
-
k
,
y
)
,
wherein D2[y] represents the difference between pixels in the second scan line; p(x, y) represents a pixel value at position (x, y); W is an image width of the frame of the 2D multimedia data; and k is a constant.

### Claim 9 (depends on 6)

The method of claim 6, wherein the absolute difference between pixels in the second scan line is calculated according to the following:
D
3

[
y
]
=
∑
x
=
1
W


abs

(
p

(
x
,
y
)
-
p

(
x
-
k
,
y
)
)
,
wherein D3[y] represents the absolute difference between neighboring pixels in the second scan line; abs( ) is an absolute value function; p(x, y) represents a pixel value at position (x, y); W is the image width of the frame of the 2D multimedia data; and k is a constant.

### Claim 10 (depends on 6)

The method of claim 6, wherein determining vertical characteristics further comprises determining a running sum for each position y, wherein determining a running sum is calculated according to the following:
Sum

[
y
]
=
∑
i
=
1
y


(
w
1
*
D
1

[
i
]
+
w
2
*
D
2

[
i
]
+
w
3
*
D
3

[
i
]
)
wherein Sum[y] represents the running sum, D1[i] represents the scan line absolute difference, D2[i] represents the difference between pixels in the second scan line, D3[i] represents the absolute difference between neighboring pixels in the second scan line, and w1, w2, w3 are weighting factors of D1, D2, D3.

### Claim 11 (depends on 10)

The method of claim 10, wherein the running sum is calculated beginning with a difference value associated with the first scan line at the top of the frame and progressing on a pixel-by-pixel basis to the scan line at the bottom of the frame.

### Claim 12 (depends on 6)

The method of claim 6, wherein determining vertical characteristics further comprises determining a running sum for each position y, wherein determining a running sum is calculated according to the following:
Sum

[
y
]
=
∑
i
=
1
y


max

(
w
1
*
D
1

[
i
]
+
w
2
*
D
2

[
i
]
+
w
3
*
D
3

[
i
]
,
lower_bound
)
,
wherein Sum[y] represents the running sum, D1[i] represents the scan line absolute difference, D2[i] represents the difference between pixels in the second scan line, D3[i] represents the absolute difference between neighboring pixels in the second scan line, max ( ) represents a maximum function, and w1, w2, w3 are weighting factors of D1, D2, D3.

### Claim 13 (depends on 1)

The method of claim 1, wherein the depth at a specific vertical position y is determined by:
Depth[y]=g(f(y),Depth[y−k]),
wherein the function f denotes vertical characteristics and g denotes a function to transform the depth at reference position y−k and the vertical characteristic into a depth at the target position, where k is greater than zero.

### Claim 14

A method implemented in a multimedia conversion system for converting two-dimensional (2D) multimedia data to stereoscopic multimedia data, comprising:
determining, for each scan line in a frame of the 2D multimedia data, a scan line absolute difference comprising an absolute difference between pixels in two scan lines;
accumulating the absolute difference values;
deriving a depth map based on the accumulated difference values; and
applying the depth map to the frame in the 2D multimedia data.

### Claim 15 (depends on 14)

The method of claim 14, wherein applying the depth map to the frame in the 2D multimedia data is performed according to a depth-image-based rendering (DIBR) technique.

### Claim 16 (depends on 14)

The method of claim 14, wherein determining the scan line absolute difference comprises determining a difference of a pixel in a current scan line and a pixel in a second scan line, wherein the pixel in the current scan line and the pixel in the second scan line are in a same relative horizontal pixel position.

### Claim 17 (depends on 14)

The method of claim 14, wherein accumulating the difference values is performed for a same relative horizontal pixel position in both the current scan line and a second scan line.

### Claim 18 (depends on 17)

The method of claim 17, wherein the second scan line is a neighboring scan line of the current scan line.

### Claim 19 (depends on 17)

The method of claim 17, wherein accumulating the difference values is performed sequentially beginning with the first scan line at the top of the frame to the last scan line at the bottom of the frame.

### Claim 20 (depends on 14)

The method of claim 14, further comprising storing the accumulated difference values in a one-dimensional (1D) array, wherein the total number of entries in the 1D array corresponds to the total number of scan lines in the frame.

### Claim 21 (depends on 14)

The method of claim 14, further comprising:
based on a resolution of the 2D multimedia data, reducing the resolution of the 2D multimedia data prior to calculating a depth for each pixel in the 2D multimedia data.

### Claim 22

A system for converting two-dimensional (2D) multimedia data to stereoscopic multimedia data, comprising:
a comparator configured to determine, for all scan lines in the frame, an absolute difference value between corresponding pixels in pairs of scan lines in the frame;
an accumulator configured to accumulate the absolute difference values;
a depth map generator configured to derive a depth map based on the accumulated relative difference values and based on the 2D multimedia data; and
a 2D-to-stereoscopic converter configured to apply the depth map to generate a stereoscopic multimedia data from the 2D multimedia data.

### Claim 23 (depends on 22)

The system of claim 22, further comprising a pre-processing module configured to remove any letterboxing effect present in the 2D multimedia data.

### Claim 24 (depends on 22)

The system of claim 22, wherein the comparator determines an absolute difference value between corresponding pixels in pairs of scan lines in the frame across all pixel positions in the frame.

### Claim 25 (depends on 22)

The system of claim 22, further comprising an object detector configured to determine whether a current frame of the 2D multimedia data only contains scenery objects.

### Claim 26 (depends on 25)

The system of claim 25, wherein the object detector determines whether a current frame of the 2D multimedia data only contains scenery objects based on whether the current frame contains any human faces.

### Claim 27 (depends on 22)

The system of claim 22, wherein the pairs of scan lines comprise a current scan line and a second scan line.

### Claim 28 (depends on 22)

The system of claim 22, wherein the accumulator accumulates the absolute difference values beginning with the first scan line to the last scan line in the frame, the first scan line being located at the top of the frame.

### Claim 29 (depends on 22)

The system of claim 22, wherein the 2D-to-stereoscopic converter is configured to apply the depth map to the frame in the 2D multimedia data according to a depth-image-based rendering (DIBR) technique.