# US 2017/0289516 A1 · Depth map based perspective correction in digital photos

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0289516 A1
- Document ID: US20170289516A1
- Status: Published application
- Kind: A1
- Application number: 15/508,390
- Filed: 2015-09-04
- Published: 2017-10-05
- Earliest claimed: 2014-09-05
- Assignees: Polight AS
- Applicants: Polight AS
- Inventors: Tore Svortdal; Pierre Craen; Nicolas Tallaron; Jacques Dumarest
- Web page: https://patentgrep.com/patent/US20170289516A1

## CPC

- G06T 3/00: Geometric image transformations in the plane of the image
- G06T 5/80: Geometric correction
- H04N 13/122: Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence)
- H04N 13/128: Adjusting depth or disparity
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- H04N 2213/003: Aspects relating to the "2D+depth" image format
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

The invention relates to post-processing of a digital photo to correct perspective distortion in the photo. The correction applies a digital photo of a scene and a depth map associated with the photo and comprising, for each pixel in the photo, a depth being a distance between a part of the scene in that pixel and a position of the camera at the time of acquisition. The correction is performed locally, so that the correction of any pixel in the photo depends on the depth of that pixel. The correction can be implemented as a transformation of each pixel in the original photo into a new position in a corrected photo. Afterwards, pixel values has to be calculated for the pixels in the corrected photo using the original pixel values and their new positions. The invention is particularly relevant for photos where objects or scenes involves a large magnification variation, such as selfies, close up photos, and photos when the extension of a large, object is not orthogonal to the optical axis of the camera (low/high angle shots).

## Claims

### Claim 1

A method for automatically performing perspective correction in a digital photo of a scene recorded with a camera, the method comprising:
providing, using only a single camera with a single acquisition position,
a digital photo of a scene represented by pixel values, P(x,y), for an array of pixels, (x,y), the photo having perspective distortion effects;
a depth map associated with the photo and comprising, for each pixel in the pixel array, a depth, d(x,y), being a distance between a part of the scene represented by that pixel and an acquisition position of the camera at the time of acquisition of the photo; and
performing perspective correction of the photo using as photographic input only the photo of the scene from the single camera and acquisition position and its associated depth map by, for each pixel, (x,y), in the photo, determining a new position, Dproc, in an image plane for a virtual camera position from at least the pixel's depth, dacq(x,y), its position, Dacq(x,y), in the image plane at the acquisition position, and a displacement, C, between the virtual camera position and the acquisition position as:
D
proc
=
D
acq

(
x
,
y
)
*
d
acq

(
x
,
y
)
d
acq

(
x
,
y
)
+
C

### Claim at position 2

2-14. (canceled)

### Claim 15 (depends on 1)

The method according to claim 1, wherein the step of performing perspective correction further comprises adjusting a magnification of the new position using also a depth, dacq_ref, of a reference plane chosen to preserve a magnification in a selected plane of the depth map.

### Claim 16 (depends on 15)

The method according to claim 15, wherein the step of performing perspective correction comprises determining the new position as:
D
proc
=
D
acq

(
x
,
y
)
*
d
acq

(
x
,
y
)
*
(
d
acq_ref
+
C
)
(
d
acq

(
x
,
y
)
+
C
)
*
d
acq_ref

### Claim 17 (depends on 1)

The method according to claim 1, wherein the step of performing a perspective correction further comprises using the pixel values P(x,y) and the new positions (x′,y′) to determine new pixel values, P(i,j), for an array of pixels, (i,j), representing a corrected photo by, for each new position (x′,y′), adding the corresponding pixel value P(x,y) to the new pixel values P(i,j) of the pixels (I,j) surrounding the new position, wherein the pixel values P(x,y) are weighted by a weighting factor that is a function of the relative positions of the new position (x′,y′) and each pixel (i,j).

### Claim 18 (depends on 17)

The method according to claim 17, further comprising subsequently dividing each new pixel value P(i,j) with a normalization factor.

### Claim 19 (depends on 17)

The method according to claim 17, further comprising subsequently, for a new pixel with undefined values of P(i,j), calculating an interpolated pixel value from surrounding pixels in the corrected photo having defined values of P(i,j).

### Claim 20 (depends on 1)

The method according to claim 1, wherein the displacement between the virtual camera position and the acquisition position is a linear displacement along an optical axis of the camera.

### Claim 21 (depends on 1)

The method according to claim 1, wherein the steps of providing a photo and providing a depth map comprise providing a series of photos and associated depth maps, and wherein the method further comprises detecting and evaluating perspective distortion in the photos based either on the distance of the closest object in the scene or on the analysis of vanishing lines, selecting photos with perspective distortion that would benefit from perspective correction, and automatically performing perspective correction on the selected photos.

### Claim 22 (depends on 1)

The method according to claim 1, further comprising performing perspective correction of the depth map by, for each pixel, (x,y), in the depth map, and determining a new position, Dproc, in an image plane for a virtual camera position from at least the depth held by that pixel, dacq(x,y), its position, Dacq(x,y), in the image plane at the acquisition position, and a displacement, C, between the virtual camera position and the acquisition position.

### Claim 23 (depends on 1)

A digital storage holding software configured to perform the method of claim 1 when executed by one or more digital processing units.

### Claim 24 (depends on 1)

An integrated circuit configured to perform the method of claim 1.

### Claim 25 (depends on 23)

A handheld or portable device with a camera comprising the digital storage holding software according to claim 23.
